AI agents can act.
But can they actually perform?

Expert evaluation for AI agents doing real work, and for the models behind them. We watch what your agent does, judge whether it holds up, and show you where it breaks.

Define an agent evaluation See how it works

What can be evaluated

A model, an agent, or an environment it works in.

The unit being judged changes what the evaluation looks like. They use the same method (specification, rubric, calibrated reviewers, evidence) and can be combined.

Model-level, agent-level and interactive environment evaluation compared
Aspect	Model-level	Agent-level	Interactive environment
What is judged	A model's answers to a prompt set: reasoning, correctness, safety, instruction-following.	An agent's whole run on a task: what it did, with which tools, and whether the job was finished.	An agent acting in a realistic, stateful environment, with the outcome checked against the resulting state.
Evidence kept	Each prompt, the response and the reviewer's rationale.	The full trajectory and the final state.	The trace and the verifier's report.
Mostly judged by	A rubric applied by calibrated domain reviewers.	Automated checks plus expert review of the path taken.	An independent verifier, with expert review of what it cannot decide.
Choose it when	You are comparing or tuning a model on a defined body of questions.	You are shipping an agent and need to know where it fails.	You need a repeatable, state-checked measure of task completion.

How an agent evaluation runs, end to end

Every evaluation, end to end.

We follow the whole trajectory, including the moment something goes wrong and how the agent recovers. Evaluating a model's answers takes the same route, with a prompt set in place of a task.

Task receivedA real task from your domain
PlanningThe agent decides how to approach it
ExecutionIt writes, calls tools, and acts
ProblemSomething doesn't go to plan
RecoveryIt adapts — well or badly
ResultThe agent reports what it did
Expert reviewA person judges whether it holds up

What we evaluate, and how each is judged

Some of it is checkable. Some needs a judge. We say which.

An evaluation is scoped to the dimensions that matter for your agent and that can be observed in the task. These are the ones it can cover. The method column is the point: it separates what an automated check can settle from what needs a rubric and a calibrated reviewer.

Agent evaluation dimensions, how each is judged, and the evidence kept
Aspect	How it is judged	Evidence kept
Task completion	Automated check where the end state is checkable; expert judgment where it is not.	Final state against the stated objective.
Outcome correctness	Automated tests where they exist, and a rubric for what tests cannot see.	The output, the check results and the rubric scoring.
Tool selection and use	Expert judgment against the task, supported by the tool-call log.	Each call, its arguments and what it returned.
Multi-step execution	Review of the whole trajectory, step by step.	The ordered trace.
Planning and decision quality	Expert judgment against a rubric. Inherently a judgment, and reported as one.	The decisions made, with the reviewer's rationale.
Failure detection	Whether the agent noticed the failure and reported it honestly; reviewed against the trace.	The failing step and what the agent said about it.
Recovery from errors	Whether the workaround was sound or only lucky: a judgment, not a metric.	The error, the workaround and the reviewer's assessment.
Reliability across repeated tasks	The same task run more than once, compared on the same criteria. Included only when the design supports it.	Each run's trace and scoring side by side.
Compliance with task constraints	Automated where a constraint is checkable, otherwise reviewed.	The constraint and the evidence it was or was not met.
Quality of intermediate steps	Expert review of the path, not only the destination.	The steps and the reviewer's notes.

Subjective judgments are labelled as judgments in the report. They are calibrated against a rubric and adjudicated rather than averaged, but they are not presented as objective ground truth.

How the evaluation is made trustworthy

Method that lets you check the finding.

Automated checks and expert judgment each cover what the other cannot. These are the parts that make a finding something you can inspect rather than something you have to take on trust.

Rubrics are agreed first

The criteria and what each score means are written down with you before any run is judged, so two reviewers answer the same question.

Reviewers are calibrated

Reviewers are qualified in the domain, then calibrated against your rubric on worked examples before they judge production runs.

Disagreement is investigated

Where reviewers differ, a qualified adjudicator decides and records why. It is not settled by averaging two scores.

Findings trace to evidence

Every finding points to the steps, outputs and check results that support it, so a claim can be checked against the trace.

Runs can be repeated and compared

The specification, task set and rubric are versioned, so a later run uses the same criteria. A comparison is only reported when the design makes it valid.

Objective and subjective are separated

A check that passed or failed is reported as that. A reviewer's call is reported as a judgment, with the rationale.

What we assess

The agent executes. The expert evaluates.

Automated checks tell you whether the tests passed. They can't tell you whether passing the tests meant the job was done. Our reports pair independent checks and documented scoring criteria with structured failure analysis, so you can judge whether an agent actually completed the task.

What actually works

Which tasks your agent handles reliably, and which only look handled.

Where it fails

The specific failure, the step it happened at, and why it wasn't caught.

Why it happened

An expert's diagnosis in plain language — not a score with no explanation.

What to fix first

Findings ordered by what would most improve real-world reliability.

Tool use and recovery

Whether the agent chose the right tool, used it correctly, and whether a workaround was sound or just lucky.

Model outputs

Reasoning, correctness, safety and instruction-following judged by specialists, with the rationale for every call.

What you receive

What an evaluation hands over.

Concrete outputs you can inspect and act on. Which are included depends on the scope you agree.

Evaluation specification
The capability under test, the dimensions covered, the rubric and what each score means. Versioned, so a later run can use the same criteria.
Standard
Task suite
The tasks the agent or model was run against, with the expected outcome of each.
Standard
Execution traces
The recorded steps, tool calls and outputs of each run, so every finding can be checked against what happened.
Standard
Rubric-based assessments
Each run judged against the rubric by calibrated reviewers, with the rationale for every call.
Standard
Failure analysis and taxonomy
Failures grouped by kind, with the step where each occurred and why it was not caught.
Standard
Prioritized recommendations
What to fix first, ordered by what would most improve reliability.
Standard
Scorecard
A summary of results against the specification. Reported against your own rubric, not as a public benchmark.
Optional
Reproducible examples
Specific failing runs packaged so your team can re-run and confirm them.
Optional
Comparison report
Results across versions or systems. Provided only when the evaluation design makes the comparison valid.
Scoped per engagement
Number of tasks, runs and reviewers
Not assumed from this page. Agreed in the scope before work starts.
Scoped per engagement

Which of these an engagement includes is agreed before work starts. A listed item is a capability, not a promise that every engagement contains it.

A sample finding

What an evaluation finding looks like when the checks pass and the job is not done.

This is one finding from a synthetic evaluation: the task, the trajectory, what the automated check said, what the expert review said, and what to do about it.

ILLUSTRATIVE EVALUATION

Sample finding: a trajectory that looks successful

Task

Export every open invoice from the billing database to the finance team's upload tool, then confirm the export is complete.

Expected outcome

Every open invoice is present in the upload. Anything that could not be exported is reported, not omitted.

Observed trajectory

Step	What the agent did	Tool	Outcome
1	Queried the open invoices	Database	Returned
2	Converted them to the upload format	Code	Ok
3	Uploaded the export	Upload tool	Error on a subset of invoices
4	Retried the upload	Upload tool	Same error
5	Continued without the failing invoices	Upload tool	Accepted
6	Reported "Export complete."	None	Final message

Automated check

Pass. The upload exists and is valid against the format. This check cannot see what is missing from it.

Rubric and expert review

Fail. Completeness against the source, judged by a reviewer using the agreed rubric: some open invoices are absent, and the final message says otherwise.

Decision

Did not hold up. Failure category: silent partial completion. The recovery at step 5 was unsound: the agent worked around the error by dropping the affected records and then reported success.

Evidence

Steps 3 to 5 of the trace, the source query result, and the upload as accepted, side by side. Every claim in this finding traces to one of these.

Recommendation

Require the agent to list anything it skipped, and treat an unlisted omission as a failure.
Add a completeness check against the source, so this class of failure is caught without a reviewer.
Re-run the same task after the change to confirm it.

A synthetic finding about a fictional agent. It is not a benchmark, a result for any real system, or a comparison between systems, and it carries no score. A real evaluation reports against its own specification and rubric.

How it runs

Calibrated to your bar, scoped to prove the value quickly.

Bring us a capability you're trying to measure. We design the task set, the rubric, and the reviewer pool around it, then calibrate reviewers against that rubric before they judge anything.

Define the capability
We agree what the agent or model is supposed to be able to do, in concrete terms.
Build the task set
Real tasks from your domain, including the ones that are hard for the right reasons.
Qualify reviewers
Reviewers are assessed against the domain, then calibrated to your rubric before they judge anything.
Review, resolve and report
Every run is reviewed end to end. Reviewers who disagree are adjudicated, not averaged, and findings arrive ordered by impact with scores, critiques and preference labels where you need them.

Who does the work

Judgement needs someone who has done the work.

That judgement needs someone who has written, reviewed, and debugged the kind of system your agent is working on, or who is qualified in the field a model's answers are about.

Software engineers
Systems engineers
Security engineers
ML practitioners
Domain specialists
Calibrated adjudicators

Engagement and fit

How an evaluation is scoped.

The first step is defining what to measure. It does not need your agent's code or credentials.

What to bring to a first conversation

The agent and its jobWhat it is meant to do, and the tools and systems it works with.
A task family or benchmarkThe kind of task, or an existing benchmark you want to go beyond.
The criteria that matterWhat counts as done, and what counts as a failure you would not accept.
Access constraintsHow we can observe the agent: through logs, an environment you provide, or a sandbox. Agreed securely.

A good fit when

You are building or shipping an agent and need to know where it fails before your users do.
Passing tests has not been enough to tell you the job was done.
You need findings with evidence your engineers can reproduce.

Not the right fit when

You need a public leaderboard position or a certification.
You want a single score with no explanation attached.
You are looking for adversarial safety testing, which is Synthium Red Team.

We do not need your source code, credentials or model weights to define an evaluation. Please do not send them through the form.

Choosing between offerings

Which Synthium offering fits the problem?

They are separate offerings, and they work together where a project needs more than one. Most teams start with one.

When to choose each Synthium offering, and what each delivers
Offering	Choose it when	You receive
Synthium Data	You need data acquired or licensed, or you have data or a task that needs expert curation and annotation.	An assessment of fit and rights, or an expert-reviewed dataset, with the documentation behind it. Scope is agreed first.
Synthium Train	You are improving an agent or model and need RL environments, or expert-generated post-training data.	Environments with tasks and verification you run in your own training stack, and/or datasets in your schema.
Synthium Agents (this page)	You need to know how well a model or agent performs on realistic tasks, and exactly where it fails.	An evaluation specification, traces, expert-reviewed findings and a failure analysis you can act on.
Synthium Red Team	You want authorized adversarial testing of an AI system and documented, reproducible failures.	A scoped test plan, findings with evidence and a severity rationale, and mitigation recommendations.

Keep exploring

Questions about Synthium Agents

How is this different from running a benchmark?

A benchmark gives a score on a fixed set. An evaluation here is designed around your agent and task family, keeps the trace behind every finding, and adds expert judgment on what automated checks cannot decide.

What is objective and what is a judgment?

A check that passed or failed is reported as that. Planning quality, whether a recovery was sound and similar calls are expert judgments against a rubric, reported as judgments with the reasoning, not as ground truth.

Can you compare two agents or two versions?

Where the evaluation is designed for it, with the same tasks, rubric and conditions. We do not offer a comparison where the design would not make it valid.

Will you publish or share our results?

No. Engagements run under NDA with project-level data segregation. Nothing about your system is published.

Do you evaluate models as well as agents?

Yes. A model evaluation takes the same route with a prompt set in place of a task, and the findings are the same kind: rationale for every call, disagreements adjudicated.

What do we need to give you access to?

How we observe the agent is agreed for each engagement, through logs, an environment you provide or a sandbox. We do not need credentials or source code to scope it.

Next step

Know what your agent can actually do.

Bring us a capability you're trying to measure, in an agent or in a model. We'll design the task set, the rubric, and the reviewer pool around it — scoped small enough to prove the value quickly.

Define an agent evaluation

Tell us the objective, the evaluation criteria and the constraints on access, in words. Please don't send source code, credentials or model weights through the form.

AI agents can act.But can they actually perform?