Synthium isn't only a delivery partner. The expert layer we operate generates hard-won signal about how frontier models behave under real evaluation — and we work to turn that signal into defensible method. This is where we publish what we learn.
What we study
An early-stage hub. We're open about what we investigate and are building toward — without inventing findings we haven't published.
We're building repeatable ways to measure where frontier models actually break — not headline leaderboard scores, but the task-level and domain-level gaps that matter for deployment. Methods and findings are shared in pilots as they mature.
Where does automated judging agree with expert human judgment, and where does it quietly fail? We study disagreement patterns, failure modes and when human review is non-negotiable.
How do you turn a qualified domain expert into a consistent, measurable evaluator? We investigate calibration, gold-task design and the processes that make expert labels trustworthy at scale.
Coding and AI agents fail in ways generic annotators miss. We analyze the failure modes of agent trajectories — correctness, specification drift, edge cases — and how to evaluate them reliably.
Beyond final-answer accuracy, we study how to assess multi-step reasoning: where models fabricate, where they shortcut, and how expert graders can separate real competence from plausible output.
Legal, medical, financial and scientific evaluation require domain experts who can tell right from wrong. We build toward evaluation frameworks grounded in real professional judgment, not surface fluency.
In preparation
These are in-preparation research notes — real directions we're investigating, not published findings. We share them here to be transparent about what we're working on.
RESEARCH NOTE
Static question-answer pairs miss the reasoning path. Trajectories capture the decisions, tool calls, and recoveries that distinguish capable agents from lucky ones.
RESEARCH NOTE
Binary pass/fail hides whether an agent solved the problem or stumbled into the answer. Multi-dimensional evaluation surfaces the difference.
FOCUS AREA
Not all enterprise data improves models. The signal is in the decisions people made, not the records they stored.
RESEARCH NOTE
Automated metrics scale but miss nuance. Expert evaluation is slow but catches what metrics cannot. The question is how to combine them.
FOCUS AREA
A sandbox that is too simple teaches agents to cheat. A sandbox that is too complex teaches nothing. Environment design is evaluation design.
RESEARCH NOTE
Operational records contain implicit decision sequences. Extracting them requires domain expertise, not just data engineering.
How we share
Much of what we learn is tied to client work and unreleased models, so the most detailed methodology, datasets and results are shared under NDA and in active pilots — not on a public blog. If you're building evaluation for a frontier system, you can work directly with our research team on the problems that matter to you.