A wind tunnel
for your AI

Know whether your agent's changes made things better.

Request Invite

A loop needs a referee

Did the change actually help?

Set a bounded goal. Lock the measurement. Titration tells your agent whether an attempt earned another one.

Independent referee

Goal01

Choose what to improve

Pick a behavior in an AI system you already run, then define what better looks like.

Harness02

Lock the test

Set the examples, baseline, and scoring rules before the first attempt.

Agent attempt03

Try an improvement

Your agent makes the next change and runs the frozen test. Titration grades the outputs.

Referee04

Decide what happens next

Titration compares the evidence and returns one decision: continue, converge, or stop honestly.

What happens next Titration returns one clear decision.

Continue

Try another improvement against the same test.

02 HarnessSame test

Converge

Keep the improvement and save what worked.

Approved learningPrivate memoryShapes the next test

Stop honestly

Stop when the evidence does not justify another try.

Loop ends hereNo improvement is claimed.

Generation is cheap. Knowing what improved is not.

Agents can make more changes than a team can confidently evaluate. Titration gives each bounded loop an independent measurement layer.

Traditional computers automate what you can specify in code. This latest round of LLMs can automate what you can verify. Andrej Karpathy · Sequoia 2026 ↗

AI agents can now produce software faster than any team can verify it. Datadog ↗

The bottleneck is no longer generation. It's verification. Addy Osmani ↗

A narrowly scoped agent that compounds feedback over time outperforms a broadly scoped agent loaded with everything at once. Augment Code ↗

Loops are replacing prompts. Verification is about to be your biggest problem. The New Stack ↗

Wind tunnel · live run●

Build the test before you start improving.

Before the agent makes any changes, Titration helps shape a controlled test around the exact behavior you care about. The Harness holds the examples, the baseline, and the rules for success.

First, make sure the test can see the original failure. Otherwise, you cannot know whether a change actually helped.

Let your agent try a change.

Your agent makes the next change and runs it locally. Titration compares the result with the goal and tells it what to do next.

Because the goal and test stay locked, the agent cannot move the goalposts and call its own work a success.

Try again. Keep the improvement. Or stop honestly.

Titration compares every attempt with the same starting point. It tells your agent whether to try again, keep a verified improvement, or stop.

If the result could be chance, Titration says inconclusive. It never calls uncertainty a win.

Keep what you learn.

When you approve a learning, Titration saves what failed, what fooled the measurement, and what finally worked — privately in your workspace.

The next test starts with what you already know, so improvement does not reset to zero.

Private memory

Keep what works. Learn from what doesn’t.

Completed runs reveal what failed, what fooled the test, and what held. You decide which lessons are worth saving.

That private memory stays in your workspace. Titration uses it only when it applies to a new test, and never mixes it with another customer’s.

Signals retained From completed work

Failure

The baseline did not match production.

Improvement

The grounded answer held across judges.

Approval boundary Only approved learning enters

Private to your workspace Private memory

Connected evidence · reusable lessons

Retrieved when relevant

Next harness Starts smarter

Known traps, stronger rubric, better watchlist

Next run adds new evidence

Four checks before you call it a win.

A change counts as an improvement only when it clears all four.

Consensus

More than one judge

Important verdicts use AI judges from at least two providers. No single vendor decides the result.

Baseline

Keep the comparison fair

The starting point and success rules are fixed before anything changes. No moving the goalposts.

Diagnosis

Find the real problem

Before suggesting a change, Titration checks whether the problem is the AI, the test, the judge, or the measurement.

Uncertainty

Sometimes the answer is “not sure yet.”

If the difference could be noise, Titration says inconclusive. It never calls uncertainty a win.

It stops you fixing the wrong thing.

The first metric rarely tells you what actually failed. The fault may be in the AI, the test, the judge, or the way success was measured.

Production gap

"The eval was measuring a world production doesn't run."

A test held a component ON that prod doesn't have; two prior sessions "proved it fine."

→ flipped it to match prod: 50% of long conversations broke

Judge failure

"Your test says 0 / 12. The grader is broken."

The judge mis-scored legitimate content as errors, through a whole iteration loop.

→ fixed the judge: 12 / 12

Evaluation blind spot

"Regex said 0% failures. The judge said 47%."

A content-policy prompt looked ship-ready under keyword checks; paraphrases hid the failures.

→ a 47% failure rate the keyword check couldn't see

Measurement noise

"85% of your 'broken' number was measurement noise."

A metric counted natural short phrases as repetition.

→ 67% → 0% after de-artifacting

Prompt anchor

"One word in an example collapsed every output."

A literal example value in a prompt anchored the model.

→ 33 of 35 outputs identical → 7 distinct

False trend

"A clean improvement curve that was pure luck."

A bimodal metric averaged over 2 runs produced a fake monotonic trend.

→ a "curve" from a 12× run-to-run spread

In one live system, catches like these made up a third to a half of apparent failures. Real private dogfood, not a large-N study, with no customer claims and no fabricated proof.

Your work stays yours

Titration measures the result. It does not take over the work.

Your agent runs locally with your code, tools, and data. You send Titration only the outputs and evidence you choose.

Your environment Declared outputs + evidence Titration

Titration does not run your repo or require your source code.

Questions you might have.

How can an agent improve something without running wild?+

You set the goal, examples, and definition of better first. Your agent can then make changes toward that target. Titration measures every attempt and tells it whether to try again, keep the improvement, or stop. The agent cannot change the rules to call itself successful.

Does Titration run or see my code?+

No. Your agent works locally. You send Titration only the outputs and declared evidence you choose to share.

Why use more than one AI judge?+

One judge can be wrong. Important results are checked by judges from at least two AI providers, and Titration shows you when they disagree.

Is this only for prompts?+

No. You can test any AI behavior: a prompt, agent, retrieval system, classifier, content pipeline, or multi-step workflow.

Does every run become memory?+

No. You choose. Only approved lessons are saved privately in your workspace and reused when they apply to a new test.

For people ready to let AI run, but not run blind.

AI product teams

Improving behavior in production

Turn real failures into repeatable tests and know whether the next change actually helped.

Agent builders

Giving AI more autonomy

Give an improvement loop a trustworthy goal, referee, and stopping point.

Quality owners

Responsible for what ships

Replace screenshots and impressions with evidence that survives the next run.

Private beta

Request an invite.

Tell us what you are building and what you want to improve. We review every request before granting access.

The confidence to let it run

Let your agent improve without flying blind.

Set the goal. Build the test. Let Titration measure every change, while you stay in control of what counts as better.

Invite-only private beta · Sign in if you already have access.