Three tools, tried far enough to have an opinion
LangSmith, promptfoo and Braintrust each answer a different question, and an evidenced preference between them is worth more in the interview than a survey of all three.
The question in the interview will not be which eval tool is best. It will be something closer to “what did you use, and why that one.” A survey of three answers that one badly. An opinion formed by running something answers it in a sentence.
So this lesson is not a comparison table with a winner at the bottom. It is the smallest amount of orientation needed to pick one, plus a pass condition that requires you to have actually run it.
Say the bias first
All three of the sources below are vendors describing their own products. LangChain writes the LangSmith documentation and sells the hosted platform it leads to. Braintrust writes the Braintrust guide and sells the platform. promptfoo writes the promptfoo docs and sells a commercial offering alongside the open-source tool. There is no independent benchmark of these three that was opened for this course.
Which is exactly why the lesson ends in a terminal. Every capability claim below is the vendor’s own, unverified by anyone else, and the cheapest way to convert one into knowledge is to make it true or false on your own machine.
promptfoo — the offline one you can run tonight
A config file and a CLI. Test cases, deterministic assertions and graders, model-graded checks, red teaming, CI integration — you write a YAML file, you run a command, you get a table. It is MIT licensed, so there is no account, no seat and no trial clock.
What it does not have is an annotation-queue or online-evaluation primitive comparable to the other two. It is built for the pre-deployment half of the split: a dataset you curated, run on a commit, in CI.
One ownership fact, because it will come up and because it has moved. promptfoo announced on 9 March 2026 that it “has agreed to be acquired by OpenAI,” adding that “the closing of the acquisition is subject to customary closing conditions,” and stating that the project will remain open source. As of 5 September 2026, promptfoo’s own blog, OpenAI’s announcement and the press coverage found alongside them all still describe the deal as agreed rather than closed, and no source stating that it has formally closed was found. Say “agreed to be acquired,” not “acquired,” and treat the remain-open-source line as what it is: a vendor’s commitment about its own future, published on the day of the announcement.
LangSmith — the one built around the trace
LangChain’s framing starts from the trace as the base record. Datasets are built out of production traces, evaluators live at the workspace level and include LLM-as-judge and pairwise scorers, and the offline and online split is explicit — datasets for benchmarking and regression testing before you ship, production traffic for monitoring after.
The part relevant to a review gate is the annotation queue: a place where humans leave structured feedback on traced runs, and where that feedback becomes dataset material. That is the primitive your corrections have to land in.
Its published pricing on 5 September 2026: a $0 Developer tier including 5,000 base traces per month, and a Plus tier at $39 per seat per month including 10,000 base traces, with Enterprise quoted individually.
Braintrust — the one that scores production
Braintrust decomposes an eval into data, task and scorers and leans hardest on online scoring: production traces graded as they are logged, using LLM-as-judge, because live requests have no ground truth. It also has an annotate phase where human review feeds dataset construction.
Read that sentence about ground truth next to everything the judge lesson established. Online scoring is where position bias, verbosity bias and self-preference stop being academic, because the score being produced has nothing to check it against. If you evaluate one thing on this platform, evaluate how the judge is configured.
Its published pricing on 5 September 2026: a $0 Starter tier and a Pro tier at $249 per month, which the page describes as including roughly 5GB of data and 50,000 scores per month with per-unit overage beyond that, plus Enterprise quoted individually.
The axis that matters for your job
Ignore the feature grids. There is one distinction on which these three differ in a way that changes an interface decision, and it is this course’s own framing rather than anything the vendors present:
| Tool | Answers the question | Where a human verdict enters |
|---|---|---|
| promptfoo | Did this commit make the output worse, against my dataset? | When you write or fix the dataset, offline. No queue primitive comparable to the other two. |
| LangSmith | What happened in this run, and can I build a dataset from it? | Annotation queues, on traced runs. |
| Braintrust | How is production scoring right now, with no reference? | An annotate phase feeding dataset construction. |
Third column, every time. Your review gate produces human verdicts on real items, and the only question that matters to you is where those verdicts can land such that they become evaluation data instead of a row in a database nobody reads.
How far is far enough
The lesson title promises a stopping condition, so here it is. You have tried a tool far enough when you can answer four questions without looking anything up:
- What did the config look like? Not the concepts — the file. How a test case is written, where the expected value goes, what a scorer looks like when you declare one.
- What broke? Every tool has a first failure that is about the tool rather than about your data. That failure is the most interview-durable thing you will learn all evening.
- Which checks were deterministic and which were not? A string match and a model-graded check sit next to each other in the same file and read almost identically. Knowing which of your fields resist the first kind is a fact about your document type, and it is the fact that transfers to the next product.
- What would you reach for instead, and for what requirement? This is the question that turns a run into an opinion. “promptfoo for regression in CI, but I would need something with an annotation queue the moment reviewer feedback has to land somewhere” is an answer with a shape.
Notice that none of the four requires you to have run all three. Evenings are finite, and two tools understood to that depth is a stronger position than three surveyed. If time only stretches to one, run one and say so — a stated boundary reads as judgement, while a survey presented as experience reads as a bluff until somebody asks a second question.
Where the survey habit costs you
“We looked at LangSmith, promptfoo and Braintrust” invites exactly one follow-up, and it is “what did you find?” Everything a landing page taught you dies at that question, because the interviewer has read the same landing pages.
What survives it is small and specific: what the config looked like, what broke, how long the first useful run took, which assertion you expected to be deterministic and turned out to be model-graded. Two tools understood that way beat three tools skimmed.
Check your recall
Answer from memory — no scrolling back.
Retrieval check
You have one evening. Which tool do you run, and what makes the run worth more than reading all three sites?
Check your answer
promptfoo, because it is MIT-licensed and config-driven, so nothing between you and a result is an account or a trial clock. The other two are platforms; evaluating a platform properly takes longer than an evening and produces an opinion about onboarding rather than about evaluation.
The run is worth more because it converts a claim into an experience you can be questioned about. You will find out how a deterministic assertion differs from a model-graded one by writing both, what your dataset was missing by watching a case pass that should not have, and how long the loop takes. None of that is on a landing page, and all of it survives a follow-up question.
Hands on
Run one, on ten real items
Done when: You have a promptfoo config in the repo, run against at least ten items taken from HouseWarm’s gate, with at least one deterministic assertion and one model-graded check — and at least one case fails for a reason you can explain in a sentence.
- Take ten reviewed documents from the gate and reduce each to an input and the field values a reviewer approved. Ten is enough to learn the tool. It is not enough to conclude anything about the model, and the spec should say so.
- Write the config with two kinds of check side by side: a deterministic assertion on a field whose correct value is exact, and a model-graded check on a field where correctness is a judgement. Notice which fields resist the first kind.
- Run it. Write down how long it took from empty directory to first table, including the parts that went wrong.
- Make one case fail deliberately, by changing an expected value. Confirm the failure is legible in the output. A green suite you have never seen go red is not evidence of anything.
- Write four sentences you would say in an interview: what you ran, what the config looked like, what surprised you, and what you would reach for instead if the requirement were structured human feedback on production runs.
- Add a row to section 4 of
learning/agent-evaluation/INSTRUMENTATION.mdnaming which tool the eval owner actually uses, since the routing of your correction events depends on the answer and it is not yours to decide.
What this does not cover
No recommendation is made here, and none should be repeated from here. Three vendors describing themselves is not a basis for a preference; the run is. Pricing and ownership were both checked on 5 September 2026 and both are the kind of fact that moves, so re-check before quoting either.
The corrections lesson closes the module and is the one that turns this into design work: what the review gate has to emit for a correction to become an example an eval can run against, and what that requires of the interface. Where those events are allowed to carry the corrected content at all is settled by the never-logged lesson in the instrumentation module, not by this one.
Read this next — primary source
promptfoo — documentationpromptfoo — vendor documentation, fetched 2026-09-05. promptfoo writes this and sells a commercial offering alongside it; the tool itself is MIT-licensed, which is the only reason this lesson can ask you to run it tonight without a signup.
This is the one of the three you can have running against your own data inside an evening, which is the entire point of the lesson. Read the intro and the assertions reference together: the assertions page is where the abstraction becomes concrete, because it lists exactly which checks are deterministic string or schema comparisons and which hand the decision to a model. Seeing that split in a config file is worth more than any explanation of it, including this one. Everything on the site is the vendor describing its own product.
Stuck, curious, or think this lesson is wrong? Ask your teaching agent. The lessons are the scaffold; the conversation is where the learning gets unstuck.