Where the hours actually go
You cannot cut what you have not timed, and every engineer guesses this wrong — so the first prototype in this course is instrumented with a stopwatch rather than improved, because a measured breakdown is the only thing that tells you which hour is worth removing.
You have built the review gate before. HouseWarm’s approve-or-correct step — per-field confidence, the source image crop, the raw OCR text sitting next to the extracted value, nothing downstream firing until a human says yes — is a real, shipped thing. You have also built an advisory chatbot that takes a request, waits, and renders an answer. Neither is a mystery to you.
The Vista job is not asking whether you can build those. It is asking whether you can build one again, against an unfamiliar portfolio-company domain, starting from an empty directory, this afternoon. And the honest answer today is that you do not know, because you have never timed it. You have shipped under compressed timelines plenty of times, but a compressed timeline is not a measurement — it is a deadline you met, with no record of what it cost to meet.
So the first thing this course does is not build anything. It runs a stopwatch. Not because measurement is virtuous, but because you cannot cut what you have not timed, and the phases engineers cut on instinct are reliably the wrong ones.
Engineers are bad at estimating their own speed, provably
This is not a character claim, and it is worth grounding rather than asserting. In 2025 METR ran a randomised controlled trial on sixteen experienced open-source developers working on their own mature repositories — repos they averaged five years of history with — across 246 real tasks, with AI tool access randomised per task. The tooling was Cursor Pro with Claude 3.5 and 3.7 Sonnet, over February to June 2025.
Three numbers from that study, in the order they were produced:
- Before starting, developers forecast that AI access would make them 24% faster.
- Measured, AI access made them 19% slower — “they take 19% longer to complete issues”.
- After finishing — having just lived through the slowdown — they estimated AI had made them 20% faster (arXiv:2507.09089).
The third number is the one this lesson is actually about. The first two are a story about AI tools. The gap between the second and the third is a story about self-report: these were competent, experienced people, in their own codebases, asked immediately afterwards, and they were wrong about their own throughput by roughly forty percentage points in a direction they could not feel.
Do not turn this into a punchline
The “19% slower” figure travels well and is routinely misused, so here is what it does not support. METR’s own write-up lists the claims it explicitly does not provide evidence for, including that “AI systems do not currently speed up many or most software developers”. They describe their result as a snapshot of one setting, and note it is consistent with greenfield projects and unfamiliar codebases seeing substantial speedup — which is precisely the setting a three-hour prototype lives in.
It has since weakened further. In a February 2026 update, METR reported the original effect’s confidence interval as +2% to +39% — barely excluding zero — and published a larger follow-up whose intervals straddle zero in both cohorts. Their stated reason is selection bias: 30% to 50% of developers told them they were declining to submit tasks they did not want to attempt without AI, systematically removing the highest-value cases from the sample. METR is redesigning the experiment. Citing the 19% as settled fact today is citing a number its own authors have qualified.
What survives all of that qualification is the part this lesson needs, and it survives because it does not depend on the effect size at all: the developers’ own estimates were uncorrelated with the measurement, before and after. Even in the redesigned follow-up, the thing being argued about is what the true number is — not whether people can feel it. Nobody in this literature thinks they can.
Belief is near-universal; measurement is contested
The 2025 DORA report — State of AI-assisted Software Development, from a survey of nearly 5,000 technology professionals — puts the same gap at industry scale. It reports that 90% of respondents use AI at work and more than 80% believe it has increased their productivity, while 30% report little or no trust in the code it generates. On outcomes, DORA finds a positive relationship between AI adoption and delivery throughput — a reversal of their own 2024 finding — alongside a persistent negative relationship with delivery stability.
Two honesty notes on that source. DORA is published by Google Cloud, which sells AI coding tools — a vendor publishing research about its own market, and worth reading with that in mind even though the methodology is public. And the pages that are freely readable give the direction of those relationships without effect sizes, so no percentage should be attached to the throughput or instability claim.
Put the two sources next to each other and the shape is clear. Belief about personal productivity is overwhelming and self-reported. Measurement of it is contested, effortful, and keeps producing results that surprise the people being measured. Your intuition about which part of a prototype eats your afternoon is exactly that kind of self-report.
What to measure, and the trap in how you measure it
A baseline run is one agentic prototype, built from an empty directory, with no kit, against a three-hour wall clock, with phase times written down as they happen. Phases worth splitting, because they fail for different reasons:
Understanding the problem
Project scaffold + a running app
Design system / styling wired
Auth stub
Fake agent backend
Streaming / run-state plumbing
Trace view
Review gate
The actual domain surface
Demo polish + walkthrough prep
Yak-shavingThree rules make the resulting numbers worth anything, and each one exists because of a specific way this measurement fails:
- Record live, not afterwards. Reconstructed times are not noisy, they are biased — systematically inflated toward whichever phase you found most irritating. That is the METR gap reproduced at the scale of a single afternoon.
- Wall clock, not effort. The thing being claimed in an interview is “this afternoon,” and an afternoon contains interruptions, a stuck install, and the ten minutes you spent reading a changelog. Those are part of the number.
- Give yak-shaving its own row and fill it in honestly. It is the row people fold into whichever adjacent phase looks more respectable, and it is frequently the largest one.
The phase that felt slowest is not the phase to cut
Once the numbers exist, there is one more inversion to get past. The obvious move is to attack the largest measured phase. The correct move is to attack the largest value of duration multiplied by how often it recurs across future prototypes, which is not the same ranking.
A ninety-minute fight with a specific domain model is ninety minutes you will never spend again, because the next portfolio company has a different domain. Twenty-five minutes wiring a trace view is twenty-five minutes you will spend on every single prototype for the rest of the role. The domain fight is the bigger number and the worse target. The trace view is the smaller number and the thing that belongs in the kit — and you would not have known which was which without the measurement, because the domain fight is the one you will remember being painful.
Check your recall
Answer from memory — no scrolling back.
Retrieval check
Someone at Vista asks: “can you really do an agentic prototype in an afternoon?” What makes that answer credible rather than aspirational?
Check your answer
Not confidence, and not a portfolio of finished work — finished work proves you can build the thing, not that you can build it inside three hours from cold. What makes it credible is a measurement: a named number of timed runs, against different domains, with a record of what got cut in each and where the clock actually went.
That is also the answer that survives the follow-up question, which is always some version of “what did you have to leave out?” A person who has measured has a cut list. A person who has not has an adjective.
Hands on
The baseline run
Done when: PRACTICE-LOG.md contains a completed Baseline run section — a filled phase table with live-recorded minutes, an honest yak-shaving row, and one named phase you would cut on recurrence rather than on size.
- Pick a target you have never built: an agentic surface for a domain you do not know, with at least one step a human would have to approve before it fires. A vendor-invoice approver, a support-ticket triager, a lease-abstraction reviewer — something portfolio-company shaped, not a chatbot.
- Start from an empty directory. No kit, no copying from HouseWarm, no starting from this repo. The point of the baseline is that it is the from-scratch number the rest of the course is measured against, so contaminating it with a head start destroys the comparison.
- Open
learning/prototyping-velocity/PRACTICE-LOG.md, copy the phase table into a new Baseline run section, and put a clock where you can see it. Write each phase’s minutes down as you leave that phase — not at the end. - Stop at three hours whether or not it works. An overrun is data about the estimate; a rescue is data about nothing. Record whether it was demoable, and what “demoable” meant in this case.
- Fill the yak-shaving row honestly, including the parts that are embarrassing — the wrong Node version, the CSS that would not load, the twenty minutes reading a migration guide. This row is where the kit’s value usually turns out to be hiding.
- Under the table, name one phase you would cut based on recurrence, and separately name the largest phase by duration. If they are the same phase, say why you believe it will recur. Bring both into the chat and I will push on any candidate that looks like it is really a domain decision in disguise.
What this does not cover
A measured baseline tells you which hours exist. It says nothing about which of them are safe to pre-decide, and that is the harder question — a kit built by simply removing every phase that took a long time will encode decisions the next prototype needed to make for itself, and will be slower than no kit at all. Choosing what is allowed into the kit gets its own lesson, on decisions you have genuinely already made, immediately after this one.
Timing also says nothing about the second half of the problem: the agentic behaviour itself. Every phase in that table assumes something on the other end returns data, and standing up a real agent graph is far and away the slowest way to get one. That is what the fake-backend module solves, and why the practice runs proper do not start until it exists.
Read this next — primary source
Measuring the Impact of Early-2025 AI on Experienced, Open-Source Developer ProductivityMETR, 10 July 2025 — free; the paper is at arXiv:2507.09089, also free
This lesson takes one number and one gap from it. The write-up itself spends most of its length on the part worth your time: a long, unusually candid section listing what the result does not show, and a set of twenty-odd candidate explanations for the slowdown which the authors rate individually for how much of the effect each could account for. That section is a working model of how to reason about a measurement you did not expect, which is exactly the position you will be in after your own baseline run. Read it alongside METR’s February 2026 update, which walks back how strong the evidence is and explains why.
Stuck, curious, or think this lesson is wrong? Ask your teaching agent. The lessons are the scaffold; the conversation is where the learning gets unstuck.