Reading the practice log back
Six timed runs are a dataset about you, not a pile of anecdotes — reading them back finds the bottleneck that is actually yours, which is rarely the one you would have named before you started measuring.
Six rows. A phase table each, a cut list each, a column nobody enjoys filling in. The runs are over and the question is what they are worth, and the honest starting point is that the answer depends entirely on how you read them. Read badly, six runs become six war stories. Read properly, they become the only claim about your own speed you will ever be able to defend.
Start with what a single run is worth, which is not much. One run measures the target, the tool, your sleep and the afternoon. Six runs with the target deliberately varying start to measure the constant across them, and the only constant is you and the kit. That is the whole reason the log wants six and forbids analysing one.
Your prediction about yourself is the thing to distrust
This course opened with METR’s 2025 randomised trial and it is worth bringing back here, with a clear label on what it is doing. In that study, sixteen experienced developers worked on their own mature repositories across 246 tasks. Before starting they forecast AI access would make them 24% faster. Measured, they were 19% slower. After finishing the tasks — having just lived through it — they still estimated they had been 20% faster.
The effect size has since been walked back by METR themselves: the original confidence interval ran from +2% to +39%, barely excluding zero, and a larger follow-up produced intervals straddling zero in both cohorts, which METR attributes to developers declining to submit tasks they did not want to attempt without AI. What survives all of that is the part that never depended on the effect size: the participants’ estimates did not track the measurement, before or after.
This is an analogy, and it is being labelled as one
METR measured AI-assisted coding speed on mature open-source repositories. It did not study prototyping, practice logs, timed runs or bottleneck identification, and nothing in it is evidence about your six rows. This course is drawing a deliberate analogy: competent people, measuring their own performance in their own familiar work, held beliefs their own data did not support. That is the failure mode the read-back has to defend against, and the analogy is the reason to expect it, not proof that it will happen to you.
No source studies reading back a personal six-run practice log, because there is nothing there to study. The log ships empty and the six runs are yours. The method below is this course’s own, and it is presented as a method rather than a finding.
Five reads, in order
Do them in this sequence. Each one narrows what the next has to explain.
1. Level and spread, not just level
Take total time to a demoable state across the six and look at the median and the range together. A falling median with a wide range means the kit helps on some domains and not others, which is a different finding from a kit that helps consistently. For a promise measured in afternoons, the range matters more than the median: an average of two and a half hours is useless if one run in three overran.
2. The recurring phase, not the biggest one
The ranking that decides what enters or leaves the kit is duration multiplied by how often the phase recurs across runs, not duration alone. This was the opening lesson’s point and six rows are the first time you can apply it properly — recurrence was a guess before, and now it is a count. A phase that ate ninety minutes once is noise. A phase that ate twenty minutes six times is the kit’s next item.
3. The cut-list delta
For each run, compare what you cut against what you wrote at T-0. Count the times you cut something that was not on the list. This is the cleanest measurement in the whole log, because both sides were recorded before you knew the outcome, and it measures the exact skill somebody will probe in an interview: whether you can predict, cold, what will have to go.
A shrinking delta across the six is the strongest single piece of evidence the course can produce that the practice worked. A flat delta says your run-planning is not improving, whatever else did.
4. The column everyone leaves blank
What the kit had that got in the way. Six blanks there does not mean the kit is clean; it means the read-back has no evidence about removal, and a kit that only ever gains items becomes the framework this course exists to avoid. If the column is empty, go back through the runs and find the moment you worked around something the kit had already decided. There is always one.
5. Yak-shaving as a trend, not a total
The yak-shaving row is the one people fold into a respectable neighbour, so read it as a line across six runs rather than a number in each. If it is flat while everything else falls, the kit is speeding up construction and doing nothing about the environment — which points at the cold-clone drill, not at another kit component.
Naming the bottleneck
The output of the read-back is one sentence: the phase that is actually yours. It is rarely the one you would have named beforehand — rarely, not never, and the difference matters, because a read-back that goes looking for a surprise will manufacture one.
The reason it tends to shift is structural rather than mysterious. The phase you would name is the one you remember disliking, and irritation scales with friction rather than with minutes. The phases that quietly eat the clock are the ones that feel like progress: reading yourself into a domain, deciding what the surface should show, rehearsing the walkthrough. None of those feel like being stuck, and all of them are recorded in the table.
Write the finding with its evidence attached, in this shape, so it survives being repeated:
Across six runs, [phase] took [range] minutes and appeared in [n] of 6.
I predicted [other phase] before starting.
The kit change this justifies is [add / remove one item], because
[what it removes from the next run's clock].What six runs cannot tell you
State these limits before anyone else does, because they are real and because volunteering them is what makes the rest credible.
- The sample is six and the design has no control. No effect can be attributed to the kit with any confidence. What the log supports is a description of where your hours went, not a measured improvement.
- The targets differ deliberately. That is what makes the runs about you rather than about one product, and it is also what prevents any clean run-to-run comparison.
- You improved at the course, not only with the kit. By run six you are better at cutting, at scoping and at the demo narrative. Those gains are real and they are not the kit’s.
- You chose the targets. Selection runs through everything here, exactly as it did in the study METR had to redesign.
None of that makes the log weak. A dated, specific, self-limiting account beats a confident number every time in a room that knows what a confident number is worth.
Check your recall
Answer from memory — no scrolling back.
Retrieval check
After the read-back you are asked: “so how long does an agentic prototype take you?” What is the answer that holds up?
Check your answer
A range, a count and a limit. Six timed runs across six unfamiliar domains, a demoable surface in every one of them inside three hours — or in however many of them it actually was — with the median and the worst case both named, and the cut list for each recorded before the clock started.
Then the limit, unprompted: six runs, no control, targets you chose yourself, and part of the improvement is you getting better at scoping rather than the kit getting better. Saying that does not weaken the answer. It is the thing that makes the rest of it believable, and it is the difference between a measurement and a claim.
Hands on
The read-back
Done when: PRACTICE-LOG.md ends with a Read-back section containing all five reads, one named bottleneck with its evidence, one kit item to add and one to remove, and a written statement of what six runs cannot show.
- Before opening the log, write down which phase you believe is your bottleneck. Seal it at the top of the section. Doing this afterwards is worthless, for the reason the METR analogy exists.
- Read one: total time to demoable across the six, median and range together. Write both. If any run overran three hours, say so in this line rather than in a footnote.
- Read two: for each phase, total minutes and how many of the six runs it appeared in. Rank by the product of the two. This table is the section’s core.
- Read three: count the cuts you made that were not on the pre-written list, per run, and say whether that count fell across the six.
- Read four: fill any blank in what the kit had that got in the way by going back through the runs. Then name one kit item to remove. One is the minimum, not a target.
- Read five: plot the yak-shaving row across the six. If it is flat, book a cold-clone drill rather than adding a component.
- Write the bottleneck sentence with its evidence, compare it to the sealed prediction, and update the kit manifest with the one addition and one removal. Bring both into the chat — I will push hardest on a bottleneck named without a count behind it, and on an addition that appeared in only one run.
What this does not cover
This is the last lesson, so what follows it is not another lesson. The read-back’s output is a change to the kit, and the kit manifest is the page this course leaves you with: every piece, why it is in, what it costs to keep current, and which decision it removes from the clock. Keep it open, and treat it as the thing that changes after every run.
Two questions the log will raise and this course deliberately does not answer. What a trace view should actually show — how to render a tool-calling loop so a human can audit it — belongs to the observability course; here you wired an existing one in and never designed it. And every run in this log was standalone and disposable, which is what made three hours possible. Landing an agentic surface inside someone else’s production codebase is a different problem with a different clock, and nothing you measured here transfers to it unexamined.
Read this next — primary source
Measuring the Impact of Early-2025 AI on Experienced, Open-Source Developer ProductivityMETR, 10 July 2025 — free; read alongside METR’s February 2026 update, which weakens the headline result
You read this at the start of the course for the perception gap. Read it again now for a different reason: the long section where the authors enumerate candidate explanations for a result they did not expect and rate each one for how much of the effect it could account for. That is the exact task in front of you with six filled rows and a phase you cannot explain. It is also a demonstration of the discipline this lesson needs most, which is separating what a small, uncontrolled dataset shows from what you would like it to show. This study is used here as an analogy about prediction and measurement, not as evidence about practice logs — it studied neither prototyping nor practice.
Stuck, curious, or think this lesson is wrong? Ask your teaching agent. The lessons are the scaffold; the conversation is where the learning gets unstuck.