Hofstadter’s Law: Why Projects Always Take Longer

Where does Hofstadter’s law come from?

Hofstadter introduced the law in 1979, in *Gödel, Escher, Bach*, inside a discussion of computer chess. Writers kept predicting that a machine would hold the world championship within ten years. Every ten years the prediction reset to ten more. He treated that loop as evidence for a “rather recursive” law.

The point is often missed. Hofstadter was not saying estimates are worthless. He was saying the *distance* to a hard goal is misjudged even by people who can see the goal clearly. Computer chess did eventually fall: Deep Blue beat Garry Kasparov in 1997, roughly twenty years after the book. The forecasters were right about the destination and wrong about the length of the road. Most project estimates have the same shape.

Why is the law self-referential?

Because the correction is an estimate too. Suppose you learn that your projects run 40% long, so you add 40% to the next one. You have produced a new number with the same inside-out reasoning that produced the first, and it carries its own unmodelled tail: the specific surprises of *this* project, which no historical average anticipates. That is why “we already accounted for that” is the most dangerous sentence in a planning meeting.

Why doesn’t knowing about the bias fix it?

Because the bias is not a knowledge gap. The planning fallacy was proposed by Daniel Kahneman and Amos Tversky in 1979; Dan Lovallo and Kahneman widened it in 2003 to cover underestimated costs and risks alongside overestimated benefits. People do not underestimate because they have never seen a project slip. They underestimate because they reason about the specific plan in front of them, and a plan is a story about things going roughly right.

The experiments are blunt about the size of the effect. In a 1994 study by Roger Buehler, Dale Griffin and Michael Ross, 37 psychology students estimated how long their senior theses would take. The average estimate was 33.9 days. Asked for a best case, they said 27.4 days. Asked for a worst case, they said 48.6 days. The average actual completion time was 55.5 days.

Read that again: the *worst case* they could imagine was still seven days shorter than the average outcome. A related study asked for dates at 50%, 75% and 99% confidence. Only 13% finished by their 50% date, 19% by their 75% date, and 45% by their 99% date. A 99% interval was wrong more than half the time. People are not merely optimistic about the midpoint. They are optimistic about their own upper bound.

What is the difference between the inside view and the outside view?

The inside view builds an estimate from the components of this plan: task durations, dependencies, known risks, team availability. The outside view ignores the internal detail and asks a different question — what happened to a class of projects like this one?

Kahneman and Tversky concluded that disregard of distributional information is perhaps the major source of forecasting error, and recommended using data from similar past ventures. Bent Flyvbjerg and the consultancy COWI turned that into a documented method for the UK Department for Transport in 2004; it has since been applied in the Netherlands, Denmark and Switzerland, and on the £15 billion Crossrail programme.

The inside view is not stupid. You need it to design the work and sequence tasks. It is simply not a forecasting tool, and asking it to be one is how eight-week plans become fourteen-week projects.

How large is the gap in real projects?

These cases are famous *because* they failed, so the table is not a random sample. It still shows the shape of the error, which is rarely 5% or 15%.

Edinburgh is the most useful case because the forecast was computed and then set aside. In October 2004, Arup reviewed the business case using the newly published reference-class guidelines. It calculated that a budget with an 80% chance of holding was £400 million — a 57% contingency — and that a 50% figure was £357 million. The project’s own budget was £320 million. The analysis was documented, defensible and ignored. Final outturn was £776 million, beyond even the 80th percentile.

> Important note: the lesson is not “projects always double”. It is that the distribution has a long right tail, and a single-point commitment near the middle of it will be missed more often than met.

How do you estimate with the law instead of against it?

Five routines, in order of payoff.

Step 1: Build a reference class before you build the plan

Collect five to fifteen genuinely comparable projects, internal or external. For each, record planned and actual duration, then compute the ratio. You want the median and the 80th percentile, not the average — the mean is dragged up by a few disasters and will over-pad ordinary work.

Step 2: Anchor on the ratio, then adjust for specifics

Start from the reference-class forecast, then list the reasons this project differs: new technology, unfamiliar client, regulatory approval, a vendor with a poor record. Each adjustment should be a sentence someone can argue with. An unexplained multiplier is padding with better branding.

Step 3: Quote ranges with probabilities

Replace “twelve weeks” with “twelve weeks at 50%, sixteen at 80%, twenty-one at 90%”. Whoever approves the work then chooses a risk position consciously instead of inheriting one by accident. That change alone removes most of the arguments that happen in month four.

Step 4: Decompose until the tasks are boring

David Forsyth’s 2008 research on the segmentation effect found that allocating time to sub-tasks reduces planning-fallacy bias, though it demands real cognitive effort. The practical version: if a task cannot be described in a way that lets you picture the work, it is too large to estimate. “Build the reporting module” hides a dozen decisions; “connect it to the billing API and render three sample charts” does not.

Step 5: Put the buffer where it can be defended

Contingency belongs at project level, in a visible line, owned by someone who is not rewarded for spending it. More on that below.

Worked example: a website replatform with a reference class

A digital agency is quoting a replatform. The bottom-up team estimate is 14 weeks. Last year’s comparable builds look like this:

Sorted, the nine ratios run from 1.20 to 1.83. The median is 1.30 and the 80th percentile is 1.75. Applying those to the 14-week estimate gives 18.2 weeks at the median and 24.5 weeks at the 80th percentile.

So the honest framing is roughly 18 weeks as a working plan and 24 weeks as a commitment the agency can defend. Quoting 14 weeks at a fixed price is four weeks short of the agency’s own median and ten weeks short of its 80% case; adding three weeks of slack still leaves it seven weeks short. That is not optimism; it is a decision to absorb the overrun later, usually as unpaid overtime.

Worked example: why the buffer disappears

A 20-week project with a four-week contingency. Two ways to deploy it.

Buffer announced at the end. The date becomes 24 weeks. Work expands toward it, reviews drift later, and the buffer is consumed by ordinary finishing rather than by surprises. The project lands at 24 weeks and everyone says the buffer was needed.

Buffer held by the sponsor, unannounced, released only for approved scope change. The team plans and reports against 20 weeks. When a real surprise lands, four weeks exist and can be granted in days.

The arithmetic is identical; the behaviour is not. A buffer everyone can see and no one owns becomes a deadline.

Which buffer location works best?

The further the buffer sits from the individual task and the clearer its release rule, the better it works. Task-level padding is invisible, gets consumed silently, and destroys comparability. Phase contingency gets released early and then re-requested. Project contingency is read as available budget. Management reserve and a buffer protecting the schedule constraint both hold up, provided someone senior owns the release.

Padding every task feels safer and produces a plan nobody can reason about.

What makes projects slip beyond estimation error?

Estimation error is only one input, and often not the largest. Six others recur:

• Scope growth. Requirements added without re-baselining. An estimate that was accurate in January is wrong by June for reasons no estimator could have priced.
• Queue time. Work waits between people far longer than it takes to do. A five-day task that sits in a review queue for nine days costs fourteen, and only five appear on a timesheet.
• Coordination cost. Fred Brooks observed that adding people to a late software project makes it later. Communication paths grow faster than output.
• Rework. Defects found late cost multiples of defects found early, and rework rarely appears in the original plan.
• Approval latency. Decisions waiting on committees, legal review, or a client on holiday.
• Student syndrome. Work starts at the last responsible moment.

Queue time matters most because it is usually the biggest and the least measured. If you only track effort, you will systematically underestimate elapsed time.

When does Hofstadter’s law mislead you?

The law is a useful default and a poor absolute. Four honest limits:

Blanket multipliers waste money. Repetitive, well-instrumented work — a standard payroll run, a mature support queue — is genuinely predictable. A 1.4 multiplier on work you have done forty times inflates cost and erodes the credibility of every other estimate.

Padding is not honesty. If every task carries a hidden reserve, the plan stops modelling the work and becomes a negotiation document.

Awareness can be gamed. Larry Jones and Kenneth Euske described “strategic misrepresentation” in budgeting: planners with an interest in approval have an incentive to understate effort, because forgiveness for overruns is easier to obtain than permission to start. Hofstadter’s law explains one cause of lateness; incentives explain another.

Late is not the same as failed. The law describes duration, not value. A project that arrives nine weeks late and delivers the revenue case beats one that arrives on time and misses the point. Treating every overrun as a planning crime pushes teams to cut scope quietly.

> Practical tip: Track one number per project — actual duration divided by approved duration — and publish the median every quarter. A team that knows its own ratio argues about evidence instead of whose estimate to trust.

How do you run a project that assumes it will be late?

You need three things: an honest forecast, actuals captured as work happens, and a buffer with a rule.

Quote forecasts at 50%, 80% and 90% confidence with the reference class named. Record actuals per task and per owner as work moves, not from memory at close-out — otherwise your reference class is fiction. Release buffers against written criteria, such as an approved scope change or a documented dependency failure.

That is where tooling matters. If the schedule lives in a Gantt chart, effort in a timesheet system, and the plan in a spreadsheet, planned-versus-actual per task becomes a monthly reconciliation nobody completes. Keeping tasks, owners, due dates, dependencies and progress reports in one workspace shortens the loop from days to minutes — the difference between a reference class that exists and one that is aspirational.

Doitify is built for that layer: Kanban boards and multi-level tasks with checklists and owners, WBS dependencies and Gantt views, workload and resource management, and work and performance reports that show planned versus actual. Transparency note: Doitify is our product, and it does not generate a reference class for you. The historical ratios still have to come from your own completed projects; the software only makes them cheap to collect.

What mistakes should you avoid?

• Treating the law as permission to skip estimation. “It’ll take longer anyway” is not a forecast; it is an excuse to stop measuring.
• Using a percentage nobody can trace. “Plus 30% for safety” without a reference class is a feeling with decimals.
• Forgetting to re-baseline after approved scope change. Then the variance report measures a project that no longer exists.
• Ignoring queue time. The most common reason a well-estimated task arrives late is that it waited.
• Punishing the honest forecast. If the person who quotes the 80% case is overruled and then blamed, next year everyone quotes the optimistic number.

Conclusion

Hofstadter’s law is the shortest accurate description of why project plans miss: the estimate is a story about the work going roughly right, and the story omits the queueing, the rework, the approval that took three weeks, and the requirement that arrived in month two.

The response is not cynicism and not a bigger pad. It is a discipline: collect what similar work actually took, forecast from that distribution, quote a range at a stated confidence, hold the buffer at project level with a written release rule, and record actuals as work happens.

Do that for four or five projects and the conversation changes. You stop arguing about whether the estimate is optimistic — it always is — and start arguing about which risk position the business wants to buy. That is a decision people can make.

If this post on Hofstadter’s law was helpful, you might also enjoy How to Plan Team Capacity (Step-by-Step Guide) and Design Project Management Template: Sections, Example, and How to Use It.