Can story points measure the growth AI gives you?

One of the companies I worked at leaned on AI heavily across the whole Software Development Life Cycle (SDLC). Using that much AI is a form of investment, so the CEO asked for productivity to improve by 10% every quarter — and productivity, on the R&D (Research and Development) side, was measured in story points (SP). Whether the story point is the right way to measure productivity is a different argument and this piece will not have it; assume it is. The question here is narrower: is “10% growth every quarter” something you can actually establish?

Velocity and the basic arithmetic

In software the word “productivity” usually turns into the word “velocity ”, and velocity is measured as the number of SP finished in one sprint . Productivity over a quarter, then, is many sprints — which means it is not a single number at all. It is a series.

Here is Q1, as logged:

38   45   31   42   36   48

What can we pull out of that?

The mean is ${\mu }_1=40$, and yet not one sprint was 40. The gap between the best sprint and the worst is $48-31=17$ SP, and nothing on the outside changed across those twelve weeks: same people, same codebase, same way of working.

So the first thing to accept is that velocity swings even when nothing is happening. Not big changes — small ones. Somebody was off sick. A story turned out easier than it looked, or harder.

That swing has a size, and the sample standard deviation is how you get at it: ${\sigma }_1=\sqrt{\frac{1}{N-1}{\sum }_{i=1}^N(x_i-{\mu }_1{)}^2}$ With $N=6$ and the series above, ${\sigma }_1=6.23$, and the sample variance is $s_1^2=38.8$. A lot of numbers, I know. The short version: with nothing whatsoever going on, a sprint can land 6.23 points off the mean.

One more number falls out of the same data, the coefficient of variation $c$: $c_1=\frac{{\sigma }_1}{{\mu }_1}=\frac{6.23}{40}=0.156$ which says that this team’s ordinary swing is about 15.6% of its own average. As teams go, that is a steady one.

Comparing two series

The question at the end is not any of the numbers above — trust me, they matter — it is a comparison between two series.

Q2 comes in, and the log now reads:

Q1: 38  45  31  42  36  48   →  avg: 40.0
Q2: 41  49  36  47  40  51   →  avg: 44.0

Q2’s average is 4 points above Q1’s. Exactly 10%. Time to celebrate?

Celebrate first, sure. But then: if nothing at all had acted on this team, AI included, could a 4-point gap have shown up anyway?

The usual way to compare two series is to compare their means, which is what we just did — and a mean carries an error of its own, the standard error of the mean : $S{E}_1=\frac{{\sigma }_1}{\sqrt{N}}=\frac{6.23}{\sqrt{6}}=2.54$ What that number says is this: if you had the power to re-run Q1 in some other timeline, same team, same conditions, the mean ${\mu }_1$ (40) coming out 2.54 points different would be an ordinary Tuesday.

So if we could re-run both quarters in another timeline, how far apart could the two means land?

● Figure 2 re-run the quarter

run it again run 10 times clear

You have re-run it 8 times, in a world where nothing changed at all — no AI, no improvement, nothing. 2 of those came out +4 points or more.

With $S{E}_2=\frac{{\sigma }_2}{\sqrt{N}}=\frac{5.87}{\sqrt{6}}=2.39$,

$SE=\sqrt{S{E}_1^2+S{E}_2^2}=\sqrt{2.54^2+2.39^2}=3.49$

Which gives us something to say: with nothing changed at all, two quarterly means drifting 3.49 points apart is normal. The gap we observed is 4 points, and noise alone reaches 3.5. The quick conclusion is the honest one — there is not enough here to say whether AI helped, because what we are holding is barely distinguishable from noise.

How big does a difference have to be?

Before the arithmetic, one more thing about that $SE$: it is also the width of the bell the noise is spread along. Go back to re-running both quarters in another timeline — do that endlessly and about 70% of the time the difference lands between -3.49 and 3.49 points.

Figure 1 the noise distribution

To detect anything and mean it, there are two ways to be wrong, and both have to be guarded against.

Type I error

A type I error is a false alarm: nothing changed, but noise happened to throw up a gap wider than the ordinary one we just measured (3.49). To guard against it, we allow only a 5% chance of noise clearing the bar — and on a normal curve that point sits 1.96 widths out from the centre. So the bar is $1.96\times 3.49=6.84\approx 7$, which means: under 7 points, say nothing.

Type II error

A type II error is a miss: the improvement was real, but noise dragged the number below the bar and it got read as “nothing happened”. Concretely — if the real improvement is exactly the 7 points we just computed, then across endless re-runs you catch it half the time and lose it the other half. To push the catch rate to 80%, the real improvement has to clear 7 by a margin: on a bell, shifting the centre 0.84 widths further puts 80% of the distribution to the right of the bar.

So what is needed is ${\Delta }_{min}=(1.96+0.84)\times SE=2.8\times 3.49=9.77\approx 10$

Which is to say, a gap of at least 10 points before it counts as improvement.

● Figure 3 drag it

Suppose the real improvement is: 6.8 the bar80%

noise, when nothing has changedif the improvement is real

If the real improvement is exactly 6.8 points, then across endless re-runs you would call it an improvement about 50% of the time. The shaded tail is that share.

So how long does it take?

Let me restate the question. We know a gap has to reach at least 10 points before it counts, and we collected $N=6$ per quarter. The new question is “how many sprints (N) does it take to establish that the team improved by 10%?”

Generalising SE

The $SE=3.49$ from earlier is one specific number for $n=6$, not a formula. Assuming the two series are always the same length, it can be written out:

$SE=\sqrt{S{E}_1^2+S{E}_2^2}=\sqrt{\frac{{\sigma }_1^2}{n}+\frac{{\sigma }_2^2}{n}}=\sqrt{\frac{{\sigma }_1^2+{\sigma }_2^2}{n}}=\sqrt{\frac{2}{n}\times \frac{{\sigma }_1^2+{\sigma }_2^2}{2}}={\sigma }_{pooled}\times \sqrt{\frac{2}{n}}$

where

${\sigma }_{pooled}=\sqrt{\frac{{\sigma }_1^2+{\sigma }_2^2}{2}}=\sqrt{\frac{38.8+34.4}{2}}=\sqrt{36.6}=6.05$

Working the threshold backwards

We also have ${\Delta }_{min}=2.8\times SE\approx 10$ as the point where a gap counts. Substituting the formula for $SE$ gives an inequality:

$\Delta \ge 2.8\times SE=2.8\times \sigma \sqrt{\frac{2}{n}}$

In words: a real improvement has to be at least 2.8 times the noise of the comparison.

Now rearrange it for $n$.

$\Delta \ge 2.8\times \sigma \times \sqrt{\frac{2}{n}}\iff {\Delta }^2\ge 2.8^2\times {\sigma }^2\times \frac{2}{n}\iff {\Delta }^2\ge \frac{15.7\times {\sigma }^2}{n}\iff n\ge \frac{15.7\times {\sigma }^2}{{\Delta }^2}$

That is a formula for $n$, but not yet one in percentages. To get there, divide top and bottom by ${\mu }^2$ — the square of the series’ mean.

And $\sigma /\mu =c$ is the team’s coefficient of variation. So what does the formula say?

The numerator says how much noise, in percent, this team brings. The denominator says how much improvement, in percent, you want to catch.

With the pooled $c$ across both quarters, $c=\frac{{\sigma }_{pooled}}{{\mu }_{pooled}}=\frac{6.05}{42}=0.144$, and a 10% improvement to catch, so $\Delta /\mu =0.1$, $n$ has to be

A quick check

Put $n=33$ back into the formula for SE: $SE=\sigma \times \sqrt{\frac{2}{n}}=6.05\times \sqrt{\frac{2}{33}}\approx 1.49$. That gives ${\Delta }_{min}=2.8\times SE=2.8\times 1.49=4.17$, and $4.17/{\mu }_{pooled}=4.17/42=0.099=9.9\%$. Good — the arithmetic holds.

So it takes at least 33 sprints per arm, meaning 66 sprints, before you can catch whether something has lifted productivity by 10%.

Conclusion

Back to the CEO’s question: “productivity has to improve 10% every quarter”. The answer is not “yes” or “no”. It is something more uncomfortable: with one quarter of data, that question has no answer at all.

Three numbers from this piece, side by side, say the whole thing:

• The team already swings 15% around its own average when nothing is happening.
• The gap to reach is 4 points; the noise of the comparison is 3.49 points. What we need to measure is barely distinguishable from pure noise.
• To tell those two apart takes 33 sprints per arm — with two-week sprints, two and a half years of data for one conclusion.

Here is the paradox. The review cycle is a quarter, six sprints, while the measure being used to do the reviewing needs ten times that before it can conclude anything. Asking for “10% a quarter” is not wrong as an expectation. It is just not verifiable in story points — and an unverifiable demand gets answered each quarter by instinct, by a number that happened to look good, or by whoever was loudest in the meeting.

If you still want to measure it, the formula $n\ge \frac{15.7\times c^2}{(\Delta /\mu {)}^2}$ leaves exactly three knobs:

1. Cut $c$ — make the team less noisy. $n$ scales with $c^2$, so the steadier the team the less time it takes: pull the swing from 15% down to 7.5% and the wait drops to a quarter of what it was. Stories split small and evenly, estimates kept consistent, no changing the scale halfway through. This turns out to be the most valuable work on the list, and it has almost nothing to do with AI.
2. Accept a bigger target. $n$ scales inversely with $(\Delta /\mu {)}^2$, so hunting a 20% improvement costs a quarter of what hunting 10% costs. A genuinely large effect is easier to prove than a small one is to evidence.
3. Widen the observation window. Stop asking “did we grow 10% this quarter”; ask “after a year and a half, has the trend actually turned”.

And the last one, which matters most: none of the numbers above conclude anything about whether AI is useful. They say story points are not sensitive enough to serve as evidence for the question being asked. Those are two entirely different statements — and in an end-of-quarter review they get treated as one.

● Figure 4 your team's numbers

Series 1: Series 2:

Improvement to catch: 5%10%20%

sprints to observe, per arm

variability	7.5%	10%	15%	20%
5%	36	63	142	251
10%	9	16	36	63
20%	3	4	9	16

Your team swings 14.4% around its mean. The observed difference is +4.0 points against a bar of 9.8 points — not enough to say anything yet. To catch the level you picked, you need 33 sprints per arm, about 30 months.