What Makes a Test Repeatable

You rowed 6:42 last night and 6:51 this morning. Same erg, same damper, same warm-up. The question you actually have is not whether the second piece was worse. It is whether the difference means anything, and if it does not, what to change. There is a literature for that, and it is more useful than most coaching folklore about testing.

The context

A rowing test is a measurement, and a measurement carries two error sources: the machine and the person on it. Get those apart and the rest becomes arithmetic.

Start with the machine. A Concept II ergometer instrumented for stroke-by-stroke power came out of calibration with measurement errors of less than 3 per cent, and when seven experienced oarsmen repeated a 90-second maximal test, only two of the fourteen variables used to describe rowing performance failed the reliability criterion ([2] Macfarlane 1997). The erg is not your problem. It is the most controlled object in the room.

Start with the person, and the useful number comes from the best-designed study of the 2k itself. Eight well-trained rowers rowed a 2000 m time trial of roughly seven minutes on a Concept II on three occasions three days apart. The coefficient of variation for mean power was 2.0 per cent, the retest correlation was 0.96, and the variability of the finish time was about one third as variable as that ([1] Schabort 1999). Read that twice. The finish time is the less variable of the two measures, and it is the one that sets your corridor: one third of the 2.0 per cent power figure is about 0.7 per cent of your finish time, which on a seven-minute trial is roughly three seconds either side of your true time rather than a minute.

What the 2k actually measures is worth knowing, because it tells you what a change in the number means. In finalists from World Championship events, the strongest correlates of 2k ergometer speed were power at maximal oxygen uptake, maximal power and maximal force, all at r = 0.95, with maximal oxygen uptake at r = 0.88 and oxygen uptake at the blood lactate threshold at r = 0.87. Together those variables explained 98 per cent of the variance, and when the model was validated on 18 more elite rowers the limits of agreement on time ran from -1.5 to 6.9 seconds ([8] Ingham 2002). A 2k is a composite of aerobic and anaerobic power. It is not a fitness score, and it is not a race result.

The core idea

The machine is repeatable. You are not. Everything else in this article is a consequence of that.

The ergometer's own contribution to the error is small and bounded, and the machine upgrade that arrived part-way through most rowers' careers turned out to be almost cosmetic. In eight national-level rowers tested on the older model C and the newer model D with standardised drag factor settings, power, oxygen consumption, rowing economy, heart rate, blood lactate, stroke rate and perceived exertion all looked the same, with one small significant difference in blood lactate at the lactate threshold of 0.2 mmol per litre. The authors concluded that testing can be carried out on either ergometer and the results compared directly ([5] Vogler 2007). If you have changed machines, your history is still your history.

What is not a machine variable is the resistance you pull against, and this is where indoor rowers most often fool themselves. In twelve club rowers on an air-braked ergometer rowing at a drag factor of 100 or 150, the low setting produced significantly greater cadence and heart rate, while rowing economy was significantly greater at the high setting ([6] Kane 2013). The damper does not cancel out. It changes what the piece asks of you, so a damper change is a protocol change, not a piece of administration.

Then there is the number you cannot take to the water. In a Great Britain training camp, 49 junior elite male rowers completed both a 2000 m Concept II model C test and a 2000 m single-scull test on the water. Water speed was 3.66 m per second against 4.96 on the erg, and the two performances shared only 28.9 per cent of their variance ([4] Nevill 2010). Body mass makes the point cleanly: it correlated with erg performance at r = 0.68 and with water performance at r = 0.039 ([4] Nevill 2010). A repeatable erg test is a repeatable measurement of something that is not quite boat speed.

One last piece of the machine, and it is the piece that tells you what protocol to run. Fifteen national standard rowers did five 500 m races and two 2000 m races on both a Concept II and a RowPerfect ergometer. Over 500 m the standard error of mean power was 2.8 per cent on the Concept II and 3.3 per cent on the RowPerfect. Over 2000 m it was 1.3 per cent on the Concept II and 3.3 per cent on the RowPerfect, and the authors named 2000 m races on a Concept II as the most appropriate protocol for monitoring a rower's ability to produce power ([3] Soper 2004). The long piece on the standard machine is the most repeatable thing available to you.

The how-to

1. Write the protocol down before you chase the number. Machine, damper setting, distance, warm-up, target stroke rate, pacing plan and the hour of the day. The statistical floor for a usable reliability estimate is about 50 participants and at least three trials, which tells you something about how few trials you can afford to waste: the same discipline that governs a published reliability study governs your own logbook ([7] Hopkins 2000).

2. Treat attempt one as a familiarisation piece, not a calibration. In the eight-rower reliability study, mean power improved by 2.3 per cent in trial two, with a confidence interval of 0.1 to 4.5 per cent, and by a further 0.9 per cent in trial three ([1] Schabort 1999). That learning effect is not a rounding error, and it is why the statistics of reliability insist that systematic changes in the mean between consecutive trials, whether from learning, motivation or fatigue, have to be removed before you can call the remaining spread random variation ([7] Hopkins 2000). Your first 2k of the season is a rehearsal.

3. Fix the clock and stop shopping for a better hour. Sports performance is affected by time of day in normal entrained conditions, and the variation has at least some input from internal biology, though the relative weight of internal and external causes is not known ([12] Drust 2005). There is no honest number to tell you whether your evening is faster than your morning. Pick an hour you can keep and never move it.

4. Log what you cannot control, and do not test on a bad day. Sleep is the hardest one, because the evidence is equivocal: some maximal efforts and gross motor performances survive sleep loss, but the few studies in athletes report reduced sport-specific performance, and reduced sleep quality and quantity can produce an autonomic imbalance that mimics the symptoms of overtraining ([10] Fullagar 2015). Illness and hard training blocks are easier. A layoff moves you too, though more slowly than you would expect. After 84 days without training, heart rate, ventilation, respiratory exchange ratio and blood lactate at the same absolute submaximal intensity were all still up, mitochondrial enzyme activity had fallen by 40 per cent, and the lactate threshold still sat at 75 plus or minus 2 per cent of maximal oxygen uptake against 62 plus or minus 3 in untrained controls ([9] Coyle 1985). A test after a layoff is not a measurement of anything you have done lately.

5. Know what a moved number means before the number moves. The consensus statement on overtraining separates functional overreaching, a performance decrement without severe psychological or lasting other symptoms that resolves into an improvement after recovery, from non-functional overreaching and overtraining syndrome, a distinction that is very difficult and depends on the clinical outcome and on excluding organic disease, negative energy balance, insufficient carbohydrate or protein, iron deficiency, magnesium deficiency and allergies. Fatigue, performance decline and mood disturbance are generally more severe in overtraining syndrome, though the statement is explicit that there is no scientific evidence to confirm or refute that, and none of the available markers, hormonal, performance-based, psychological or biochemical, meets all the criteria for general acceptance ([11] Meeusen 2013). So a single slow 2k is not a verdict. It is one data point that needs at least two more before it means anything.

6. Read the number against the right yardstick. The typical error, which is the standard deviation of an individual's repeated measurements and is best expressed as a percentage of the mean, is the measure that tells you how big a change has to be before it means something. The limits of agreement describe the 95 per cent likely range of change between two trials, and retest correlation is genuinely hard to interpret because its value depends on how similar the participants are ([7] Hopkins 2000). Compare consecutive comparable trials, or a rolling window of them under the same protocol and conditions, rather than your best-ever time against your most recent one: the typical error and the limits of agreement are estimated from comparable repeated measurements, and comparing the best of a sample with a later draw tells you about selection, not about test error.

The trade-offs

Three honest limits. First, the rowing-specific literature is small. The 2k reliability work is eight well-trained rowers, the ergometer-versus-water work is 49 junior elite men, the ergometer-type comparison is fifteen national standard rowers, and the drag factor work is twelve club rowers. These are enough to size a corridor and not enough to settle a question. The Concept II dominates the field because it is the machine most clubs own, not because it was shown to be best; over 2000 m it produced a standard error of 1.3 per cent while the RowPerfect produced 3.3 per cent on the same distance ([3] Soper 2004).

Second, these are laboratory protocols, not races. The physiological model that predicts a 2k from aerobic and anaerobic power was validated on 18 elite rowers with limits of agreement running from -1.5 to 6.9 seconds ([8] Ingham 2002), and a piece that is seven minutes of maximal effort tells you about a capacity rather than about how you will handle a start line with three hundred other people in it.

Third, some of the inputs have no number attached at all. Nobody can tell you whether your 5 pm is better than your 7 am ([12] Drust 2005). The sleep literature is explicitly unresolved on how much sleep loss costs you and by what mechanism ([10] Fullagar 2015). And the measurement statistics are subtler than a percentage, because a typical error and a limit of agreement answer different questions and only one of them is unbiased ([7] Hopkins 2000).

What to do tomorrow

1. Write your protocol on a card next to the erg: machine, damper setting, distance, warm-up, target rate and the hour you test. Change one thing a season, not one a test.
2. Take three pieces before you believe a number, and count the first one as rehearsal.
3. Write down your sleep, your health and anything unusual on the day, next to the split.
4. When a result moves, read the log before you read the training plan.