Sample Size

Part of my purpose here at the ThoroughMetrics Blog is to help you evaluate the available research on the thoroughbred industry. As I've said before, the majority of it is flawed in one way or another, and insufficient sample size is one of the most frequent offenders.

One blog that I read fairly regularly is Scot Gillies' Five Cross Files at The Bloodhorse. Scot has a lot of interesting thoughts on the breeding industry, and he's also been very generous with his time, providing me with really detailed feedback on some of the research I'm working on. That said, in his latest post, Scot falls prey to the temptation of basing conclusions on ridiculous small sample sizes. The article is about undiscovered 'breed to race' sires. He does warn the reader that some his selections are lightly bred, and that he's tried to keep the small number of offspring in mind when ranking them. However, he goes on to include Raj Waki and Intidab in his top six. Raj Waki has only 29 starters, but "over 10% stakes winners". Intidab has 27 starters, but "tops 11% black type winners". The problem here is that "over 10%" of 29 is 3, and "topping 11%" of 27 is also 3. A sample size of 3 is not good news, and there is almost certainly no predictive value to the statistics these sires have compiled. We simply have no idea if they're any good or not. When you're thinking about whether a small sample size is meaningful, there's a good 'mental test' you can use. If you removed 1 or 2 from the total, would the result still be impressive? If I remove 2 from 3...well we wouldn't be talking about Raj Waki with 1 out of 29 stakes winnering offspring. When you have small sample size, the results have to be mind blowing to have any predictive value.

In fairness, drawing conclusions based on insufficient sample sizes can be very, very tempting. When I read Scot's article, my first thought was "he should have included Repent instead of one of those others". I made the exact same error as Scot! Repent has less than 100 offspring so far and just a handful of stakes winners. It's too soon to really tell if he's going to be a star at stud.