Your Retention Rate Dropped. Or Did It?
Why a single year can’t answer that question – and what three simulations show about how long it actually takes to know your retention rates are changing.
Retention rates are make-or-break numbers for most schools. A two-point gain is a great year, while a two-point loss can mean a hiring freeze or budget cuts. So, it’s understandable that leaders want an answer to the question “Are we improving our retention rates?”, and use that answer to guide their decisions.
The problem is that retention rates rarely give a clean answer. Even setting aside a shock like COVID, they tend to bounce from year to year: up a point, down two, up two. If you work with these numbers, you’ve probably noticed this, and you may have suspected that chance, not just effort, plays a part.
What I hope to demonstrate in this article is that chance plays a bigger and more distorting role than is often appreciated. For example, a school’s retention rate can swing several points in a year when nothing real has changed. Meanwhile, an initiative that genuinely works may not improve retention every year, and may even show declines in certain years.
These distortions can cause ripple effects in decision-making, leading institutions to cut initiatives that work, invest in ones that don’t, and generally focus on artificial, short-term volatility over stable, long-term trends.
The three interactive simulations below are meant to illustrate these effects and to encourage data-driven institutions to view each new retention rate with a more patient, longer-term perspective.
Randomness Is More Important than You Think
The most important concept to take from this article is that a college with the same “true” retention rate, its actual ability to retain students, will have a different retention rate from year to year due to random chance. While this is obvious at a rational level, it’s easy to underestimate, intuitively, how much chance alone can move the number.
Below is a quick experiment to see it. Imagine one college with 1,000 new students whose true retention rate is fixed at 78 percent and never moves; the only thing that changes from year to year is the luck of the draw. Now simulate that one year over again, 20 times. Try it yourself:
One year, twenty times
The true rate is exactly 78% every single time. Each dot is a reported rate for the identical year.
Now, consider the question: which one of the 20 dots is the “true” retention rate? It is the vertical dotted line: the yearly average over the long run. However, no institution ever actually gets to observe it. The institution only gets to report one of them, and that one number becomes the official retention rate that everyone treats as the truth.
In the above simulation, with an entering class of 1,000, 20 versions of the same year can vary about four to six points. You can set the simulator to your school’s own parameters and try it yourself. In many cases, an ordinary two-point swing up or down - the kind that could trigger emergency meetings or task forces - sits comfortably inside that range.
Real Change Is Usually Small
Another point is that genuine improvement or decline in retention usually comes in small amounts, often smaller than a single year of data can reliably detect.
Consider COVID-19, a generational disaster that upended every campus in the country. For most schools, it knocked retention down only a point or two in a single year.
In practical terms, if your retention truly did improve or decline in a real but small way, as is the case for most schools, there is a good chance you will not see it immediately. Instead, your real retention gain can hide inside the noise for years. And the reverse happens just as easily, with an ordinary year of luck presenting itself as progress or loss that was never there.
The simulation below shows a college whose true retention rate really did rise one full percentage point this year, and shows how likely an up or down year is in that situation.
One real improvement, twenty times
This college’s true retention rate really did rise this year – by exactly the amount you set, from a starting point of 78%. The simulation rolls that same one-year change 20 times; each dot is the change the college would actually have reported.
In this case, ask yourself: how does a school in this scenario know if it is truly improving? Run it a few times and the pattern is clear: real changes do not always produce positive results. In this particular case, a school with 1,000 entering students that truly improved retention by one percentage point, a small but meaningful feat, will still report a lower rate than the year before roughly 3 times in 10.
Small Changes Take Years to Trust
So if one year can’t be trusted, how many years do you need instead? We can test that through simulations, as well. The bottom line is that real changes can take several years or more to truly be seen.
The simulation below starts with an institution whose initiative worked: its “true” retention rate improved by anywhere from 1 to 5 percentage points (you can set how much in the simulator). However, the institution doesn’t know the “true” impact of the initiative; that would be impossible to know. Instead, it needs to judge the initiative’s success by what it can see: the actual observed retention rate changes each year.
In the simulation, you can decide how much of an improvement the institution needs as evidence to declare the initiative a success. The question is: how many years of data does it take for a school to reliably see that improvement, and how often does it see too little improvement (or none at all) and conclude that the initiative failed?
Will the numbers catch a gain that really happened?
Every simulated college here truly improved by the amount you choose. The only thing standing between the college and the truth is ordinary luck: which students happen to come back each year.
Take the example shown above: a college with an entering class of 1,000 whose true retention rate rose 2 points. It is hoping for a two percentage point increase, but it would settle for an improvement of half the true effect (one percentage point) as strong enough proof of success to keep funding for the initiative.
How long would the school need to wait to reliably see that improvement? The answer: 6 years. If they wait 6 years, there’s about a 9 in 10 chance that if the initiative is successful, they will be able to see those gains in actual retention rates. What this means is that if the school is impatient and waits less than 6 years, there’s a strong chance it could prematurely judge the initiative a failure, even if it worked.
Communicating with Your Leaders
This article argues that a single year’s retention rate often carries less signal than the weight we tend to place on it. It can move on its own when nothing has changed; it can fail to move when real improvement is there; and, it can take years to discover whether an initiative actually worked.
The main thing I want to say is that there is a cost to impatience when judging whether your school is improving or not. A genuinely improving school could have a statistically unlucky year – for instance, its retention rate drops a couple points – and its leaders may cut the advising program that was, in fact, working. On the other hand, a declining college could get a lucky statistical draw – “achieving” a two-point gain – and miss negative trends on campus that are not yet visible in the data.
Even though this more cautious approach may be an improvement in a statistical sense, it is also true it can make leaders’ jobs more difficult. It could be interpreted as saying that retention can’t be reliably changed or that leaders will be unable to succeed on the timeline they’ve promised. Meanwhile, others may be able to offer simpler stories that promise quick results, and those stories are often easier to sell.
So, what can we do to help leaders make the right decisions? I have a few practical suggestions. First, report multi-year averages of retention rates so that leaders don’t overinterpret year-to-year changes. Second, set reasonable expectations, ideally before an initiative launches, for how many years it may take to observe a change. Third, when a retention change does appear, look for evidence beneath it: for example, whether the students an initiative served actually did better than similar students who weren’t served (though, of course, the same uncertainty issues still apply).
So when this year’s retention rate drops and someone asks whether the initiative failed, the most honest answer may be: “It’s too early to tell – and here’s how long it will take to know.” That answer may not be satisfying in the moment, but it’s the right way to guide your school to a path of genuine change.
Steve Sherrin works at Wentworth Institute of Technology and specializes in higher education analytics. He can be contacted at sherrins@wit.edu.