Historical vs Monte Carlo

In my experience, historical tends to result in a lower probability of success. Is that because it's factoring in financial/economic circumstances such as the Great Depression, which (allegedly) can't occur again due to the many circuit breakers and checks and balances built into our system now? Does Monte Carlo factor those things in, which could be a reason it typically shows a higher rates of success? For historical, should we use 'modern' periods (i.e. 1950 to start) to incorporate new fiscal/economic policy? That might be a good option to add.

Historical is always way worse for me than Monte Carlo. To quantify: with Monte Carlo, my final 10 year success rate is 100%. With historical, it is below 50%. Is this normal?

@hines202 I spent a good deal of time investigating this over the past two days. The historical analysis definitely does include the Great Depression, but only for the first several test cases out of something like 40 test cases overall. The first thing I did was to convince myself there were no errors in the PRC algorithm, which I did successfully. Then I created a greatly-simplified model that used the same historical data as PRC Gold but with very simple formulas for cash flow and projections of the portfolio balance. This simple model also performed fixed rate projections using the average ROR and inflation values derived from the actual historical data. Then I replicated PRC's historical analysis algorithm on top of this and was able to get very similar results to the actual model using a 50/50 mix of stocks and bonds for all accounts. What I observed, which is consistent with what you reported in your tests, is that the historical analysis results were consistently much worse than fixed rate results in the presence of significant negative cash flows (note that Monte Carlo results always straddle fixed rate results in PRC Gold, but I didn't attempt to replicate that in my simple model). My conclusion is that this is simply the mathematical reality of the historical sequence of returns and the corresponding sequence of inflation rates and, very importantly, it is also a function of the annual cash flow. This phenomenon only presents itself in conjunction with a long term and fairly consistent negative cash flow ($15,000 or more in my testing); if the cash flow is better, the historical analysis produces results that straddle fixed rate results (as Monte Carlo results always do in PRC Gold).

So, I think this simply verifies what we all know: an unfortunate sequence of events occurring early in your retirement can have devastating long term results if it occurs in conjunction with significant negative cash flows. Apparently, the sequences contained in the historical record starting in 1928 have this characteristic. With that said, I did another test with PRC Gold using only the stock asset class because it has historic data back to 1871. In that test, the historical analysis had a much different and much better result, very similar to Monte Carlo results.

Regarding your suggestion to limit the historical analysis to "modern periods", maybe starting with 1950's data to avoid the Great Depression, this has a different problem: the more-restricted amount of historic data would cut the number of test cases roughly in half (such that a user with a 50-year remaining lifespan would only have about 25 test cases). The statistical significance of that result would definitely be in question.

@smatthews51 Thanks for the excellent detail and diligence chasing this down!

When I explain the "Great Depression" angle to my clients, most feel comfortable that it's in there, even though it's not theoretically possible due to today's economic circuit breakers and checks and balances. I wouldn't want to limit the test cases, so I'm with you on that one. Disregard the "modern historical" suggestion 🙂

I agree that sequence of returns is probably the most dangerous risk to new retirees. For those not familiar, what that means is that if there's a big and prolonged downturn very early in your retirement, that suddenly depleted nest egg is a bad thing. For example, if you retired with $1 million and were using the 4% rule to "safely" withdraw $3,333/month at the start of retirement, and now we have a crash where the market drops 33%, your $1 million is now only $750,000 and you're now withdrawing **5.4%**. The 4% rule does take that into account, and it should work as long as you're within the parameters (proper asset allocation, 30 year retirement).

This is why folks should have multiple "safety valves" and backup plans for this sort of event. I like the bucket system - two years of cash. Then you can laugh at market downturns, keep watching those dividends accumulate and reinvest at bargain prices, and ride it out.

I also set my clients up with a viable small business, monetizing some hobby or interest, something they love doing (part-time of course, it's retirement after all) so that they have a way to ramp income up if they need to, and also all those tax write-offs and the opportunity to contribute to a solo 401k to manage taxes in retirement. It also keeps you busy mentally and physically, a bonus for longevity.

Another way to mitigate this is making sure you're invested in funds that don't have big expense ratios, commissions, etc. I teach my clients enough about investing to manage their own, so they're not paying out badly needed money to an advisor every year or in those expensive funds.

Another way to mitigate it is to be flexible. Too many folks turn on the 4% rule distributions and just go with it. During the 11-year bull run we've had, there was lots of speculation that it was too conservative - investments pile up in traditional accounts and lead to huge RMDs at age 72 and a tax problem. The key is to be flexible - maybe cut back during any recession that may happen and enjoy the boom periods more.

Of course, Pralana gives us much more granularity and confidence than the simple 4% rule. It manages our withdrawals from different accounts optimally for taxes, subsidy cliffs, and longevity. That's why I think it's the bomb!

@suzy-leemail-com Sorry for the delay in responding; I needed to do some digging before being able to answer with confidence. Please see my response above to the question from Bill Hines. So, you asked if it was normal for historical analysis to be much worse than Monte Carlo analysis results. In my testing, I'd say it's common for the stated reasons; however, another possible contributor could be that your assumptions of future returns is more optimistic than the historical records (roughly 8% for stocks and 2% for bonds since 1928).

Also, sequence of returns is exacerbated if you put yourself in a position where you have to sell stocks in that down market (because those shares have become devalued, you're locking in those losses). For example, your retirement funds are still in target date funds, or "balanced" funds where you're selling shares of those to finance your monthly/yearly distributions. Those funds are composed of a blend of stocks and fixed income (bonds, etc) and you can't choose which category to sell.

That's why if you relied on those types of funds while working, so someone else is managing your asset allocation glide path for you, you should split them up properly before retiring. It gives you the flexibility to sell fixed income fund shares if stocks are down and fixed income funds are up, or vice-versa.

If both stocks and fixed income are down, you rely on your cash/money market cushion. Buckets.

You did not mention what average rates of return you are using for Monte Carlo so that would be of interest. I will share my thoughts about using historical data and why I only use Monte Carlo analysis. Any feedback would be welcome.

I only use the Monte Carlo analysis. I run three scenarios to capture the range of uncertainty in future average returns and use the same historical standard deviations for each scenario. The range of average returns is based on current market conditions such as equity price valuations and bond yields. Since equity valuations are very high today and bond yields are very low relative to historical values, my expectations for both equities and bonds is lower than the historical average. Most firms (e.g. Vanguard, Research Affilites) published future rate of return expectations are also lower than historical.

In addition to my expectations that future "average" rates of return will be lower than historical, the other reason I don't use historical returns, is that the model year starting conditions for many of the 30 year periods are not consistent with where markets are today. For example, the 10 year treasury is about 1.6% today implying a future return that will be close to that or a little lower if interest rates increase. All of the historical 30 year starting yields are much higher than where we are today. Equities have the same issue. Since we are at very high valuations today, many of the historical 30 year starting valuations are much lower than today. It is the first 10 years that matter the most for retiree's.

@ Bill Hines. When I looked at historical safe withdrawal rates I was surprised to see that the lowest historical safe withdrawal rates was in the mid 60's not during the great depression I think due to the three years of deflation that averaged about -9%.

very interesting discussion...

In my "current" model, the historical results are better than the Monte Carlo results...success rate on historical is 100% and Monte Carlo is around 90%...

I've not tinkered with the RROR or asset allocation much yet, but did try to come up with something not overly optimistic based on today's current market behavior...so that may be why my results are different...

also seems that having a cash cushion entering retirement withdrawals makes a big difference on sequence risk, which i do have included in my model...

certainly something to investigate...

@golich428 Yes, actually the worst historical period to retire wasn't the Great Depression. So, starting historical simulations in 1950 wouldn't help that much. But even since the 60's, we've advanced quite a bit in our technology, preventions, and most of all knowledge of those bad economic times in the 60's, 70's, 80's, 2000's and our ability to head off or cure bad economic conditions. So, that's good news, and also why previous historical conditions for retirement forward projections aren't so useful (but are quite conservative!)

@michaellarimer I agree. I'm a fan of the buckets. If you have a 2-year cash cushion, you can laugh off most recessions and continue receiving dividends, plus reinvesting them means you're getting more shares at a bargain. Some analysts say it's silly to have so much in cash, it should be earning/growing, but many folks are conditioned in their thinking due to this 11-year bull run in the stock market, and forget what 2008/2009 was like. It's all about risk tolerance, but retirement should be relatively stress free, in my opinion, and sleeping well at night (in a hammock, under a palm tree...)

I too was a little surprised at how poor my outcomes appeared in the historical analysis. To sanity check it a bit I set up a classic simple scenario (Couple, retiring 65, $1M IRA, no SS, $30k exp for 30 yrs) in Pralana Gold, Jim Otar's Retirement Optimizer (Jim is a big proponent of historical analysis) and Firecalc.com. Actually got similar-ish outcomes so I felt reassured. Stuart's comments above helped too.

@golich428 regarding the average rates used for Monte Carlo I assumed based on Stuart's comment, "Monte Carlo results always straddle fixed rate results", that Monte Carlo analysis uses the rates you put in under Financial Assets/Asset Classes as the average rate. The standard deviation for the MC analysis is there too.

David Ray

I am a little late to the party, but here is my 2 cents. Measuring risk is at the heart of retirement planning and these two forms of computer simulations are the most common (with bootstrapping a third). However, I don't put much faith in Monte Carlo analyses for retirement planning. Historical (Backtesting) is based on real market data, Mote Carlo randomly create data. A Monte Carlo simulation is a model used to predict the probability of different outcomes when the intervention of random variables is present. Monte Carlo simulations also assume perfectly efficient markets. The global financial markets are hardly random and certainly not perfectly efficient. Markets are run by emotional, borderline knowledgeable people. Good luck with that!

Using Monte Carlo for retirement planning is only as good as the underlying model of the markets, which is problematic. There are no agreed standards for modeling markets in a realistic way. There are numerous Monte Carlo algorithms out there, which one do you pick? Critics contend that Monte Carlo analysis cannot accurately factor infrequent but radical events, such as market crashes in its probability analysis.

Backtesting takes into account asset allocation, return disposition, sequence of returns, volatility, inflation, momentum, etc. Looking at momentum, the stock market has a tendency to drop quickly and by large amounts, but then gain in value for months, sometimes years, before another big drop. This is one reason that annual re-balancing is not a good idea (but that's another discussion).

For me Monte Carlo results are always worse than Historical (backtesting) returns. I think it is mainly because I use a 70/30 asset allocation and assume a 3% withdraw rate.

Some of my information comes from Living Off Your Money by Michael H. McClung 2015.

If using historical, I'd definitely focus on recent periods (2000 dot-com bust, 2008 financial crisis, 2018, etc). Maybe back to 1965 (worst case!), 1970's inflation, and 1985 just for fun but I wouldn't put a lot of stock in those results.

The Kitces group published a very good article on comparing Historical return simulations and Monte Carlo projections https://www.kitces.com/blog/monte-carlo-analysis-risk-fat-tails-vs-safe-withdrawal-rates-rolling-historical-returns/ well worth the read 💣

They are not saying to not use Monte Carlo simulations for retirement planning, but to be aware of some of its shortcomings.

@howard230 later in the article it states that it is not a good idea to assume a low rate of return for stocks (like 2%) if looking at a long term retirement projection as you will very very likely end up with a lot of unspent money when you depart this world 🙄

Posted by: @giovanelli766

@howard230 later in the article it states that it is not a good idea to assume a low rate of return for stocks (like 2%) if looking at a long term retirement projection as you will very very likely end up with a lot of unspent money when you depart this world 🙄

Not to mention big ol' RMD payments! Of course, those can be offset by doing Roth conversions, if possible. As well, retirement planning should turn into "retirement adjusting" on a yearly basis. I meet with my clients for an end-of-year review and if things have been going well, and the tax impact isn't egregious, the fun money can be adjusted up next year 🙂 Of course, the same things goes if we go into a correction, perhaps dialing down to stay on a steady course of ~95% success and using PRC each year to keep that dialed in.