Realty Mogul AdPart 1 of this series covered the dilemmas faced by retiree investors. Part 2 warned against becoming too focused on income. Part 3 showed you how to reduce your need for portfolio income using Social Security, Pensions, SPIAs and rental real estate. In Part 4, I’m going to discuss a few issues faced by early retirees and a discussion of how to benefit from the tax diversification you worked so hard to achieve during your career.

In 2011, I wrote a post entitled 14 Reasons Why You Shouldn’t Retire Early. Although many of us dream of early retirement, especially after a bad day at the hospital, there are many good reasons not to. Early retirees face a number of financial issues that someone who retires in his 60s or 70s doesn’t.

Perhaps the biggest issue for an early retiree is that he has to save a ton of money. When you shorten your career, you also shorten the amount of time for your money to compound prior to beginning to spend it. Not to mention you have fewer years in which to save up the money, and are often forced into using a taxable account instead of a tax-advantaged one due to the heavy savings load required. There are several other issues an early retiree has to deal with.

Health Insurance

Crossing a 42 m bridge to nowhere- Val d Isere, French Alps

Crossing a 42 m bridge to nowhere- Val d Isere, French Alps

This is less of an issue for a physician early retiree than for others. Many doctors are used to paying for their own health insurance anyway. They did it before retirement, so it won’t be a shock to see what it costs after retirement. PPACA changes help keep costs down for early retirees by shifting them on to younger folks (although many would argue they’re higher for everyone now.) Doctors should also be a little wiser about what really needs medical care and what doesn’t, perhaps saving a few bucks in office visits, labwork, and imaging. Hopefully a doctor also takes good care of herself. The cheapest health care is for a healthy person who doesn’t need much! But even with these factors, someone who retires at 50 has 15 years of health care costs to cover before getting Medicare. Early retirees need to have a plan for this. The plan may simply be a larger portfolio. But it could also be retiring to another country, continuing part-time work at a level just enough to qualify for health insurance, or using an HSA built up during their working years. The important thing is to have a plan, as you don’t want health care costs to force you back to work after 5-10 years of retirement.

Social Security

Bridging the gap to Social Security can also be an issue for many. If you retire at 50 and don’t plan to take Social Security until 70, you almost have to eliminate Social Security from your planning all together. Not only are your payments lower (since you paid in less due to fewer working years) but you’ve got to make it two decades without it. If you can make it two decades, you’ve probably got a nest egg that will last another 10-25 years. Nevertheless, the principles of when to take Social Security remain the same. You don’t necessarily want to take it at 62 just because you retired early. The later you take it, the more insurance it provides against running out of money late in retirement. Social Security is insurance that you probably won’t need if you die early in retirement anyway. Yes, you’ll have spent less than you could have, but that’s much less consequential than eating Alpo in your 90s.

Breaking Into the Retirement Account Vault


Lots of people worry about how to access their money without penalty before age 59 1/2. It’s really not that big of a deal as I’ve written about before. You can access other sources of funds including taxable accounts, 457s, cash value life insurance, or an HSA (for health care costs). Depending on your 401(k), you may also be able to access that prior to age 59 1/2 without the usual 10% penalty, once you’ve separated from your employer. There are lots of exceptions for getting into your IRAs, including death, disability, health costs, an IRS levy, education or first home costs for your children etc. You can also take advantage of the SEPP rule. Basically, you can retire at 50, and as long as you take the same amount out of your IRA every year, there is no 10% penalty. Roth IRA contributions can always be withdrawn without tax or penalty as well.

Tax Diversification

I’m always harping on young accumulator investors to make sure they’ll have some tax diversification in retirement. That means making Roth contributions while in training or the military, continuing backdoor Roth IRAs throughout your career, perhaps using a Roth 401(k), and perhaps even doing some Roth conversions during low income years (perhaps even after retirement.) So what do you do with tax diversification now that you have it?

First, use up the low tax brackets, especially as an early retiree. If there is no Social Security income, rental income, or working income, then you can withdraw a substantial amount of money from your tax-deferred accounts at a very low effective tax rate. In 2014, a married couple filing jointly taking the standard deduction can withdraw $20,300 at a 0% tax rate, $18,150 at a 10% tax rate, and another $55,550 at just 15% (plus any applicable state tax, of course.) If your marginal tax rate was 28-39.6% when you put that money in, that’s a heck of a great deal. Even if you don’t want to spend all that money, you can always use some of it to pay for Roth conversions.

Second, take a careful look at that taxable account. Capital gains and qualified dividend distributions are taxed at a very low rate, especially if you’re able to stay in the low tax brackets (the rate is 0% if you’re in the 15% bracket or lower). You can also sell high-basis shares with minimal tax consequences.

Third, if you need/want more income, you can raid your Roth accounts. That married couple discussed above could easily have $200,000 in spending money while paying less than $10,000 in taxes, an effective rate under 5%. Further tax-free (but not interest-free) money can be obtained by borrowing from the cash value of permanent life insurance policies, but this does bring on some additional risk of policy collapse as well as reduce the death benefit payable to heirs.

What About That HSA?

I’m a huge fan of Health Savings Accounts (HSAs), the stealth IRA. Personally, I max mine out on January 2nd every year, invest it aggressively, and don’t plan to touch it until retirement. There are really four ways to use an HSA. The first is to use it to pay for your health care expenses throughout your career. This might be the intended use for the account, but it isn’t the wisest. The second use is as an extra IRA in retirement. After age 65 (NOT 59 1/2 like IRAs), you can pull the money out without penalty and spend it on whatever you like. However, that converts it from a triple-tax-free account to a double-tax-free account, not exactly wise.

The third use requires a lot of work. If you keep all the receipts you spent on health care throughout your career, you can pull the money out tax-free in retirement and spend it on whatever you like. There is no rule that you have to pull the money out of the account the same year you acquire the health care expense (could change, of course.) The fourth, and best use, of an HSA is simply to spend it on health care as late in retirement as possible. That preserves it’s triple tax-free benefits, and also maximizes the time for that money to compound in a tax-protected manner. Plus you don’t have to keep all those receipts. Of course, not only do you not know when you’re going to die, but you also don’t know when you’re going to get sick. But you’re not stupid, and can adjust as you go. If you have a huge HSA, start spending it earlier in retirement (or even before retirement) and if you have a tiny one, you can hold on to it for another decade or two.

medical student loan refinancingRequired Minimum Distributions

Once you hit age 70 1/2, be sure to at least withdraw the required minimum distributions (RMDs) from your tax-deferred retirement accounts. Uncle Sam wants his money eventually, but he’s perfectly content to get it bit by bit. RMDs are only 3.6% at age 70 and only 8.8% at age 90. Remember you don’t have to spend an RMD (but don’t feel badly about doing so) and can always just reinvest it in a taxable account.

In Part 5 I’m going to wrap up this series with a potpourri of issues unique to retiree investors.

Do you plan to retire early? What issues are you most concerned about? Comment below!