By David Graham, MD, Guest Writer
White Coat Investors are excellent accumulators of wealth. Once you have your accumulation plan in place, it might be time to think about some issues in physician retirement planning.
Physicians are a heterogeneous lot, this much is true. There are, however, some common issues faced in retirement by those who wear the white coat. Let’s review the epidemiology of physician retirement and then go over some common issues in retirement planning for physicians.
Epidemiology of Physician Retirement
When Do Physicians Retire?
Figure 1 Physician Retirement by Age
Source https://www.ama-assn.org/practice-management/career-development/top-considerations-physicians-ahead-retirement
Surprisingly, 58% of physicians retire after the age of 65! In the US general population, women retire at age 60-62 and men 62-64. Do physicians retire later on average because of the late start we get after a decade of training?
Why Do Physicians Retire?
What does a metanalysis say as to why physicians retire? According to this study, physicians do indeed retire later than average.
Reasons to retire early include low job satisfaction, medicolegal issues, health concerns and financial troubles. Reasons given to delay retirement include career satisfaction, institutional flexibility, feeling responsible for patients, financial reasons and a lack of interests outside of medicine.
Finances show up on both sides—retire early and delay retirement. What are these financial concerns? Net worth is one of them.
Net Worth of Physicians
Let’s look at physician net worth by age.
Figure 2 Physician Net Worth by Age
Maybe you’ve seen the data in Figure 2 before. It’s from a Medscape survey of physicians and modestly depressing.
While you can easily retire with less than $5M, especially when you are traditional retirement age, look at the purple bar (net worth > $5M) increase with age. It goes from 3% up to 22%. So, about 1 in 5 physicians over the age of 70 have a net worth greater than $5M.
The percentage between $1-5M doesn’t really change much with age. Sadly, 25% of physicians older than 65 never have a nest egg larger than $1M.
Summary of Physician Retirement Epidemiology
So, on average, physicians retire later than the general population yet with more financial resources. Still, given the high incomes physicians earn, it is depressing so many of our colleagues are not better prepared for retirement. Financial concerns are cited as reasons both to retire early and to retire late.
Let’s move on now. I assume anyone reading this is doing well for themselves. What are some issues that impact physicians planning their retirement?
Selected Issues in Physician Retirement Planning
Retire To Something.
Everyone needs to consider what they plan to do in retirement.
Many physicians find more than a mere calling in medicine. Letting go the title of “Doctor” is difficult. Finding purpose in retirement, however, is necessary to ensure health and sanity.
It will be interesting to see trends in physician retirement in the next decade given the industrialization of medicine and the loss of autonomy many physicians are facing. Encore careers are common for physicians. How will we find purpose?
Liquidity Events
From poor distribution options on Non-Governmental 457 plans, to liquidating investments in practices, equipment or real estate, many physicians will face liquidity events.
A liquidity event happens when you must recognize a bolus of ordinary or capital gain income all in one year.
Of course, ideally, you would be able to take a better distribution option on your 457, or spread out your payout for your share of the practice or investment over two or more tax years.
If you are forced to “suffer” a liquidity event, some planning is possible. Tax planning is important that year, perhaps to reduce your other sources of income, or find a way to take a major tax deduction in the same year.
Other possibilities include Oil and Gas, Opportunity Zone Investments, Real Estate depreciation through cost segregation and bonus depreciation, and more. Plan at least a year in advance with your CPA.
Large Pre-Tax Accounts
Many physicians have large 401k or 403b accounts and face huge future tax burdens as Required Minimum Distributions force out this deferred income. These 401k Millionaires have the opportunity to do some tax planning which may include partial Roth conversions.
After you retire and before you start social security and RMDs, you have the chance to control your income. This Tax Planning Window is the most important opportunity to control your future tax liability and should not be squandered.
Other Tax Considerations
White Coat Investor not infrequently calls retirement planning issues “first world problems.” Given his gift for tax efficiency while practicing and running a burgeoning business, I expect he will change his tune when he considers retirement and sees how much he will pay in taxes during his own retirement! [To be fair, EVERYTHING discussed on this website is a first world problem. But if I pay more in taxes in retirement than I do now, it's going to be one heck of a retirement-ed.]
In fact, taxes can be a physician’s largest expense not only while practicing but also in retirement. It pays to be tax efficient. After all, every dollar you save is another dollar you can send or give to heirs or charity.
Aside from partial Roth conversions, there are other tax-efficient withdrawal strategies. These include:
- minimizing your taxable brokerage account (yet maintain liquidity)
- use lower tax brackets to recognize pre-tax income
- order of withdrawal from various account types
- capital gain harvesting in the 0% cap gains tax bracket
- avoiding Medicare Surcharges (IRMAA — a Cliff Tax on the Rich)
- maximizing the efficacy of charitable deductions
Charitable Planning
Since the Tax Cut and Jobs Act, most have been using the standard deduction and no longer get tax deductions for our charitable gifts.
Not that a tax deduction is the only reason to give, but if you can find a way to give tax-efficiently, you can give more!
If you have charitable intent, there are many considerations. These include:
- bunching gifts on alternative years
- Donor-Advised Funds
- QCDs after age 70 ½
- Charitable Remainder Trusts for income during your life, or as a Stretch IRA alternative for your heirs
Legacy Planning
Leaving a legacy is important to consider as many doctors will have more money than time. The loss of the Stretch IRA has engendered some new retirement planning strategies. Unfortunately, none are as good as the Stretch IRA.
Cascading beneficiaries, disclaimer planning, multi-generation spray trusts, Roth conversions, Life Insurance, and Charitable Remainder Trusts are all considerations when planning your legacy.
Long-Term Care Insurance
Since we have all seen the ravages of age, Long-Term Care Insurance is a quandary in the back of our mind. While many retired physicians can self-fund long term care needs, there is a dearth of insurance options if you want to transfer the risk of a large Long-Term Care event.
Traditional Long-Term Care Insurance suffers from issues of premium increases and future solvency.
So-called hybrid LTC/Life Insurance policies are becoming more popular but have copious downsides. One issue I cannot get the insurance industry to understand about these policies is that you are often better off NOT TO USE one once you have it. This is because medical expenses are a tax deduction above an AGI floor. You can withdraw pre-tax money tax-free if you have a large enough deductible Long-Term Care event. So, if your goal is leaving money behind for your heirs, you are better off withdrawing pre-tax money tax-free to pay for healthcare costs, and letting the tax-free death benefit of the policy go to your heirs. Or, not having a hybrid policy in the first place!
Long Term Care considerations are the most intractable problem in retirement planning today. This is because they can have truly devastating costs, but insurance is priced to cover non-devastating events. Imagine how expensive home insurance would be if we all had small or moderate fires in our house at least at some point in our lives! Insurance is best saved for risk pooling of catastrophic events, which is not how Long-Term Care insurance operates currently.
Social Security
Social Security planning should be something that all physicians are familiar with. If you have done well and have a reasonable life span, consider delaying social security to take advantage of delayed retirement credits. Social security is, perhaps, the best longevity insurance out there, with inflation adjustment and a government-backed guarantee.
Most physicians should just assume that 85% of their social security will be included with their taxable income, as the brackets for taxation of social security are absurdly low and haven’t been inflation-adjusted since Regan.
Unwinding from Bad Financial Advice
This is, perhaps, the most interesting retirement planning conundrum. Many physicians have been suckered into bad investments or products by “financial advisors.” Unwinding these takes care, consideration, and patience.
If you have an insurance salesman masquerading as a financial advisor, you might own expensive annuities or unnecessary permanent life insurance products. Unwinding these might just involve liquidation and some “stupid tax.” There are other important considerations, though.
With annuities, if you need income in retirement, you may consider annuitizing the policy, or actually using the Income Rider that you have been paying for all these years. Or, you could 1035 into an Investment Only Variable Annuity depending on the goal of the funds. I’ve found most folks who have been sold an annuity don’t really understand “why” they own the annuity and what purpose it is supposed to serve in the future.
With Permanent Life Insurance, you may consider a 1035 into a hybrid LTC policy if you want Long Term Care Insurance. WCI writes about other options for life insurance as well. Understanding the indication for owning a Permanent Life Insurance policy is key.
If your financial advisor is a stock picker, then you might have individual stocks or crazy expensive mutual funds with a low basis. Unloading these will involve paying capital gains taxes. Many doctors will remain in the 18.8% capital gain tax bracket on future projections, so a decision must be made regarding the purpose of the money. If you plan to unwind it at some point due to the tax-inefficiencies of the funds, earlier is better as it allows you to reset your basis and chose funds more in line with your goals and chosen asset allocation.
If you have a donor-advised fund, or bunch deductions, these low basis bad investments are perfect to give away.
Should I Work a Few More Years?
Ultimately, there is no perfect answer to the question “do I have enough to retire?”
Clearly, according to figure 2, some will never have enough to retire! If you spend more than you earn, you will never get there.
Others, like Physician on Fire, prove that you can limit your spending and goose your savings rate, such that early retirement is possible after a mere decade or so of practice. This, of course, requires continued spending discipline, but I doubt there are many who question Physician on Fire’s resolve to stay disciplined.
Summary
Retirement Planning seems backward after being in the accumulation mindset.
Instead of growing your wealth, you need to consider spending some of it down!
There are additional pitfalls and traps in retirement that take some planning. Many of these issues uniquely affect physicians and other high-income professionals.
A forward-facing 20-30-year tax plan is often useful to see what your tax bracket “number” will be. For instance, if you know that RMDs will expose much of your retirement income to the 25% federal tax bracket in the future, use all your tax bracket below the 24% bracket now! A percent or two doesn’t seem like it would make all the much difference, but if you can rescue money from a pre-tax account and convert it into a Roth, all future growth is also tax-free.
The goal, of course, it not to perfectly predict how much you will be paying in taxes even five or ten years from now. Everything can and will change and many assumptions will be wrong!
The goal is to optimize the next 1-3 years so you set yourself up in the best possible position to roll with the punches as they come, all while efficiently funding retirement goals.
What are you doing to optimize your retirement planning? What mistakes have you made? Comment below!
[Editor's Note: David Graham, MD, is an infectious disease physician and advice-only financial planner. This article was submitted and approved according to our Guest Post Policy. We have no financial relationship.]
I am comfortable directly owning investment real estate so am utilizing the cost segregation with bonus depreciation strategy you mentioned to move $ out of an IRA without having to pay taxes on the distributions. I am 68 and will move $1.5M this year and plan on moving more next year. The strategy is time limited as you know as Trump’s tax law will start “sunsetting” in 2023. We are going to save a lot of taxes and avoid having to take RMDs at 72 because of this strategy. I also am moving money from a Fidelity IRA into a Fidelity Roth IRA for the same reason. We are doing that over several years to try to avoid getting pushed into a high tax bracket in any given year. The recent pushing back of the age at which we have to take RMDs to 72 helps with the ROTH conversions.
How do I learn more about the cost segregation with bonus depreciation strategy ?
Bonus depreciation is new with the 2018 tax law. Let’s you get the depreciation out of a property much faster than under the old laws. The IRS has publications on it, you can Google it and read it blog posts, or you can discuss with an accountant.
Cost segregation means different things in different situations. With real estate, it basically means doing an analysis and calling some components (appliances?) of the property “personal property” and some components (the walls and floors for instance) “real property” and then depreciating them differently. You can depreciate personal property faster. That can be useful if you have real estate income to offset, but if you don’t, then perhaps it is better to delay taking the depreciation. Thus the point of a study by a specialist.
When I informally polled nearly a thousand physicians sometime back, more than 10% did not know how much they would need in order to retire. How do you get to a goal if you cannot define it? Hopefully, with improving financial awareness, this number will shrink and disappear.
Thank you for the information on tax-free withdrawal of pre-tax money in case of LTC events.- I wasn’t aware of it. This adds another reason not to go with complex forms of insurance. LTC insurance definitely remains suboptimal.
Thank you for a very well-written and comprehensive rundown of the considerations in retirement planning!
-PFB
Great article and timely for those of us late in our careers.
First, let me say there is lots of good advice here, but the following quote suggests some “not so good.”
“Aside from partial Roth conversions, there are other tax-efficient withdrawal strategies.”
Why do people always think the number one strategy here is a bigger Roth and a smaller Traditional? What do you think happens when you have a bigger Roth than traditional? You have the chance of running out of Traditional money at some time during retirement. While that may seem like a good problem to have it just means you gave the government much more tax than you needed too! BECAUSE if you do end up with a smaller tax burden in retirement than while working or doing Roth conversions that is the IDEAL place to be spending traditional money! The other behind the scene thing that happens in retirement is you typically are spending retirement withdrawals across multiple tax brackets which makes your effective “marginal” tax rate a combination of these multiple brackets and NOT just the highest bracket.
One of the BEST ways the higher wealth individuals CAN save taxes, which I didn’t see mentioned (?) is to keep those tax-generating investments to a minimum in your taxable account – meaning no dividends or very cost-effective index funds with low dividend payouts.
It is always nice to have “some” Roth and in most cases, you WILL pay higher taxes to get it. The purpose of that Roth is then to avoid the super-high tax brackets that come along with some large lump sum obligations that might arise during retirement.
If you’re spending the entire qualified dividend in retirement, I don’t see a problem with getting it. The problem is being forced to take a dividend you wouldn’t “declare on your own.”
Most docs and most other professionals who save and invest wisely will have multi million tax deferred accounts at age 60-67 or about
Whenever you can do Roth Conversions to stay within the 24% bracket might be advisable
I do it because I am a Florida resident and gladly pay 16-18% effective tax rate on 250-325K in income under the new tax law
Michael Kitces writes article about efficient withdrawal strategies in retirement to minimize taxes; basically using taxable, tax deferred, and tax free in combination for least taxes
One of the advantages of being able to retire early (I am personally shooting for age 53 (the age my daughter goes to college) is that you extend the window where you can do ROTH conversions.
I am hoping to have a bucket strategy employed with 2-3 years of living expenses in cash equivalents when I finally do retire.
I then plan on withdrawing from my traditional IRA/401k and convert it into the much more desirable ROTH designation.
I will need to see which tax bracket would be the best to aim for (would ideally be the one where capital gains tax is taxed at 0% but that might be too low an amount to withdraw).
I simply cannot understand why the percentage of MDs with net worths north of 5 million dollars isn’t higher. I wonder if there isn’t systemic under-reporting of both income and net worth among MDs.
I doubt it, having talked to many of them. What seems easy to you doesn’t happen very often. There’s a reason Stanley and Danko called docs “Underaccumulators of wealth.”
Yes I understand the label but given that the prototypical ‘Millionaire Next Door’ was often a pretty regular Joe, ‘under accumulator’ seems like a wild under statement. And our listed net worths are not in 1980s or 1990’s dollars as in the Millionaire Next Door.
Interesting post that unfortunately does not likely offer us any tax saving options.
I am an over-accumulating, supersaving doc. We are perhaps in the 1% of net worth for a physician family. It took a while in the early years to pay off loans and hit a net worth of 1M, but after that, the doubling time for the net worth keeps getting shorter. I am still working as a doc part time. I do sometimes think about the theoretical potential for a malpractice risk to my assets. I think for the most part I am safe as I practice in a hospital and the hospital would typically be on the hook for any award greater than my 1.3M policy. I was sued twice so far during my long, fairly high risk specialty, but both suits were dismissed with no payment.
The comment about taxes being the largest expense hits home. Our taxes now and into the future are likely going to remain significantly more than all of our other expenses combined, and that may continue out to the horizon. Our tax deferred accounts as a couple are already at 6.3M and the financial calculators forecast a potential for a hefty 8 figure tax deferred balance at first RMD. Due to real estate investment income and other passive investment income, it is likely we will always be in the highest bracket and will never have an opportunity for Roth conversions. First world problems, indeed.
These days we simply keep upping the charitable contributions. We have been writing extra large checks these past months to the food banks given all of the unemployed struggling due to Covid.
You should probably consider doing lqrge Roth conversions, even at 37%, just to reduce your likely estate tax bill.