By Dr. James M. Dahle, WCI Founder

I met a well-to-do physician not too long ago. Now retired, the doc has a mid-eight figure net worth. Yes, you read that right.

Despite massive wealth, the doc only spends a couple hundred thousands per year. That did not surprise me too much; many very wealthy people spend “like doctors,” i.e., $200,000-$300,000 per year. What surprised me was to learn the doc was paying less than $100,000 per year in taxes. While I didn't get the details from the doctor, I did spend some time thinking about how someone worth $50 million could get away with paying less than $100,000 in taxes. I came up with several different scenarios and discovered it was actually really easy to have a low tax bill. Certainly, the less you spend, the easier it is to do. But even with significant spending, it can still be done.

 

Case Study #1: A Relatively Frugal Real Estate Investor with a Very Tax-Efficient Portfolio

Imagine a 60-year-old retired doctor who:

  • Lives in a tax-free state,
  • Spends $300,000 per year, and
  • Gives away $100,000 per year.

This doctor has the following assets:

  • $5 million in a primary and secondary home
  • $10 million in muni bonds
  • $15 million in a Total Stock Market Fund (TSM)
  • $5 million in a Roth IRA and a traditional IRA
  • $30 million in investment real estate with $15 million in mortgages against it
  • Net worth = $50 million

How much is this doctor going to pay in taxes?

Remember, there is no wealth tax (at least until death), only an income tax. So, how much taxable income is this doctor going to have? Let's look at those assets again:

  • Homes: $0
  • Muni bonds: Let's say the average yield on these bonds is 2%, so $200,000 and none of it is taxable.
  • TSM: The fund made almost 30% in 2021, but the yield is only 1.14% and it is all qualified dividends. 1.14% x $15 million = $171,000
  • Retirement Accounts: No money taken out, no taxes due. No RMDs due for 12 more years.
  • Real Estate: We'll assume a yield of 5% ($30 million x 5%) = $1.5 million, but it is entirely possible that it is all covered by depreciation.
  • Total spendable income: $1,871,000
  • Total taxable income: $171,000 – $95,500 in deductions = $75,500
  • Tax due: $12,359 single ($8,662 MFJ)

See how that works? While it is hard to have a lot of earned income and avoid massive taxation, it's relatively easy to have a lot of unearned income and assets and pay almost nothing in tax. Now, you can quibble about little things in this case study, but the bottom line is the tax bill is incredibly low for the level of wealth.

 

Case Study #2: A Big Spender Takes Advantage of Loans

Let's imagine another 60-year-old doctor. This doc:

  • Lives in a tax-free state,
  • Is retired,
  • Is married,
  • Gives away $500,000 a year, and
  • Spends $2 million a year.

Never mind the fact that I somehow found two doctors with $50 million a year. It's pretty rare but it happens. This doctor's assets look like this:

  • Homes (4 of them!): $15 million
  • Roth IRA: $1 million
  • Traditional IRA: $9 million
  • $15 million in Total Stock Market Fund
  • $10 million in Total International Stock Market Fund (TISM)
  • Net worth: $50 million

How much is this doctor going to be paying in tax? Let's take a look.

  • Homes: $0
  • Roth IRA: $0 no matter how much is withdrawn
  • Traditional IRA: $0 if none is withdrawn
  • TSM: $15 million x 1.14% = $171,000
  • TISM: $10 million x 3% = $300,000
  • Total spendable income: $471,000

So far, so good . . . except the doc is going to spend $2 million and give away another $500,000 this year. But there is only $471,000 in income? What is the doc to do? Here are the options:

  • Borrow against the home (Costs future interest but no taxes)
  • Withdraw from the Roth IRA (No taxes due, but only $1 million in there)
  • Withdraw from the traditional IRA (Some taxes due)
  • Sell some mutual fund shares (Some taxes due)
  • Borrow against the mutual fund shares with a margin loan (Costs future interest but no taxes)

There are lots of options and no reason that one cannot use a combination to meet the $2,029,000 shortfall. Here's one option:

  • Withdraw from the Roth IRA: $29,000
  • Withdraw from the traditional IRA: $200,000
  • Sell mutual funds shares: $800,000 but with these high-basis shares, basis = 3/4 of value
  • Donate low-basis mutual fund shares: $500,000
  • Take out a 2% margin loan: $500,000
  • Total spendable income: $2,000,000 + $500,000 charitable gift
  • Total taxable income: $471,000 in qualified dividends + $200,000 ordinary income + $200,000 in long-term capital gains – $250,000 qualified charitable contribution = $621,000
  • Total tax: $147,798 – $27,900 foreign tax credit = $119,898

avoid paying taxes for rich people

The doc will also pay about $10,000 a year in interest going forward on that loan. Hard to complain too much about spending $2 million and giving away another $500,000 while only paying $120,000 in tax.

 

Case Study #3: The Roth Doctor

Now let's talk about a 75-year-old doctor who

  • Is retired,
  • Is married,
  • Is debt-free and plans to stay that way,
  • Spends $500,000 a year,
  • Receives $50,000 a year in Social Security,
  • Gives away $1 million a year.

This doctor was pretty aggressive about Roth contributions and conversions during a long career and the assets now look like this:

  • Home: $2 million
  • Roth IRA: $15 million
  • Traditional IRA: $8 million
  • Muni bonds: $10 million
  • TSM fund: $15 million
  • Net worth: $50 million

This doctor has lots of options. Let's start by calculating the RMD. At age 75, it's 4.4%. Take the 4.4% x $8 million = $349,345. Let's call it $350K. The doctor takes $100K of that as a Qualified Charitable Distribution (QCD) and donates the other $250K to charity after withdrawing it. The doc also donates another $650,000 in Total Stock Market Fund. Those are two very tax-efficient ways to give to charity. So, that takes care of the giving. Now, what about the spending?

  • Social Security: $50,000
  • Muni Bond Interest: $200,000
  • TSM Dividends: $171,000

The doc is still $79,000 short. So, the doc takes it out of the Roth IRA. Or sells some muni bonds. The basis is about equal to the value, so there's no tax due there. The muni bond interest is tax-free. Only 85% of the Social Security is taxable so the adjusted gross income = $42,500 in ordinary income plus $171,000 in qualified dividends. However, half of that taxable income is wiped out by the tax deduction from the charitable donation (and the doc carries the rest forward). I think that applies first to the ordinary income, so that is wiped out completely, leaving $96,750 in qualified dividends. At 15%, that's a tax bill of under $15,000. Practically non-existent compared to $50 million in wealth.

 

As these three case studies demonstrate, a very wealthy person can spend quite a bit of money without paying much at all in tax, especially if they are charitable. Whether you think that's good tax policy, that is the way the tax code is written. Structure your retirement income accordingly.

What do you think? Are you surprised by how low the tax bill can be for very wealthy people? If you are retired, how much do you pay in taxes? What are you doing to reduce your future tax bills? Comment below!