By Dr. Jim Dahle, WCI FounderRegular readers know that Katie and I became financially independent sometime around 2018, six-plus years ago. They also know we are continuing to work, save, and invest. Despite spending and giving more than ever before, that nest egg continues to grow. That means we're wealthy and getting wealthier. One of the side effects of this fact is that I've noticed I don't care nearly as much about some things I used to care more about. I suspect this is common to all wealthy people.
7 Things the Wealthy Don't Worry About
Here are seven things the wealthy no longer worry much about.
#1 Running Out of Money
If you retire just when you hit “enough” (or worse, before you have “enough”), you worry about running out of money. You budget carefully, watch your spending, pore over safe withdrawal rate studies, and worry about the sequence of returns risk. If you work for five years after you hit “enough,” you no longer worry about this.
#2 Your Withdrawal Rate
Similarly, that 4% rule is endlessly debated on financial forums. I'm not sure why. If you just leave your money alone, it doubles every 7-10 years. If you're adding to it, your money may double in just 3-5 years. If you had enough five years ago to live on 4%, now you have enough to live on 2%. Only the truly paranoid worry about a sub-2% withdrawal rate.
More information here:
The Silliness of the Safe Withdrawal Rate Movement
#3 Asset Allocation
Studies have shown that asset allocation has a far more important effect on your portfolio returns than security selection or market timing. But you know what is much more important when it comes to reaching your financial goals? How much you have and how much you save. Save enough and you can have a portfolio that is 100% cash. When you can have a portfolio that is 100% cash, 100% stocks, or anything in between and still meet all your goals, you don't worry so much about asset allocation.
#4 Rate of Return
I still calculate our rate of return each year. As a student of finance, I find it academically interesting. Important to our financial goals? Not anymore. For the same reason our asset allocation no longer matters, our rate of return doesn't matter either. If we had a massively negative return for several years in a row, I suppose that would be a problem, but barring nuclear catastrophe, we're not running out of money during our lifetimes.
#5 Whether Social Security and Medicare Are Viable
Social Security and Medicare make up an important part of the financial plan in retirement for most Americans. Forty percent of Americans live ONLY on Social Security in retirement. Wealthy people were often high earners, and they now receive a substantial Social Security benefit. Most white coat investor families will receive $50,000 a year or more from Social Security, the equivalent of an inflation-indexed Single Premium Immediate Annuity (SPIA) valued at $1 million or more. Medical insurance as you get older could run as much as $30,000 a year. But as your wealth increases, the contribution of these social insurance programs to your retirement security becomes smaller and smaller. Politically, you care about these programs. Financially? Not so important.
More information here:
6 Tips for Those Who Have Enough
How Much Money Does a Doctor Need to Retire?
#6 Tax Rates
Likewise, your tax rate stops mattering as much. Don't get me wrong. High earners stop caring about many tax deductions because they are phased out of them. You still care about how much of your money is going to taxes even once you're wealthy. It's probably your biggest single expense, even in retirement. But does it really matter if you're paying 35% or 39% on your income or 15% or 20% on your capital gains? Not really. And the more wealthy you get, the lower the percentage of your wealth that your income becomes.
#7 Leverage
Many people choose to use leverage to help them to build wealth. White coat investors endlessly debate whether to pay off debt and get a guaranteed return or carry the debt and invest the now available dollars. You know who doesn't worry about that? The wealthy. They often just pay off all their debt to simplify their life. But even if they carry some, they don't carry enough to worry about how they're going to service it.
7 Things the Wealthy Do Care About
If the wealthy don't care about all of those things they used to care about, what is important to them? Plenty, as it turns out.
#1 Whether They Are Ruining Their Kids
A frequently cited concern of the wealthy (along with high earners) is whether the lifestyle they have adopted and can easily afford is having untoward effects on their children. Many live well below their means because they don't want to set their children up for failure. How many children of doctors have had a rude awakening when they realized that piano teachers don't command the same income and lifestyle their parents had? How many people have had their work ethic sabotaged by an inheritance given too early in the wrong form? This might be the biggest worry of the wealthy. Wealth means having enough money to screw up your kids' lives.
More information here:
Economic Outpatient Care and the Aspiring Millionaire Next Door
#2 Their Responsibilities to Society and the Planet
“With great power comes great responsibility,” taught Uncle Ben to young Peter Parker (aka Spiderman). The Notorious B.I.G put it differently: “Mo' money, mo' problems.” Somebody once asked me if I wanted to be a billionaire. I said no. I didn't want that kind of responsibility. When you are in the working class, you lay awake at night worrying about the rent payment. When you're in the middle class, you lay awake at night and worry about the mortgage payment. When you're in the professional class, you lay awake at night and worry about your student loans and practice loans. When you're Bill Gates, you lay awake at night and worry about the next pandemic and whether you're doing enough to prevent it. Who needs that kind of pressure?
#3 Personal Risks
Billionaire is an insult these days, and millionaire is heading in the same direction. The wealthier you become, the more you worry about your own personal security and safety. In the early days of this website, we were hyper-transparent with how we made money and how much. Then, I started worrying about my kids getting kidnapped. Things got a lot more vague after that. Think about your favorite celebrity and their encounters with the paparazzi. Then read Tim Ferris's 11 Reasons to Not Become Famous. Tim isn't exactly Beyoncé or Prince William, but what he has encountered is frightening. It comes more due to fame than wealth, but they often go together.
In addition to security, there are health risks. Would you trade places with Warren Buffett? I wouldn't. Warren Buffett is 94 years old. His life expectancy is what . . . just a few years? I'm 49 years old. My life expectancy is, what, 35 years? In essence, I'm turning down more than $3 billion a year ($114 billion/35 years) by not making that trade. Seems like a pretty good hourly rate. But health and life are more important than money, and the wealthy know it and should act like it.
#4 How Your Favorite Charities Operate
Giving is one of the five money activities to master in your life. But if you're giving $100 to a charity, you're not going to affect how that charity operates. What happens if you give them $1 million, though? Or $100 million? Most charities aren't set up to deal with that sort of a problem. So, wealthy people face a dilemma. They can support a gazillion charities with small donations each (introducing massive complexity to their giving program). They can not give at all (aka the Phil Demuth plan). They can only give to huge charities that can handle donations of their size (there's a reason Warren Buffett primarily gives to the Gates Foundation). Or they can get involved in the charity and help it to manage that sort of money.
Either way, this is a problem the wealthy have to deal with, and it starts earlier than you might think. We were going to give a $10,000 donation one year to a charity whose work and ideas we liked, but then we looked at its total fundraising for the prior year and realized that we would have been a huge percentage of it. We still gave but in a smaller amount. We didn't want that charity to change how it operates in expectation of that sort of donation coming in every year.
More information here:
Charity — How to Give, Why to Give, and the Tax Benefits You Can Receive
#5 Estate Taxes
Speaking of charities, if you are wealthy and your favorite charity is not the US government, you're going to need to do some estate planning to decrease or eliminate your estate tax. Federal estate tax is only paid by the wealthy. The exemption under the current law is $13.61 million ($27.22 million married) [2024]. If you have that much or expect to have that much when you die, you should start your estate planning process as early as possible. Most other people can get by with a will and maybe a revocable trust.
#6 Asset Protection
The more assets you have, the more you worry about protecting them and the more complex your asset protection plan becomes. For many people, including most white coat investors, a completely appropriate asset protection plan consists of umbrella and malpractice insurance, titling your home properly, and maxing out your retirement accounts. That's not going to cut it for the wealthy. Irrevocable trusts, multi-member LLCs, family-limited partnerships, and even overseas trusts may have a place. The cost of these relatively expensive asset protection techniques becomes trivial in comparison to net worth.
#7 AUM Fees
Asset Under Management (AUM) fees really start to matter when you have a lot of money. You can find someone to do your financial planning and manage your investments for under $15,000 per year. At 1% a year, a $1.5 million portfolio gets you there. But if you have a $10 million portfolio, a 1% AUM fee is a ripoff ($100,000 a year). For $100,000 a year, you should be getting family office type services. Maybe you're sharing them with multiple families, but by the time you have a $50 million net worth, you can probably afford your own family office. If you're not negotiating AUM fees down as your assets go up, you're being taken advantage of, and there is no level of wealth at which that feels good. Rick Ferri will tell you that reasonable AUM fees for investment management of large amounts should get down into the 0.2%-0.3% range.
Wealthy people have different worries than the rest of us. They have different problems, but they are still problems.
What do you think? What do you worry less about as your wealth has grown? What do you worry more about?
Rick Ferri will tell you that reasonable AUM fees for investment management of large amounts should get down into the 0.2%-0.3% range. What would you consider too much as an actual amount at that suggested percentage 50k, 60, 100k. More or less than that?
Any AUM fee greater than Blutarsky’s GPA is too high.
I’m not sure, but I’ve been thinking about it a lot lately. But I don’t think it’s crazy to charge something more than $5-15K to someone with $50M being managed.
One other worry for the wealthy: Cognitive Decline.
As a DIY investor for 40 years, I’ve recognized the need to outsource our wealth mgmt at some point. I’m currently 61, and am planning on migrating our investments to a planner I respect by my mid-70’s. An added bonus – my wife won’t have to worry about it if I die before she does.
That one is a worry for everyone!
Everyone needs to read that link to Tim Ferris’s blog entry, and also Gavin de Becker’s book The Gift of Fear. Fame definitely falls into the ‘be careful what you wish for, because you just might get it” category!
Awesome article.
I used to do my taxes myself way longer than anyone in a similar situation would’ve. Finally I got an accountant. But I would spend a couple hours reviewing every line and discussing with him many questions so I’d understand it all. Last year I just glanced through it briefly and paid my balance with essentially no questions.
I still do backdoor Roth on principle every year but it’s becoming a moot point so o don’t know how long I’ll continue it. I stopped tax loss harvesting multiple times a year. I still will if there’s a big bear market everyone talks about but I stopped following the daily market changes. Just doesn’t matter.
And the issues you discuss that do matter are insightful. I really want to fly first class with my family. And cost would be negligible. But I don’t want my kids to expect that so we fly in the back of the plane usually. Spoiling kids is probably the biggest concern. Along those lines though if you feel that way, estate tax isn’t really a big concern. That exemption is so far above what I’d want relatives to have that it can be handled with just giving most of it to various charities.
Really good article though. It’s a blessing to be in this position and we never thought we would be but deciding what charities to give 6 figure donations to is a fun problem.
We just landed in South Africa last night. The kids did indeed ride in the back. But they’re old enough they don’t have to sit by their parents. They complained about our fancy seats/meals a couple of times and we told them next time we’d leave them home and use the money saved to fly Delta One instead of Premium Select.
The wealthy also do not worry about employment, since they do not need a job in order to pay living expenses. Those who want to work can take a job based on how they will enjoy it, not on amount of wages or moving up a career ladder.
I feel that truly wealthy people can take whole lot more risk (i.e, being close to 100% in equities) since their investment horizon can be much longer than their own life expectancy, therefore potentially improving their return
Sure. They can take as much as they want. Or they can take “enough”, invest it very safely, and then invest the rest very aggressively. If 100% cash works and 100% equities works you can choose anything you want and should no longer care all that much about asset allocation.
I think what happens is an awareness of one’s legacy. If your number is 10M that is surely enough for your lifetime. But what about your 4 kids that want to live in desirable HCOL cities where a 4k sq ft home in a good neighborhood with a view will be 4M each. Yes, there are different philosophies when it comes to this type of decision making. I suppose you’re talking about wealth far beyond this.
There’s also a component of competitiveness as well as satisfaction with maximally efficient investing.
I was just a regular doc in my early years. I made over 200k as a young attending, and that annual income went up over time. But I also worked a side gig, and I maxed all retirement accounts, both with the primary job and the side gig. I also invested in real estate. I pretty much saved and invested in all kinds of assets. While the growth rates on different types of investments have varied, after compounding for several decades, we now have a mid 8-figure net wealth. We spend on travel and more, but if I live a long and healthy life, we may well get to 9-figures prior to death.
We no longer need to think about what car we might buy, or what international vacation we might take. We spend time thinking about different things. How might we increase our annual charitable gifting from 6-figures to 7? How much should we leave to the kids and the grandkids?
And I stopped working clinically this year. Even though I never paid a dime in a malpractice suit after a long and busy career in a fairly high risk specialty, I just don’t want to create a complex asset protection plan to allow me to derisk my clinical practice malpractice exposure. Malpractice insurance only provides 2.3M in coverage. It doesn’t feel worth it to continue seeing patients, given the potential risk. Outside of medicine, we have 10 million in umbrella liability coverage, but beyond that, there is theoretical liability exposure on that side as well, say from a car accident.
Mo’ money, mo’ problems, yes. But good problems to have, nevertheless.
I’m curious why you put taxes in the “no worries” bucket but AUM fees in the worry bucket. Does it really matter whether someone with an 8 figure net worth pays 0.3, 0.5, or 0.8%, especially if the higher fees potentially lead to higher returns?
I disagree that higher AUM fees generally lead to higher returns. In fact, the opposite likely happens more often. We also have way more control over AUM fees than tax rates.