In 2003, I purchased a 1,000 square-foot, two-bedroom, two-bathroom condo in San Francisco for $580,000, with a 30-year fixed-rate mortgage of 4.25%. At 26 years old, I had put almost all my savings into the 20% down payment.

Don't miss: The best credit cards for building credit

Over the next few years, I became obsessed with trying to pay off my mortgage early.

In 2015, my wife and I finally did it. Eliminating one of our largest reoccurring expenses felt so satisfying. With one less burden, we could live life more freely.

But just a year later, that feeling went away — along with the fire to improve my finances.

The biggest downside to paying off your mortgage early

There are several studies citing the psychological benefits of paying off debt early, such as emotional relief and the strength to avoid slipping back into debt.

All of that is true. But the biggest downside that no one talks about is the complete loss of motivation to take on calculated risks and to work as hard as you can to grow your wealth.

After being mortgage-free, my and wife and I lived comfortably off the severance checks that we negotiated when we quit our six-figure jobs in finance (by that time, we had amassed a net worth of $3 million), and the $150,000 in annual passive income — mostly from real estate, dividend stocks and bonds.

But my entire attitude slowly changed once I sent that final mortgage check. I stopped aggressively looking for new freelance consulting work. I went from taking on three contracts per month to just one. So instead of working 60 hours, I was only working 20 hours. At around $10,000 per contract, I was losing out on $20,000 of monthly income.

To reward ourselves for paying off the mortgage, my wife and I also took a month-long trip to Asia. We visited friends overseas and saw the Ruins of Angkor in Cambodia, then spent weeks camping in Yosemite. After that, we flew to New York for two weeks to watch the U.S. Open. We spent more than $10,000 on the entire trip.

It was a lot of time off — so much, in fact, that I was behind by $50,000 on my goal to reach $200,000 in annual passive income. The plan was to grow our money so we could feel financially secure by the time we had a baby.

If I could go back, I would have spent less time vacationing, and more time buying rental properties and investing in dividend stocks.

A mortgage kept me hungry

I've talked to several people, including readers of my personal finance blog, Financial Samurai, who had similar experiences after paying off large amounts of debt early.

"I've definitely noticed that the closer I get to financial freedom, the less motivation I have to go above and beyond at my job," one person told me. "It's almost as if we need that monkey on our backs to keep us moving forward."

And then there are some who feel the opposite.

A reader on Financial Samurai commented: "I just paid off my house after eight years and four months. Now I'm extremely motivated to put as much [money] as possible into other investments, such as the stock market, my own business and rental property. I'm 35 and self-employed, so I don't have a pension or company 401(k) to rely on."

Of course, everyone is different. For me, a mortgage fueled my hunger. Without it, our monthly cash flow increased by $2,500 per month, and we saw it as extra spending money.

The most important lesson I learned from paying off my mortgage early is that you should always have something that keeps you motivated and financially driven. When you've got your living costs covered and things are easy, it's tempting to get soft and ignore your finances.

Only the most "gung-ho" people will bother to take risks or put in the extra hours to get that raise. If you have no financial burden and no one is depending on you to provide, it doesn't make sense to work twice as hard.

My best advice: Plan for the future

If you're focused on paying off your mortgage, good for you. It's generally always good to get rid of debt. Plus, with no mortgage, you get a guaranteed, risk-free return.

Just make sure you consider the downsides.

Aside from losing motivation, you also tie up capital in an illiquid asset when you pay your mortgage off early. Unless you have a very diversified net worth, having a lot of capital in the form of home equity can be a bad thing. Your home could collapse in the next storm or burn down in a fire.

And with interest rates at all-time lows, it might make more sense to refinance your mortgage into a low fixed-rate term for as long as you plan to own the property — and then invest the rest.

The right answer depends on your current situation, tolerance for risk and long-term goals.

My best advice is to pay off your mortgage by the time you no longer want to work. Figure out when you plan to retire and divide your debt amount by the number of working years you have left.

There are free retirement planning calculators to help give you a realistic picture of your financial future. After all, there's no rewind button in life. It's always better to over-plan than it is to under-plan.

Sam Dogen worked in investment banking for 13 years before starting Financial Samurai, a personal finance website. He has been featured in Forbes, The Wall Street Journal, The Chicago Tribune and The L.A.Times. Sign up for his free weekly newsletter here.

Don't miss:

• Always follow the '30/30/3 rule' before buying a home during Covid-19, says finance expert—here's why
• I retired at 34 with $3 million—here are 5 downsides of early retirement that no one tells you
• 42-year-old retiree: 'I lost more than $600,000 due to the pandemic. Now I'm second-guessing early retirement'

Check Out: The best credit cards for building credit of 2021