If you’re not worried about your Social Security benefits…you should be.

Did you know there are currently 13 states that tax your Social Security benefits? I didn’t either. Don’t be surprised if many more adopt the same tactics.

We keep hearing stories that Social Security is in long-term trouble. This is extremely bad news for the large and growing numbers of retirees whose future is growing more dependent upon this failing system.

According to a national survey conducted on retirees, 59% rely on Social Security as a major source of their retirement income and an additional 31% cite Social Security benefits as a minor source of their income. As for non-retirees, more than one-third expect Social Security to represent a major source of their retirement income and almost half anticipate it will be a minor source of the income they depend upon. In short, without Social Security, a large number of Americans are going to struggle just to meet their basic needs.

So if you’re in or approaching retirement you need to be aware that Social Security is in trouble, real trouble. Beginning around 2020, according to a report by the Social Security Board of Trustees, the program’s cash inflow including interest earned, will turn into a cash outflow due to the increasing number of boomers leaving the workforce, a falling worker-to-beneficiary ratio and longer life expectancies. By the year 2034, per the Trustees, the program will have used up the entirety of its spare cash, potentially leading to a 21% cut in benefits.

The conclusion you should draw from this is simple. If you’re young, get prepared. If you’re older, you should be scared.

Do older people like surprises? Not if the surprise is finding out their Social Security benefits are going to be reduced.

And that’s not the only surprise retirees won’t like. A majority of retirees will also pay federal tax on their Social Security benefits, which will further reduce what they get to keep.

In 1983, Congress enacted an amendment to the Social Security program allowing the Internal Revenue Service to tax up to 50% of the benefits of individuals earning more than $25,000 a year and couples with more than $32,000 in earned income. A second amendment was added in 1993 that further expanded what benefits were taxable. Up to 50% of a retirees’ Social Security benefits become taxable with earned income of $25,000 to $31,999 for an individual, and $32,000 to $43,999 for couples who file jointly.

Any earned income above these amounts would allow for up to 85% of a Social Security recipient’s benefits to being taxed. Ouch! Thank you Uncle Sam.

When these taxes were first enacted in 1983, they were designed to only affect about ten percent of Social Security recipients. However, because the income thresholds haven’t been adjusted for inflation over the past 33 years, more than half of all Social Security recipients now owe at least some tax on their Social Security benefits according to data from The Senior Citizens League.

And that’s just the beginning of the bad news! Did you know there are currently thirteen states that tax your Social Security benefits?

As you can see below, even though some states offer very high-income exemptions that should only affect a very small percentage of the population, four states in particular, have adopted the tax schedule of the federal government without exemptions. These four states are considered to be the worst for the purposes of Social Security taxation and are probably not the best places to spend your final years if you’re living on a budget.

So if you’re living in Minnesota, North Dakota, Vermont or West Virginia, my advice is to check with your CPA or attorney now.

The following nine states also tax Social Security benefits but these offer some degree of income exemptions along the way.

Montana

Montana’s income exemptions are akin to the federal schedule at $25,000 for individuals and $32,000 for couples who file jointly. However, the net amount retirees wind up paying in Social Security taxes in Montana should still be lower than what they’ll pay on the federal level or in the four states mentioned above.

Colorado

In Colorado up to $20,000 in income for persons between ages 62 and 64, and $24,000 for those aged 65 and over is exempted from taxation. More importantly, Social Security income not taxed by the federal government isn’t added back to the person’s adjusted gross income (AGI) for state income tax purposes.

New Mexico

In New Mexico Social Security benefits are only partially taxed. Persons under the age of 65 can claim a deduction of $2,500, with those aged 65 and over able to claim a deduction of $8,000. For senior citizens, the deduction limits are $28,500 in AGI for individuals and $51,000 in AGI for couples who file jointly. The tax rate on Social Security benefits ranges from a low of 1.7% to as high as 4.9%. 

Utah

Utah offers a retirement credit of up to $450 per person for those over the age of 65. These credits can be used to offset any Social Security taxes paid within the state. So if you’re under the age of 65, you’ll be offered a 6% tax credit on your taxable Social Security benefits.

Nebraska

According to AARP’s 2015 analysis of Nebraska, individual taxpayers whose AGI is below $43,000, and joint filers whose combined federal AGI is under $58,000 will be exempt from taxation. Nebraska also recently indexed its tax brackets for inflation, meaning simple cost-of-living increases won’t move Social Security recipients into a higher tax bracket in subsequent years.

Kansas

Most Social Security beneficiaries in Kansas are going to be able to avoid paying state tax on their benefits. The reason is that Kansas exempts taxing Social Security benefits on persons with AGIs below $75,000.

Missouri

Missouri offers the heftiest exemptions among those states that do tax Social Security benefits. Individuals earning up to $85,000 in AGI and married filers with up to $100,000 in AGI are exempt. Furthermore, if you exceed the exemption you may still qualify for a partial exemption.

Connecticut

Connecticut offers some pretty generous Social Security tax exemptions but considering that the state also features one of the highest annual incomes in the country, a higher exemption limit makes sense. Individuals can earn up to $50,000 in AGI and married filers up to $60,000 and be exempted from paying Social Security taxes within the state.

Rhode Island

Next to Missouri, the exemptions in Rhode Island are the second highest in the country (of the 13 taxable states). Rhode Island exempts single filers with less than $80,000 in AGI and couples with up to $100,000 in AGI from paying tax on their Social Security benefits.

As you can see from the above, where you retire matters. Choosing to retire in one of the 37 states that doesn’t tax Social Security benefits can be just one small step in ensuring that you minimize what you’ll hand back in taxes. To some extent, states like Missouri, Rhode Island, and Kansas offer generous exemption levels that most beneficiaries will fall under, meaning they could still be smart retirement options.

However, retirees considering any of the other states should give strong consideration to the consequences of what those extra taxes could do to their monthly or annual cash flow before making the move.