It’ll happen, eventually. Every business runs into the “problem client.” This is the customer who you find yourself wishing you could be brutally honest with, explaining that if he or she would just stop doing A, B or C, everyone would have a smoother time.

When you vent to friends and family members, they say things like “Why don’t you stop working with that client?” and you usually toss off some reason having to do with needing the money.

The truth is that for some clients, no amount of money in the world is worth it for the problems they cause. The question for the entrepreneur in charge becomes "How do I know when it’s time to let a client go?"

Here are four scenarios that might warrant letting go of a client:  

Related: When to Fire That, Er, Abusive or Disruptive Customer

1. You feel irritated and you know exactly why.

Perhaps the customer pays late (every month, even after being reminded and being charged late fees). Perhaps the client reminds you of your estranged Uncle Ralph. Maybe the client is always asking for a little something extra, nickel-and-diming you on every new proposal. Whatever the reason is, you know exactly why you feel prickly around this customer.

The good news? If you can pinpoint the reason, you might be able to change things. As the person in charge, this is both the best and worst part of running a business: You get to call the shots. It’s worth it, both from a business and a life perspective, to use these customer situations as an opportunity to practice healthy boundaries: Try, “I’m afraid that if I receive even one more late invoice, I won’t choose to continue working together” or “We stand behind our work and we don’t offer discounts.”

When you’ve identified your bottom-line boundary and expressed it to a client, everyone knows where you stand. If the client doesn’t agree, it’s time to say goodbye. 

Related: Could Hiring Snobby Salespeople Boost Your Sales?

2. The customer is rude to you or your staff.

Clients who behave in an ill-mannered way tend to operate from the perspective that they are the only game in town and you’re lucky to work with them.

The truth? No one is the only game in town. When a customer treats your employees this way, it’s critical for you to step in.

If you’re interested in trying to turn things around, you might be able to clue the client in to his or her behavior with one simple statement: “When I heard you say that, I felt disrespected.”

Then let the silence hang in the air and wait for the client's response. Most people who are disrespectful are unaccustomed to someone's pointing it out. Making someone aware that it bothers you might be enough to prompt a change.

If giving voice to this doesn’t work, then it’s time to let the client go. Disrespectful clients are never worth it, no matter how much they might add to the bottom line.

Related: Don't Fire These 3 Rotten Customers, Profit From Them

3. The customer is lovely but takes a lot of time.

Clients who need a lot of hand-holding often have unexpressed needs and fears that, if identified, could be lessened with better communication. Strong boundaries are critical with these customers.

For instance, a client who has all the information he needs in an email but who constantly wants to get on the phone only to rehash the same details is probably someone who needs a lot of reassurance upon seeing a project laid out. If you send the customer a detailed PDF with visuals instead of an email, he or she might have a better idea of what you’re doing and feel less needy.

If the customer still needs a phone call after that, you might say, “I’m unavailable for a call this week, but I want to make sure I answer your questions. Please email me about anything that wasn’t discussed in the PDF that I sent you.”

Alternatively, you might consider keeping the client but raising your rates so that your team is compensated for the additional time.

If the client continues to demand more and more of your time, then be fair: Let the client go, so that he or she can find someone who better meets their needs.

Related: Breaking Up With a Client: What to Say

4. You feel resentful but you have no idea why.

The trickiest scenario is presented by the relationship you ruminate about, trying figure out why you don’t like working with the customer involved but can’t quite put your finger on it. Because you can’t pinpoint the issue, you rationalize that everything must be in your head, and you continue to work with the client.

Days, weeks or months later, the same old irritated resentment comes up, and you find yourself wishing you’d cut the client loose -- and you would, you tell yourself, if only you could justify it.

In these situations, the ultimate answer is to let the client go. Why? Because the mental justification is costly. It costs you peace of mind, sleep and mental energy that might otherwise be devoted to something more beneficial to your business. Sometimes, you don’t need a reason. It’s just not a match, and there’s not always a logical reason.

It’s always hard to let go of a client, but when a relationship has gone sour, it’s time to take decisive action. The client doesn't fare well if you aren’t genuinely passionate about working for him or her. When you release what isn’t working, including the clients who no longer seem exciting for you to work with, your business can grow in new directions.

Related: Don't Let Your Business Be Held Hostage by a Nightmare Client