There's a lot of chatter about culture today, with Glassdoor releasing its rankings of "best" company culture. Indeed, culture is an important part of leadership. As a manager, you need to set a tone and direction for your employees, give them a sense of purpose and community, and make that tangible enough for your customers and business partners to feel, as well.

But there needs to be more discussion of the darker side of culture. Good, strong company culture can also be exclusive and discriminatory. And you may not even know it.

First, a caveat: This is not a screed on how tech companies or Silicon Valley or U.S. corporations need to "do more" to increase hiring of women and minorities. Those moves have been well-addressed, perhaps over-covered in some quarters. The market, not quotas, should always determine who gets hired and who doesn't. I still believe technology companies are not getting the credit they deserve for showing leadership in these areas.

Yet, many company-culture programs tilt into steath discrimination. Perhaps the best column written this year on the problems inherent in "culture" was by former Facebook engineer Carlos Bueno, who noted he probably couldn't get hired nowadays because of the cultural code words used in employment.

"We’ve created a make-believe cult of objective meritocracy, a pseudo-scientific mythos to obscure and reinforce the belief that only people who look and talk like us are worth noticing," Bueno wrote for the website Quartz. "After making such a show of burning down the bad old rules of business, the new ones we’ve created seem pretty similar."

There is much truth in that, even beyond Silicon Valley. Companies are careful to make sure they check off the right boxes when they are hiring, ensuring nothing they say indicates they are making decisions based on race, religion, sexuality or gender identity. That would run afoul of the law, after all.

Related: The One Executive Position Every Startup Now Needs to Fill Today

But there are intangibles that are just as exclusionary. For instance, I recall having lunch with the executive team at a midwest financial-services firm. All were very fit. "Health is a big part of our culture," the CEO told me. "If you don't work out outside the office, you won't work out inside our office." Chuckle, chuckle. It was a line so well-rehearsed that I imagined it being painted on the hallways of the company's offices. But the flip side to that, particularly to a guy like me who struggles with an addiction to osso buco with saffron risotto, was that I should not even bother applying. (For the record, I wasn't. But I still ended up ordering a chopped salad.)

Such discrimination used to be more overt. In my office, I have a wooden sign that says, "No Irish Need Apply." It is a reminder that it was not that long ago that an Irish-Catholic couldn't break into the Episcocratic workforce of New York City. In the end, we got jobs as cops. And gave the world the Paddy Wagon.

But you can't be overt about discriminating anymore. You have to promote your commitment to diversity, and tout how much you're paying for programs that get more minorities into coding, or more women into sectors they never explored before. You advertise your same-sex partner benefits, and have photo ops every time you re-pave your wheelchair ramp.

And all the while, how many of you have your fingers crossed behind your back because of your commitment to preserving "culture?" You hire and promote based on intangibles. The danger is two-fold. First, you may be missing something. If you only like job candidates with MBA's, you run the risk of hiring a well-educated, yet dreadful manager, prejudging that someone who eschewed academia could never be the leader you want.

Related: When Will Washington Start Unleashing the Power of Entrepreneurs?

Worse, you might be getting exactly what you're looking for. When you have an intractable view of whom you want to employ, you run the risk of creating a homogeneous environment. On the surface, you might look like all colors of the rainbow, a diverse mix of genders and races, all in compliance with applicable state and federal laws. Inside, though, you suffer from a lack of diversity of ideas or backgrounds. You are all the same on the inside, and that isn't as good as it sounded when we were in kindergarten.

Many leaders who tout culture don't even know there's a problem. After all, culture is a metric that comes from evaluating the opinions of employees themselves. They are already in, among the chosen. Left unevaluated is the opinion of those who were otherwise unanointed. When it comes to culture rankings, you are voted on by an electorate you hand-picked.

The free market demands a different approach to team-building. If you are spending so much time worrying about culture that you create hiring programs to weed out the wrong candidates to the exclusion of the right ones, you will fail. Customers are a diverse lot. You need diverse ideas and backgrounds to figure out how to reach them. Sure, there are niche businesses, but most leaders want to grow outside their niche into something larger, more meaningful and more profitable. That takes a more diverse workforce. A women-owned business that only markets to other women-owned businesses is missing out on a revenue opportunity. A web-development company that markets itself on the strength of a team that all met in the same fraternity at Penn State is likely to alienate a big chunk of its customer base, as well. Neither is good business.

I'm not advocating for quotas, or cheering on the shame-game crowd, which is demanding quantitative changes for certain classes at businesses, particularly in the Valley. Those are just as anathema to the free markets as the steatlh discrimination I fret over. But this blind celebration of culture needs to be re-evaluated. This may be an instance where many companies find they have to destroy the village in order to save it. Culture can be a wonderful selling point to attract new employees and customers. But it might also be a cultural hegemony as corrosive as saying you only want a third of your workforce to be women. Think of that the next time a fat Irishman who refuses to bike to work applies for a job.

Related: The Myth of the Have-Nots