No one can tell you whether it's time to quit based on your email alone. From what you say here, staying with the business could be the worst possible decision, and leaving it could be the worst possible decision.

I suggest you get help from your nearest Small Business Development Center, where you can get workshops, classes and individual counseling related to developing a good business plan and figuring out how to either grow the business, or sell it. Further, you're talking about a deep long-term relationship with your brother, and that's a tough issue, too.

It does sound like you need something to change, and that you want that, and your brother doesn't. It's time to talk, get help, and fix things.