How Do You Know When It's Time to Quit?
I suggest you get help from your nearest Small Business Development Center, where you can get workshops, classes and individual counseling related to developing a good business plan and figuring out how to either grow the business, or sell it. Further, you're talking about a deep long-term relationship with your brother, and that's a tough issue, too.
It does sound like you need something to change, and that you want that, and your brother doesn't. It's time to talk, get help, and fix things.