The boss's remark was dumb. But, we all say dumb things sometimes. Often we don't realize how our remarks affect others. I believe that s/he was trying to give you some guidance, but it was not done in a productive way. Please take these words with a grain of salt, put the sting aside, and focus on the fact that there was probably a good intent there.
To turn this into a positive experience, ask people you trust and respect for constructive feedback on your wardrobe. They may be able to give you helpful suggestions that you can benefit by.