If your brother calls you a puritan, then he’s saying you’re very moral — possibly too moral. He’s implying that you’re intolerant and look down on others who don’t have your standards.
The Puritans were a group of English Protestants that formed in the 16th century to bring about religious reform. The Puritans wanted to “purify” the church by following intensely strict religious principles, which earned them the name Puritan. Puritan can still refer to the religious group — in that case it’ll have a capital “P” — but nowadays you’re more likely to hear it used to describe someone who follows a strict moral code and shuns almost all pleasures.