To antagonize is to provoke someone to react angrily. If your mother tells you to stop antagonizing your brother, she's asking you to stop making him mad.
Antagonize shares the same Greek roots with the word agony, so think about how much agony you cause your siblings when you tease and goad them into getting angry. Bugs Bunny is known to antagonize Elmer Fudd, who hunts but never does catch that wabbit. Actually Bugs antagonizes every cartoon character he comes into contact with! In English class, you'll often be asked to name the antagonist of a story or novel, the character who provokes the main character to act.