To rile is to stir up, like water that becomes muddy when you rile, or to bother or disturb, especially with little annoyances. For example, if you bring up an unpleasant subject, this might rile your friend's mood, or a noisy neighbor might rile your nerves.
The verb rile is derived from roil, and they basically mean the same thing. However, rile is more commonly used to mean "annoy" and roil mostly describes stirred liquid. If you rile up your sister, what you say makes gets her worked up, ready to argue with you. When a kid splashes around in the tub, the water is roiled — but if it gets all over the floor, his parents will be riled.