Vexation is both something that causes annoyance and the state of mind that results from being annoyed. The test-taker next to you tapping her pencil is a vexation. You breaking her pencil in half makes her feel vexation.
Vexation can also refer to something that causes anxiety and worry more than annoyance. When the parents of the kid you're babysitting are two hours late to return and aren’t answering their phones, that could be a vexation. You are less annoyed than worried. (Though, to be honest, you're still a little annoyed).