A feeling of being extremely irritated or exasperated is annoyance. Your neighbor's annoyance is completely justified if you've been letting your dog bark for hours in the back yard.
You can describe the general state of being annoyed as annoyance, and it's also a good word for the thing (or person) that's annoying you. While your neighbor may refer to your dog as "that annoyance next door," from your perspective it's your neighbor who's the annoyance, constantly scolding you about your barking dog. The Latin root of annoyance is (esse) in odio, "(it is to me) hateful."