Vermin is a catch-all term that can be used for any small animals or insects that we think of as pests. The cockroaches you found in your bathroom were the grossest vermin you'd ever seen.
The word vermin originally came from the Anglo-French word of the same spelling, meaning "noxious animals" — definitely a negative term. Rats, mice, cockroaches, along with pests to farmers and the animals they raise, like coyotes and weasels, are vermin. Annoying people who cause problems are also sometimes called vermin, like the vermin who bully others.