Adverse and averse are both turn-offs, but adverse is something harmful, and averse is a strong feeling of dislike. Rainstorms can cause adverse conditions, and many people are averse to rain. 

Adverse describes something that works against you, like a tornado or a computer crash, and is usually applied to things. It's often followed by the word effects:

More significantly, he has shown that if such ageing cells are selectively destroyed, these adverse effects go away. (Economist)

The pact was intended to limit the adverse effects of climate change but only obliged developed countries to reduce greenhouse gas emissions. (Scientific American)

Averse is usually applied to feelings, attitudes, or people. It's a strong feeling of opposition — it's a big "no thanks" and it's often followed by to. Averse also goes with risk to describe people (or banks) who don't like taking them:

Balth isn't averse to including human beings in his work. (Seattle Times)

Nevertheless, Ms. Fishbein is not averse to a large sociable gathering. (New York Times)

Your survey shows that banks are more risk-averse than they used to be. (Business Week)

If it's a force of nature working against you, use adverse. Kick out the "d" and you can be averse to or against anything, like rainy days or people who wear shoes in your house.