To be averse to something is to be opposed to it on moral, philosophical or aesthetic grounds: my father is averse to people smoking cigarettes in the house, but he would not be averse to your smoking a cigar.
Averse comes from a Latin word, aversus, which means "turned away from." To be averse to something is to shun it, turn away from it, or dislike it. Some people are averse to the use of cilantro in any food that comes near them, some others are averse to wearing itchy wool sweaters, and others are averse to any form of exercise. Aversions are really very personal.