A sty is an outdoor home for a pig. If your sister keeps her clothes in heaps on her bedroom floor and leaves empty cereal bowls on every available surface, your mother might complain that her room is a sty.
An actual pig sty, also called a pigpen, is typically a fenced-in area of mud or dirt. Although pigs aren't particularly dirty animals, they do need water or mud to stay cool, and they are naturally inclined to chew plants and root through dirt with their noses. The word sty is also commonly used to describe extremely messy human homes. Sty can also mean a swollen infection of the eye.