The hoi polloi is a way of referring to common people, and it is an elitist term usually used by people who consider themselves to be above the masses.
Snobs rarely like the hoi polloi — that is, the masses, a.k.a. "the great unwashed." The term comes from the Greek for "the many." Perhaps because of the similar sounding term hoity-toity to mean someone with fancy airs the phrase is sometimes mistaken for its exact opposite — the elite or upper class. Don't be fooled.