A pigeonhole is a small nesting area for a pigeon, or a similarly cozy compartment or alcove. If you feel like you're stuck in a restrictive category — only thought of as someone's sister, for example — that's another kind of pigeonhole.
A cubbyhole in your desk is one kind of pigeonhole, and the little box where a domesticated pigeon sleeps is another. These original meanings date from the sixteenth and seventeenth centuries. By the 1870s, pigeonhole had also come to mean "an overly simplistic or restrictive category," and also "to assign someone to a rigid category." So when you assume that a circus clown couldn't also be a brilliant mathematician, you have pigeonholed him.