A person who just can't seem to fit in, no matter how hard they try, is called a misfit. If all of your friends are great dancers, but you are awkward and uncoordinated with no sense of rhythm, you might feel like a misfit.
Often, people who are misfits as kids eventually find other misfits to be friends with: suddenly, because they've found a group that accepts them and makes them feel comfortable; they're not misfits anymore. In the early nineteenth century, the word misfit meant "a garment that fits badly," but by the end of the 1800s it had come to primarily mean "outcast" or "oddball." The fit in both cases is wrong, thus the prefix mis-, or "wrong."