Pssst... do you know the secret handshake? If you haven't been brought into the inner circle of those with special knowledge, esoteric things will remain a mystery to you.
In the olden days, achieving esoteric knowledge meant getting initiated into the mystical arts, learning secrets unknown to regular folks. Now when a subject is called esoteric it's usually something not so mystical but still hard to penetrate: financial accounting might seem esoteric for people who get easily stumped filling out their tax forms. Americans might find the sport of cricket to be esoteric, but the rules of baseball can be just as impenetrable to outsiders. The infield fly rule? Totally esoteric.