The adverb hardly means barely or scarcely at all. If you hardly ever visit your cousins in California, it means you almost never travel to see them.
If you hardly know your neighbor, you only know him a little bit, and if you hardly understand what your French teacher says, you're only catching every few words she speaks. Hardly can also be used to talk about time: "We can't leave yet — the party's hardly started!" Hardly originally had nearly the opposite meaning: "with great exertion or effort." The phrase "not hardly," or "a little," eventually lost the "not" and became the hardly we know today.