Primary means basically "first." When you vote in a primary, that is the first election in a series. When a matter is of primary concern, it means it's of first importance. Primary school is the first you go to (after nursery school, at least).
There's an interesting alternate system for counting first, second, third, etc. up to tenth. It's primary, secondary, tertiary, quaternary, quinary, senary, septenary, octonary, nonary, and denary. There's also a word for twelfth, duodenary, though that — along with all the words after tertiary — is rarely used.