A tenter is a wooden frame used to stretch wool after it's woven into cloth to prevent it from shrinking.
Tenterhooks are the metal hooks that hold the cloth to the frame. To be on tenterhooks means to be stretched tight or to be tense while in anticipation or suspense. The phrase is often preceded by "waiting," which emphasizes the quality of anxious uncertainty, a kind of worry that you feel physically. That's what makes the image so powerful.
These examples illustrate the tension inherent in the expression:
In the laboratory, I was always on tenterhooks, thinking "Today could be the day we get the call." (Nature)
Harry had his wand raised, waiting on tenterhooks to repel a jinx that seemed unlikely ever to come (Harry Potter and the Half-Blood Prince)
On tender hooks is a mishearing of on tenterhooks. The phrase tender hooks doesn't really make sense, though: hooks are sharp and pointy, not tender! (Tender means "gentle, sensitive, or loving," and it can also mean "easy to chew" or "sore.") Tenterhooks are real things, even if people outside of the textile industry might not be able to recognize them. Tender hooks don't exist.
The first example below shows a simple mistake. The second example comes from a comical poem stuffed full of incorrect expressions!
I was on tender hooks all the time while you were sleeping. (A Columbus of Space)
Let me cut to the cheese:
every time you open your mouth,
I'm on tender hooks. ("On Tender Hooks")
As with many such commonly misused expressions, understanding their literal origin can be helpful in remembering what's correct. Tenterhooks are real, sharp, L-shaped metal hooks arranged around the perimeter of a wooden frame for stretching cloth tightly. Tenter comes from the same Latin root as our word tent, which is another kind of cloth stretched over a frame — so use that relationship as the "hook" that helps you remember this commonly misheard expression.