The nearly impossible puzzle your math teacher gives for extra credit can be called a teaser — a tricky problem to solve.
A mystifying problem is one kind of teaser — another is a person who loves to tease. Your nose-grabbing, practical joking uncle is a teaser, especially if he enjoys gently mocking or making fun of you. Yet another teaser is the kind that opens a TV show — sort of a sneak preview of the story to come. This is closest to the original meaning of teaser, from 1934, "an introductory advertisement or sample."