If your birthday party gets out of hand, it becomes wild and crazy. When something's out of hand, you've lost control of it.
A math class that gets carried away with excitement about the coming school vacation is out of hand, and your frizzy, wind-blown hair might get out of hand during a ride on a speed boat. This common phrase most likely comes from the importance of keeping a team of horses under control, or in hand, with a firm hold of the reins. A failure to do this would result in the horses getting out of hand.