Esteem is all about respect and admiration. If you have high self-esteem, it means you like yourself. When you say, "My esteemed colleagues," you are saying you have nothing but the highest respect for them.
Esteem derives from the same Latin word that gives us estimate, and back in the day, esteem, like estimate meant "to assess, or judge the value of something." That sense lingers today. When you say you hold someone in high esteem, it means you give them a high value. Unless you're a politician, in which case, when you say, "I hold my opponent in high esteem," you are most likely to follow that statement with a big "But...."