An ivory tower is a place — or an atmosphere — where people are happily cut off from the rest of the world. It's common to talk about colleges and universities as intellectual ivory towers.
If you're in an ivory tower, you live in a world of ideas separate from the realities of most people's lives. Your university philosophy department might be your ivory tower, where discussions about theory are the most important things in life. It's a privileged situation to be in, since ordinary people tend to have more mundane priorities, concerns, and struggles. The phrase most often describes academia, and it's been around since the early twentieth century, from the French tour d'ivoire.