Formerly is something that happened before, like when a pop star changed his name to a squiggle, he became known as The Artist Formerly Known as Prince. But formally comes from formal, or fancy, like the prom.
Former is at the root of formerly, referring to something that occurred earlier. If you changed your name, you'd be referred to as formerly known as whatever-your-name-used-to-be. It's not just for pop stars. If you want a word that means in the past or before, formerly fits the bill:
Movies, books, television shows, the journalistic outlets formerly known as newspapers, podcasts, YouTube videos, actual museums, tweets—they all comprise the noise. (Slate)
Erskine Bowles, formerly President Bill Clinton's chief of staff, said, "I'm worried you're going to fail." (Business Week)
At the root of formally is formal, an adjective that describes something as following custom, regulations, or ceremonies. You might speak formally during your speech, but chat casually with everyone after. Formally describes something official:
Delaware spent years piloting and fine-tuning their system before putting it in place formally state-wide. (Washington Post)
The current council will formally disband within the next few days. (Time)
Remember where these two words came from and you'll keep them straight: formerly former, and formally formal.