To budge is to move — but just a little bit. People can budge physically from where they're sitting or standing, and people can budge from opinions and positions too.
If a bunch of your friends are sitting on a bench, and they're taking up too much space for you to sit, you could ask them to budge, meaning "Move over!" If you hate eggs and refuse to eat them no matter how they're cooked, you are refusing to budge from your anti-egg policy. Budge is most often used in a negative way, as in "I won't budge" or "They wouldn't budge." This word often applies to stubborn people. Since budge means to move just a tiny bit, what's the big deal? Someone who accuses you of not budging is basically saying you should.