Things that are inoffensive are perfectly pleasant — they don't disturb or upset anyone.
Your next door neighbor may annoy you with his late-night drumming sessions and wild parties, while your neighbor on the other side is completely inoffensive, living a quiet life and waving to you each morning in a friendly way. Things that bother, upset, or offend you are offensive, from the Latin root offendere, "to strike against," or "to displease or provoke." When you add the "not" prefix in- to it, you get inoffensive.