When a person is spoiled, they're damaged by having been given everything they want. Spoiled people are usually pretty rotten. When food is spoiled, it's also rotten—literally.
Spoiled things and spoiled people are both fairly unpleasant. A spoiled child typically whines for something and gets it, becoming used to that kind of over-indulgent treatment. Spoiled milk smells terrible and tastes even worse. This adjective comes from the verb spoil, meaning "ruin" or "destroy"; the idea was that giving in to a child's every whim would ruin him.