While lazy actually means being averse to work or avoiding it, it can also be used to describe something slow-moving or unenergetic, like when you stroll down the garden path with a lazy gait, not in a hurry to get anywhere.
The adjective lazy is thought to come from the Low German lasich, meaning "idle or languid." You were offended at being called lazy, but you just didn't have the energy to defend yourself. After all, who doesn't enjoy a lazy Sunday afternoon? Former President of Poland Lech Walesa once considered the benefits of being lazy when he said, “It's the lazy people who invented the wheel and the bicycle because they didn't like walking or carrying things."