"Oh, no, are you kidding me? He's not my boyfriend, we're just in a casual relationship," means that our relationship lacks serious intent or commitment.
A synonym for casual is "everyday", meaning nothing special, unusual, serious, or eventful. If you're going to your friend's house for a dinner of burgers and beer, they might say, "It's just a casual night at home." Or, "Dress casual, we're just going to be serving ourselves." Don't run out to buy a special outfit, because what you're already wearing is probably just fine. It's what you wear every day, all the time; it's casual. If you treat a person or a subject in a casual way, you're not paying close attention or treating them as special. That's not always a bad thing, by the way.