If your grandfather was so startled by people shouting "Surprise!" at his birthday party that he had a mild heart attack, you'd probably want to just forget the whole affair. An affair is a specific event.
You can use affair as a kind of shorthand, to mean "that big mess," or "that complicated story I already told you about." Your affair can also mean your responsibility or job, like if your grandmother said, "I don't care about balloons for the party — that's your affair." Another way affair is frequently used is to mean a love affair, particularly if the relationship is brief and intense. This meaning comes from the French phrase affaire de coeur, "affair of the heart."