Use the adverb regretfully to mean "with regret" — the sorrow you feel about something that has already happened. For example, little kids would regretfully turn over the candy bars they stole from the grocery store.
If you're sorry about something you've done or said, you might speak regretfully about it. Your feeling of regret or remorse is reflected in your actions, like when you regretfully recall something you said to a friend who was very hurt by your words. Don't confuse regretfully with unfortunately, which doesn't carry the same weight of personal responsibility — unfortunately, it's raining and you regretfully forgot your umbrella.