Use literary when you want to indicate writing with high artistic qualities. Something doesn't have to be "literature" to be literary, but they are related.
Literary comes from the Latin litterarius, meaning "letters," as in letters of the alphabet. Think of literary writing as writing that explores the richness of language or even contributes to it, especially in a high cultural way. The clever rhymes you invented about your grade school principal's bathing habits made your classmates laugh, but that doesn't make them great literary achievements.
