Swarthy means dark skinned. If you like tall, dark and handsome men, you find a swarthy complexion attractive.
Not everyone with dark skin is swarthy. The word is usually used to describe someone whose skin is weather beaten and darkened by the sun, or has an olive complexion. But the famous 19th-century American poet Walt Whitman, who was of English and Dutch stock, seemed to be describing himself in his poem "Behold This Swarthy Face," so swarthy may be in the eyes of the beholder.