If something is ornamental, it's a decoration. The red feather in your cap is most definitely ornamental; it serves no purpose other than providing a splash of color.
Ornamental things are, by definition, just for show — they're not actually useful. So an ornamental pond in your yard might be pretty, but you can't fish or swim in it, and an ornamental telephone booth in your friend's living room doesn't really work. Likewise, ornamental plants or gardens are full of beautiful flowers and foliage, but nothing in them is edible.