When something is insignificant, it doesn't matter. The fact that your mom gets to school a half hour late becomes insignificant when she announces you're stopping for ice cream on the way home.
The word signify, which is at the heart of insignificant, means "to mean." Significant means "meaningful." Add in- "not," and you have "not meaningful." A company can fire a worker from an insignificant job and still function. In a crisis, your feelings are insignificant; it is your actions that matter.