I noticed that components respond to the force:refreshView event long after they have been removed from view. I wrote some test components that I placed on the Contact, Account, and Opportunity related list pages. If I navigate to a record of each type, they all continue to respond to the refreshView event. In fact, if I navigate to different accounts or contacts, each one continues to repsond to the refreshView event. I have found that it keeps a maximum of 4 components active in the background before actually destroying (unrendering) them.

In some components, we reload data on the refreshView event. Is there a way to know that reload of data is a waste of time?

Here is my test contact component:
<aura:component implements="force:appHostable,flexipage:availableForAllPageTypes">
    <aura:handler event="force:refreshView" action="{!c.refreshData}" />
    <ui:button label="Fire" press="{!c.buttonClick}" class="slds-button slds-button--neutral" />
</aura:component>

controller:
({
    buttonClick : function(component, event, helper) {
         $A.get('e.force:refreshView').fire();
    },
    refreshData : function(component, event, helper) {
        alert('Contact data refreshed');
    }
})

The other components simply have the alert text change to reflect the location of the component.