What is the drawback of not deleting inactive device tokens returned by APNS feedback service?

The obvious problem is I will be attempting to send push to inactive devices that won't receive push anyway, therefore wasting my traffic and possibly degrading performance.

But let's say I am willing to temporarily live with this because I need to fix other issues first. Is this OK? I am worried Apple may cut you off if you don't take care of it and keep trying to send push to non-existent devices. More specifically, I don't know how to interpret this sentence in their documentation:

APNs monitors providers for their diligence in checking the feedback service and refraining from sending remote notifications to nonexistent apps on devices.

(https://developer.apple.com/library/ios/documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/Chapters/CommunicatingWIthAPS.html#//apple_ref/doc/uid/TP40008194-CH101-SW3)