After those who leave Japan, hand in their gaijin cards and apply for their 2.4 month refund, the remaining millions of yen they've paid into the system stay in Japan, where the money is used to pay pension payments for those they left behind.
The Japanese pension system is essentially run on the basis that what you contribute now is not necessarily what you receive in the future.
An inability to benefit doesn't remove your obligation under the system. Rather, successive generations are expected to provide for those that came before. What this means is that temporary foreign residents and working-age Japanese are essentially paying for the pensions of older Japanese.