Officials in charge of staging the Tokyo Olympics crowded around a low table inside Prime Minister Shinzo Abe's residence late Tuesday, wincing as they spoke by phone with International Olympic Committee President Thomas Bach.