A fire early Sunday destroyed the residence of the late Prime Minister Shigeru Yoshida in Oiso, Kanagawa Prefecture, one of the most prominent venues in modern Japanese politics and a place where Prime Minister Taro Aso spent some of his youth.