A former top U.S. diplomat on East Asia indicated Wednesday that Prime Minister Shinzo Abe should exercise restraint over a possible visit to the controversial war-linked Yasukuni Shrine in Tokyo, saying it could severely damage Japan's relations with its Asian neighbors.