At a construction site in Kawagoe, Saitama Prefecture, worker Fan Xiuyu says he’s too busy to miss the wife and 6-year-old child he left behind in China.

“I came to Japan to make money and learn advanced construction techniques,” said Fan, 29, a native of Taishan, who says his job making and installing metal ducts for Haruta Kogyo Co. pays him three to four times what he earned in his homeland.

“The working environment in Japan is much better than China,” he said. “It’s clean and Japanese colleagues are willing to teach me when I ask them for help.”

With a dwindling population, Japan needs more people like Fan to build and run venues and hotels for the 2020 Tokyo Olympics.

The nation imported about 50,000 workers annually over the past five years. But that needs to rise to 200,000, perhaps twice as many as the public would accept, a survey of economists shows.

To satisfy the demand, Prime Minister Shinzo Abe would have to break down centuries of resistance to immigration.

“Even 100,000 immigrants would do little to ease the labor population decline, and even that would be politically difficult,” said Yasunari Ueno, chief market economist at Mizuho Securities Co. “Immigration is unpopular, especially in the provinces. Abe still seems to flinch at the idea of accepting foreign workers on a large scale for fear of losing public support ahead of elections in 2015 and 2016.”

Japan is caught in an economic pincer. On one side is the declining population that saps the world’s third-largest economy of workers. On the other are some of the most restrictive immigration policies of any developed nation.

Japan will lose four out of every 10 workers by 2060, shaving as much as 0.9 percentage point off potential growth — more than half of last year’s expansion, Cabinet Office projections show. Most Japanese oppose accepting more foreign workers into the country to offset the decline, a newspaper poll in April showed.

Pressure to allow more foreign workers is increasing as a pick-up in the economy worsens a labor shortage, said Shuichi Obata, a senior economist at Nomura Securities Co.

Japan’s workforce shrank to 65 million in March from a peak of 69 million in June 1997, according to the Internal Affairs and Communications Ministry. At the current birthrate it would slide to 38 million by 2060, the Cabinet Office estimates.

“The government must act,” said Jun Saito, a professor at Keio University’s Graduate School of Business and Commerce in Tokyo. “Accepting foreign workers is the only option left.”

A decline in technical workers coupled with the increased number of projects has led to the worst shortage of construction workers since 1994, the Health, Labor and Welfare Ministry says. The government estimates foreigners will have to account for nearly half the 150,000 construction workers needed by the 2020 Olympics.

For those arriving from abroad, higher wages and better services help offset the difficulties of integrating into Japanese society.

“Some Japanese have a biased view against the Chinese and there were occasions when I was treated badly,” said Chinese national Lin Ruilian, 45. A Japan resident for more than a decade, she now runs a tiny acupuncture and foot massage shop near Tokyo’s Shinbashi Station. “But Japan is a good place to live. The food is safe and even civil servants provide good service. That is unthinkable in China!”

Kazumasa Iwata, a former deputy governor of the Bank of Japan, told the government’s Council on Economic And Fiscal Policy that Japan should accept 200,000 immigrants a year to help care for the elderly and children, as well as bringing new ideas to the country, according to papers submitted to the council, an advisory committee for the prime minister.

Foreign nationals account for 1.6 percent of Japan’s population, the third-lowest proportion after Poland and Slovakia among 26 members in the OECD that provide such data. In the U.S. the rate is 6.8 percent and in Switzerland 22 percent.

Japan’s efforts to open its immigration doors have been limited. The government last month allowed construction workers who finish a three-year technical training program to work in the country for another two years.

“This is a very important first step,” said Hidenori Sakanaka, head of Japan’s Immigration Policy Institute and a former Justice Ministry official. “This could be a touchstone before introducing controversial but necessary large-scale immigration.”

The public remains unconvinced. While a Yomiuri Shimbun poll of 1,512 people showed that 79 percent think the population decline will have a negative impact on the country, 54 percent said they don’t think Japan should accept more foreign workers.

The plan to import more laborers has spawned new anti-immigration groups such as Sakurano-kai, or Cherry Blossom Party, which said it has collected more than 16,500 signatures opposing an increase in annual intake to 200,000.

“It makes Japan a less comfortable place to live for Japanese people,” the group said in an emailed statement. “Immigration is irreversible. Once you accept foreigners, it is impossible to kick them out.”

Part of the resistance stems from a scarcity of foreign residents and cultural barriers rooted in a two-century isolation policy under the Tokugawa Shogunate, which banned most immigration until 1853.

Anxiety among local residents, competition for jobs, the formation of foreign ethnic communities and the language barrier are the main problems of integration, according to a 2013 survey of 535 municipalities by Mitsubishi UFJ Research & Consulting Co.

Among them is Hamamatsu, Shizuoka Prefecture, where Suzuki Motor Corp. and Yamaha Co. have their headquarters. Foreign nationals accounted for 4.1 percent of the population in 2008 before falling to 2.6 percent in 2013 due to the global recession.

Almost half are Brazilians, hired by factories after a 1990 revision in immigration laws allowed Japanese descendants in foreign countries to work in Japan. Thousands of Japanese families migrated to Brazil in the first half of the last century to work on coffee plantations.

“It is absolutely not negative to accept foreigners,” said Hamamatsu Mayor Yasumoto Suzuki. “But if Japan is going to accept a large number of immigrants, the government needs to work out policies to help integrate them into society.”

Etsuo Ishikawa, 52, came from Brazil in 1989 and runs a legal service for immigrants in the city and for companies looking to expand in Brazil.

“Japanese people welcome foreigners as tourists, but they look at those working in Japan in a totally different way,” said Ishikawa. “Japanese is difficult to learn and customs are a bit strict, but that’s not the main issue. Japanese people need to change their attitude and see that foreign employees work hard for their family in the same way the Japanese do.”

He said it’s still difficult for Brazilians to rent a house without a Japanese guarantor and many companies don’t register foreign workers for social welfare programs, depriving them of benefits if they get ill or injured.

“It’s not a question of being for immigration or against it,” said Jesper Koll, head of Japan strategy at JPMorgan Chase & Co. in Tokyo. “Japan can no longer afford to keep debating immigration. It has to act.”

At the construction site in Kawagoe, half an hour by train northwest of Tokyo, Fan works as many as 14 hours a day and takes just two days off a month.

With the overtime, he earns up to ¥200,000 a month. Fan hopes the government’s relaxation of rules will allow him to extend his stay in the country.

“If I have a chance, I want to bring my family to Japan and work here for longer,” said Fan.