As Japan and Australia prepared to face off in Sunday’s rugby wheelchair bronze-medal final, some viewers in Japan may have seen Yuki Goto reciting both teams’ starting lineups in sign language, hands flitting from one pose to the next in a series of precise movements.
But it wasn’t exactly Goto on their screens.
The 25-year-old reporter, who has a hearing impairment, was chosen to model for a computer-generated sign-language commentator unveiled by NHK during the Tokyo Games.
The move is the national broadcaster’s most ambitious initiative yet to bring a variety of content to Japan’s deaf population, which totaled 358,800 in 2006 — the year of the government’s most recent survey.
“I’d seen CG sign language before, and it didn’t look natural,” Goto told The Japan Times ahead of the Paralympics. “It wasn’t human, it was a little choppy.
“Then (NHK) scanned me and put together a sample, and I thought it was amazing. Speaking as someone with a hearing impairment, it's a huge step forward.”
NHK’s effort to create sign-language content began in 2009 in response to growing demand from native users who struggled with Japanese-language subtitles. While Goto can converse normally when using her cochlear implant, she admits that it isn’t always ideal and regularly takes it off when at home.
“Implants are machines; there’s a lot of ways in which they’re inferior to human ears,” Goto said. “When my implant is on it’s tiring because I can hear everything. You can’t turn on noise cancelling; everything enters at the same volume.”
In order to build a lexicon of signs, NHK staff first used motion-capture technology to record around 8,000 words for its database — ranging from “autumn colors” to “preschool” — with sign-language users providing feedback through an online portal.
According to Hiroyuki Kaneko, who manages the sign-language project at NHK’s Science & Technology Research Laboratories, early efforts focusing on hand movements were quickly found to be insufficient.
“We realized that you can’t read sign language just from the hands, so then we started capturing facial expressions including eyebrows and mouth movements,” Kaneko said.
After eight years of development, NHK unveiled CG sign-language weather forecasts for the Kanto area in early 2017, offering updates three times a day.
Soon after, bolstered by trials of text-to-speech audio commentary — based on data from the Olympic Data Feed (ODF) provided to broadcasters during the 2016 Rio Games — the network decided to pursue the creation of sign-language sports content with Tokyo 2020 in mind.
The text-to-speech commentary, known as “robot play-by-play,” debuted for four sports during the 2018 Winter Olympics in Pyeongchang, South Korea, as researchers continued to prepare sign-language commentary for Tokyo.
“From the perspective of universal service, we want to offer the same experience to everyone who’s watching, whether or not they have a disability,” said Toru Ogawa, who has overseen NHK’s universal service offerings for Tokyo 2020.
“Robot play-by-play started with four disciplines in Pyeongchang; in Tokyo we’re up to 10 for the Olympics and seven for the Paralympics, so we’re steadily expanding. We’re doing things we’ve never done before.”
After developing a system that converts ODF play-by-play data into templates that string together sequences of pre-recorded signs, Ogawa and Kaneko’s attention soon turned to the issue of presentation.
While initial plans called for an anime-style two-dimensional presenter, the one-year postponement of Tokyo 2020 allowed NHK to respond to viewer feedback asking for more identifiable, photorealistic commentators.
That’s when Goto got the call, with designers working until just before the Games to avoid the “uncanny valley” effect that often occurs when a CG human doesn’t quite hit the mark.
“When you do CG reproductions of a person, sometimes it resembles them and sometimes it doesn’t,” Goto said. “The production team said it looked like my sister, and when I saw it ... it was so unusual. It was really natural and fast, about the pace that sign-language speakers would use to talk.”
Kaneko and his team worked to develop signs that would be understood by a majority of listeners, although he admits that it remains an imperfect science.
“It took a lot of time to think about how to best express those phrases and ideas,” he said. “Before motion capture we did a lot of translation and editing. And then we’d do the capture and see how it looked, and we kept repeating that process.
“Even so, sign language is difficult ... there’s not always a right answer. But we gradually made things easier to understand for viewers.”
In addition to wheelchair rugby, the CG sign-language commentary has been used for Olympic basketball and Paralympic wheelchair basketball. All three sports feature rapid developments in play as well as frequent stoppages for fouls that may confuse hearing-impaired viewers — making them a perfect fit for the technology.
“When the game stops, deaf people who are watching may know the score but they might not understand why play has stopped,” Kaneko said. “So we created a male character to explain the rules of the game and the fouls.”
Future iterations could see both characters become more emotive: Viewers have suggested that the commentators’ moods should change depending on the state of the game, especially if Japan is doing well.
“They want the commentator to cheer for Japan,” Kaneko said. “So we’re looking for ways to get the commentator to be more enthusiastic when Japan scores, and to create sign-language commentary that excites the viewer.”
Such efforts go beyond the commentator “booth.” The website featuring NHK’s sign-language commentary also includes a volume meter, allowing deaf viewers to understand the level of excitement inside the venue.
“You’d think that because there aren’t fans at the Games there wouldn’t be an atmosphere, but if you watch the match there’s a lot of excitement at the venues,” Ogawa said. “People with regular hearing can tell from watching, but now those with hearing impairments can share that experience.
“As NHK, we have to reduce barriers and provide the same information to everyone. This is one way to do that, and there are other things we’re working on — I think that in the future we could enter a new era of communication.”
With Tokyo 2020 serving as a successful proof of concept of sign-language commentary, those involved in its creation are hopeful that it can be further improved and expanded upon.
Overcoming technical limitations such as a reliance on templated content and the need to capture each sign individually would open the door for live sign-language broadcasts not only of sports, but of breaking news and other vital information.
“Being reliant on motion capture is a big challenge for us,” Kaneko said. “But we’re looking at taking signs we’ve captured, and doing things like changing hands from right to left and changing signs to create new signs.”
That’s welcome news for Goto, who is reporting on athletic competitions at the National Stadium during the Games.
“You hear the words ‘adaptability,’ ‘diversity’ and ‘universal design’ a lot, but for those of us with hearing impairments the issue isn’t tangible barriers like elevator access, but intangible barriers like the struggle to get information,” Goto said.
“Information needs to be a part of universal design and diversity. There hasn’t been a lot of focus on changing that, but I feel like that’s starting to happen.
“People understand that ‘barrier-free’ isn’t just about wheelchair access. It’s changing slowly, but it’s changing.”