Sony Patents Revolutionary In-Game Sign Language Tool

Sony has filed a patent that aims to enhance accessibility for deaf gamers. Sony's patent illustrates how specific sign languages can be converted to another in-game.

Sony Patents ASL to JSL Translator for Video GamesProposed to Utilize VR Devices and Operate Over Cloud Gaming

Sony has filed a patent that adds a real-time sign language interpreter to video games. The patent, titled "TRANSLATION OF SIGN LANGUAGE IN A VIRTUAL ENVIRONMENT," illustrates a technology wherein American Sign Language (ASL) could be communicated to a Japanese-speaking user using Japanese Sign Language (JSL).

Sony said it aims to implement a system that can aid deaf gamers via real-time interpretations of sign languages during in-game chats. The technology described in the patent would enable virtual indicators or avatars displayed on-screen to communicate sign language in real-time. The system will initially translate the sign gestures of one language into text, then convert the text to another designated language, and finally translate the received data into the sign gestures of the other language.

"Implementations of the present disclosure relate to methods and systems for capturing sign language of one user (e.g., Japanese), and translating the sign language to another user (e.g., English)," Sony described in the patent. "Because sign languages differ depending on geographical origins, sign language is not universal. This necessitates appropriately capturing the sign language of one user, understanding the native language, and generating new sign language as output for another user in their native sign language."

One way this system could be implemented, as Sony illustrated, would be with the aid of a VR-type device or headset (HMD). "In some implementations, the HMD connects via a wired or wireless connection to a user device, such as a personal computer, game console, or other computing device," Sony detailed. "In some implementations, the user device renders graphics for display through the HMD that provide immersive viewing of the virtual environment for the user."

Sony further proposed that one user device could seamlessly communicate with another user device over a network with a game server. "In some implementations, the game server executes a shared session of a video game, maintaining the canonical state of the video game and its virtual environment," Sony said, "and to which the user devices are synchronized regarding the state of the virtual environment."

With this setup, users can share and interact with each other in the same virtual environment, aka game, over a shared network or server. Sony additionally said that in some implementations of the system, the game server could be part of a cloud gaming system, which "renders and streams video" between each user device.