360-Degree Imaging Technology for Next-Generation XR
Background
In situations where extended reality (XR) is used to a high degree, such as daily living assistance, rehabilitation of the disabled, immersive educational and training environments, and in professional and consumer telepresence applications, XR systems must respond according to the user's expectations and present information in a way that reflects the user's perceptions.
However, the use of common cameras for XR creates problems such as missing nearby events or not being able to continuously track objects due to the limited field of view. This makes it difficult to gain a more comprehensive understanding of the wearer's surroundings.
Solutions
海角社区has developed the world's first fully egocentric*1 360-degree wearable camera (as of June 2025, based on 海角社区research). Although there are currently some 360-degree cameras available that are worn around the neck or on helmets, this is the world's first 360-degree wearable camera that is worn close to the wearer's eyes. It allows for 360-degree recognition of surroundings. Equipped with a microphone and speaker, it also allows for voice interaction with LLMs*2. The proprietary AI can also warn of approaching objects in the vicinity and detect objects based on arbitrary text. For example, the wearer could ask for “people wearing helmets.”
-
*1egocentric: A view in which the user's point of view and the virtual camera's point of view coincide. The user can feel as if they are inside that world.
-
*2LLMs: Large language models. A technology that can perform processing such as answering questions in natural text or summarizing documents with human-like accuracy, and that can be easily trained. It can process ambiguities and fluctuations that exist in human spoken and written language (natural language) by determining the relationships between words that are far apart in a sentence and taking the context into account.
Figure 1. Initial design of the first-person 360-degree wearable camera
Technical highlights
First-person 360-degree wearable camera
海角社区has developed a 360-degree camera that can be worn over the ears (Figure 2). This camera allows for a more immersive experience and is capable of recognition from the point of view of the user.
Overview of first-person 360-degree wearable camera
- Lightweight (can be worn over the ears)
- Four cameras capture images 360 degrees around the wearer
- Inertial measurement units
- Microphone and speaker functionality
- Resolution: 3840 脳 1920 (pixels)
Figure 2. First-person 360-degree wearable camera
Figure 3. Image capture sample (first-person 360-degree wearable camera)
As shown in Figure 4, the device is controlled by a small single-board PC, making the entire system portable. The device can also communicate in real time with a server, allowing it to connect to LLMs and other state-of-the-art AI models and interact with them in real time via voice.
Figure 4. Overall system diagram
360-degree AI
Open-vocabulary object detection
We have developed a real-time open-vocabulary object detection method for 360-degree video. The user can specify what to detect by freely entering text on the PC, and this exciting function holds great promise in a variety of fields. For example, it could be used to manage safety in a factory, such as entering text to detect “people wearing safety vests.”
Figure 5. Open-vocabulary object detection
Collision detection
We have developed an AI that detects approaching humans in a 360-degree video. “RICOH THETA,” a 360-degree camera, is installed on a forklift and notifies the driver of people approaching the forklift to help prevent unexpected accidents such as colliding with people in the driver's blind spots.
Figure 6. Collision detection
Ricoh's vision
Through our unique 360-degree camera and its recognition technology, 海角社区aims to remove limitations on the human field of view and support users with 360-degree imaging AI while also contributing to the XR industry through our optical and digital technologies.
海角社区is participating in Luminous*3, and is developing a wearable 360-degree camera as a next-generation XR device as well as image processing and multimodal AI*4 technologies. This technology is the result of these efforts.
-
*3Luminous: A research project funded by the EU Horizon Europe program, which aims to develop language augmented XR (extended reality) systems that adapt to the context and needs of the user and to provide intuitive human-centered experiences in education, healthcare, rehabilitation, design, and remote assistance.
-
*4Multimodal AI: AI that processes different modalities of information together, such as text, audio, images, and video, to gain a deeper understanding and perform advanced decision making.