As current research has shown, Virtual Reality demonstrates great potential in the field of assistive technology. People with different extents of visual impairment, for example, can have a better vision in a Virtual reality headset (1). With this characteristic, it is assumed that special education for disabilities can be conducted through the technology of Virtual Reality and its combination with other emerging technology such as robotics and motion capture. The research focuses on the use of Virtual Reality technology in special education in China. After the visit to Shanghai special school simulation training classroom and the interview with the disability education expert Professor Hu (2)(need more information about him), a question is raised: Can simulation training for people with disabilities (mainly visual impairment and Down Syndrome) be conducted more efficiently and at lower cost in virtual reality headset. The case study introduced below mainly refers to the research of Ryo Suzuki, “RoomShift: Room-scale Dynamic Haptics for VR with Furniture-moving Swarm Robots”. A tech demo that integrates motion capture, obstacle avoidance robots, and virtual reality is the result of this case study. The experiment with students with disabilities in Shanghai is postponed due to the Covid-19 lockdown. Experiment data will be collected in the future to validate this research.
(1) https://www.boia.org/blog/what-the-future-of-virtual-reality-means-for-accessibility
(2) interview with prof.Hu below
The goal of this research was to create a working framework that would combine techniques such as real-life motion tracking with virtual reality and robotics as means to explore its possibilities and opportunities in the area of assistive technology. A study by Ryo Suzuki et al. “RoomShift: Room-scale Dynamic Haptics for VR with Furniture-moving Swarm Robots” has brought to our attention that there is a lot of potential in creating minimalistic physical settings that are enhanced with motion tracking and virtual reality, which can produce robust and interesting training and learning environments for people with disabilities. Our case study looked into different communities of people with disabilities - people with visual impairments and people with Down Syndrome - and found out that many of them could benefit from such a technological combination in areas of learning, rehabilitation, as well as training (2). The research team's focus ended up being the community of people with visual impairments for which we wanted to investigated whether it would be possible to create a usable framework which could offer more interesting and safe environments for people with varying degrees of visual impairment to practice for example walking with a walking stick, or learning how to find themselves in environments such as a metro stations or streets crossings.
(2) interview with prof.Hu below
https://drive.google.com/file/d/1sfMwWZnzLKiQYH1R9kjSc8SnNNjGVZbs/view?usp=sharing
Schematic of interaction and communication of our case study system
In a room, the research team set up 8 OptiTrack cameras $^3$ which were tracking the real-world position of the ESP32 $^6$ robots and the user $^5$ (specifically the VR headset Quest2 $^4$ , which had tracking markers attached to it). The OptiTrack cameras would capture the position data and send it to Motive $^2$. Next, the Motive program would stream the real-world position data of objects and the user in the scene in real-time to Unity $^1$. Unity was then hosting the digital environment seen by the user in their VR headsets, mirroring the real world. Any interaction of the user with ESP32 sensors (such as proximity sensors for example) would be captured and sent via an MQTT broker to Unity and any interaction of the user in the digital environment from Unity’s side would be sent back to the ESP32 via the MQTT broker (such as crossing a certain step or stage of the experience).