Next-generation online telemedicine system utilizing mixed reality for rheumatoid arthritis

Kawashiri, S. Y.; Nonaka, F.; Chiba, S.; Honda, T.; Nakajima, T.; Ishikawa, T.; Kawakami, A.

Background: Telemedicine can be performed using a conventional videophone or web conferencing system. Then, joint lesions can only be observed and inferred from two-dimensional images, and it is difficult to perform accurate joint assessments, which is essential for the management of rheumatoid arthritis (RA). Objectives: To develop the next-generation online telemedicine system utilizing mixed reality for RA. Methods: We have developed a system that can assess joints accurately in three-dimensions images in real-time, using Azure Kinect DK (depth sensor)/ HoloLens 2 (headset), which are mixed reality technologies, and Teams (online interview/chat) provided by Microsoft. Furthermore, by applying artificial intelligence (AI), we plan to implement additionally to this system 1) a function to quickly catch and automatically evaluate the patient's anxiety and changes in facial expressions at the time of examination, 2) a function to record dialogue with the patient in chronological order, 3) a function to support the detection of swollen joints, and 4) function to automatically analyze the questionnaire. Results: This system remotely connects a rheumatologist in the Nagasaki University Hospital (Nagasaki City, urban area) and a patient with RA and a non-rheumatologist in the Goto Central Hospital (Goto Island, rural area). A three-dimensional hologram of the patient's hand projected in front of a rheumatologist. Using this system, we are able to evaluate joints more accurately than using a conventional videophone or web conferencing system. Conclusion: It is expected that this system will enable remote medical care specializing in rheumatology, which is standardized at a high level even in areas without rheumatologists such as remote islands and remote areas. This system remotely connects Nagasaki City and Goto Island, but due to the performance of the system, it is not limited to these areas and it is possible to connect rheumatologists to any area that can be connected to the network. It is also effective for the purpose of avoiding the risk of infection during long-distance hospital visits under the epidemic of COVID-19 infection. (Figure Presented).