On February 9–10, 2021, Microsoft organized an unusual event: fifteen surgeons from several countries across the world almost simultaneously performed 12 orthopedic surgeries with the use of the HoloLens 2 AR headset and the voice assistant Dynamics 365 Remote Assist.
During the process, surgeons from different countries could step in to help their colleagues virtually. French professor, Thomas Gregory, who performed the first-ever HoloLens-facilitated surgery back in 2017, led the group of surgeons.
Surgeries utilizing such headsets look like this:
One of the 15 surgeons was a Ukrainian, Oleksandr Strafun, senior research associate at the Department for Microsurgery and Reconstructive-Recovery Surgery of Upper Limb at the SI “The Institute of Traumatology and Orthopedics” with the NAMS of Ukraine.
His department specializes in the treatment of severe diseases and injuries of the upper limb, performs surgeries on the shoulder and elbow joints, including endoprosthetics (i. e. replacement of a joint with a metal prosthesis for various diseases). Oleksandr used HoloLens for just such kind of surgery. Moreover, Oleksandr’s department has performed more than 100 operations with the use of a VR headset.
AIN.UA has contacted the surgeon to find out how modern medicine utilizes VR technologies.
How did it happen that your department joined the Microsoft project?
This is a joint project between Microsoft and Evolutis, a prosthetics manufacturer. As far as I know, the idea comes from Microsoft’s French office. And it is consistent with current trends in medical technology.
The use of various navigation systems to assist surgeons is now very actively spreading across the world. Perhaps, you have heard of surgical robots, the da Vinci system, etc.
To perform a high-quality operation, one must either be a hugely experienced surgeon, or have a great deal of specialized tools at hand. This is somewhat similar to automobile automation. A modern car has numerous devices which partially automate the process and help you drive; these are automatic retainers, hydraulic steering boosters, stabilizers, and so on. Loosely speaking, the car excuses some of your insignificant mistakes and fixes them automatically. Similar things take place in surgical medicine.
These include the current development of navigation systems in the world that help perform shoulder replacement arthroplasty (which means surgically replacing the affected joint with an artificial one in the patient’s shoulder – Ed.).
Imagine a shoulder blade. It is a very flat bone. And now, imagine that for the subsequent implantation of an endoprosthesis, you need to hit it with a pin that is 1.5mm thick. The pin is the base for the further endoprosthesis implantation. Due to this complexity, the pin is often placed slightly aslant, and consequently, the endoprosthesis is placed off-center, which reduces its operational life.
In order to minimize those errors, many companies develop navigation systems that help set the pin into its ideal position, so that the prosthesis can be fixed properly and serve for as long as possible.
Microsoft is a little behind other manufacturers, but they have taken on the idea in a quality way. Now, the French endoprosthetics company has partnered with Microsoft, and they have decided to try and utilize the HoloLens headset to facilitate navigation during surgical operations.
Since I had taken courses in France and had experience in the installation of such prostheses, the representatives of the French company contacted our department and offered me to take part in this project.
Please tell us about these surgeries, and particularly yours.
All of this was supposed to happen on May 25, 2020, and this must have been a 24-hour-long shoulder surgery streamed online live. It was to begin in France, and then, every one or two hours, the surgery was supposed to start in another country. During the operation, round tables and interviews were to be held.
The idea had looked brilliant, until the coronavirus came. Because of it, the date was postponed several times, with the event at last held in February. Instead of being streamed live, the surgeries were simply recorded, and now those records are little by little made available to the public.
We conducted our surgery in December, 2020, and the patient can already raise her arm with an artificial joint implanted. I was performing the surgery via HoloLens, helped by two colleagues from European clinics, who gave their advice and asked questions.
Will you tell us more on why you need a VR headset during a surgery?
In medicine there are such kinds of examinations as magnetic resonance imaging and computerized tomography. MRI uses magnetic fields and provides good images of tissues containing a lot of water (muscles, tendons, joint capsules). And CT is a kind of X-ray imaging technique that provides perfect pictures of bone tissue. Having a CT image of any bone, we can create its 3D model on a computer. A VR headset, such as HoloLens, allows us to “implant” that model into the patient.
What does it mean for prosthetic surgery? Inside this 3D model of a real patient’s bone, we can have a conditional pin set at the desired angle from the start. After that, we can simply align the 3D image with the real pin, to know for sure that we are hitting exactly where we aim, with a 99% probability.
We received a VR headset back in May 2020, and by December we had already learned how to use it, how to upload CT scans and 3D models. We have performed about 100 surgeries using HoloLens.
By the way, it looks quite funny, when a surgeon stands in the operating theatre wearing a VR headset and waving his or her hands in the air.
When Microsoft improves the software a little, this will be an extremely helpful tool in prosthetics. At this stage, the main problem is that head movement can cause the 3D model to fly off, instead of keeping its position. But this is being worked on.
I should note that this remark refers to prosthetics, but we currently use the headset not only for prosthetics, but also for the treatment of aseptic bone necrosis. It is a useful tool for such operations, and we are very happy with it.
Sadly, chemotherapy sometimes causes vascular bed destruction in young people, with humeral and femoral heads dying. Patients cannot use their joints because of the pain. With the help of HoloLens, we insert special thick needles into the affected bones with high precision and populate the dead bones with living bone marrow cells. HoloLens shows us the exact place in the patient’s limb where the affected part of the bone is located, and thus, it becomes much easier for us to hit the mark.
Since we began to use HoloLens, the number of intraoperative X-ray scans has dropped dramatically. Now we manage to hit the right place at the first try; this reduces the X-ray exposure for the patient and greatly speeds up the surgery itself.
