Telestration: How Helena Mentis Applies Design Thinking to Surgery

Telestration: How Helena Mentis Applies Design Thinking to Surgery

Telestration: How Helena Mentis Applies Design Thinking to Surgery

Helena Mentis is the director of the Bodies in Motion Lab at University of Maryland, Baltimore County (UMBC) with research spanning human-computer interaction (HCI), computer supported cooperative work (CSCW) and medical informatics. During a recent visit to the Design Lab at UC San Diego, Mentis talked about her research on surgery in the operating room.

She examines the medical world through surgical instruments and the workflow inside the operating room. Mentis hones in on minimally invasive surgery and the reliance toward images.  She is particularly interested in how medical professionals see and share visual information in a collaborative fashion, which has grown over the past several years. She asks, “What happens if surgeons were given greater control over the image? What would happen to the workflow? Would it change anything?”

In one study at Thomas Hospital in London, surgeons were using a lot of pointing gestures to direct the operation. Confusion would arise and the surgeon would need to repeat his exact intention with others. This break in the workflow inspired Mentis’ team to ask: what if we were to build a touchless illustration system that responded to the surgeon’s gestures? Her team set out to build what she calls “telestration,” which enables surgeons to use gestures to illustrate their intentions through an interactive display.

During another operation, the surgeon encountered a soft bone and had to stop the procedure. As a result, the surgeon had take off their gloves to re-examine the tissue on the visual display. Mentis notes, “There is a tight coupling between images on display and feeling with the instrument in hand.” If the image on display could be more closely integrated with the workflow, would this save time in the operating room?After publishing her findings, people raved over how voice narration rather than gesture aided imaging and collaboration in surgery. Consequently Mentis asked, “If given the opportunity would doctors use voice or gesture?” The ensuing observations revealed that while doctors stated their preference for voice, gesture was more frequently used for shaping telestration images. While voice narration and gestures provided greater interaction with the image, surgeons actually spent more time in surgery. Mentis reasons, “There is more opportunity for collaborative discussion with the information.” Interestingly, this did add time to the overall operation, but it also yielded greater opportunities to uncover and discuss critical information.

About Helena Mentis, Ph.D.

Assistant Professor, Department of Information Systems
University of Maryland, Baltimore County

Helena Mentis, Ph.D., is an assistant professor in the Department of Information Systems at the University of Maryland, Baltimore County. Her research contributes to the areas of human-computer interaction (HCI), computer supported cooperative work (CSCW), and health informatics. She investigates how new interactive sensors can be integrated into the operating room to support medical collaboration and care. Before UMBC, she was a research fellow at Harvard Medical School, held a joint postdoctoral fellowship at Microsoft Research Cambridge and the University of Cambridge, and was an ERCIM postdoctoral scholar at Mobile Life in Sweden. She received her Ph.D. in Information Sciences and Technology from Pennsylvania State University.

Helena Mentis is the director of the Bodies in Motion Lab at University of Maryland, Baltimore County (UMBC) with research spanning human-computer interaction (HCI), computer supported cooperative work (CSCW) and medical informatics. During a recent visit to the Design Lab at UC San Diego, Mentis talked about her research on surgery in the operating room.

She examines the medical world through surgical instruments and the workflow inside the operating room. Mentis hones in on minimally invasive surgery and the reliance toward images.  She is particularly interested in how medical professionals see and share visual information in a collaborative fashion, which has grown over the past several years. She asks, “What happens if surgeons were given greater control over the image? What would happen to the workflow? Would it change anything?”

In one study at Thomas Hospital in London, surgeons were using a lot of pointing gestures to direct the operation. Confusion would arise and the surgeon would need to repeat his exact intention with others. This break in the workflow inspired Mentis’ team to ask: what if we were to build a touchless illustration system that responded to the surgeon’s gestures? Her team set out to build what she calls “telestration,” which enables surgeons to use gestures to illustrate their intentions through an interactive display.

During another operation, the surgeon encountered a soft bone and had to stop the procedure. As a result, the surgeon had take off their gloves to re-examine the tissue on the visual display. Mentis notes, “There is a tight coupling between images on display and feeling with the instrument in hand.” If the image on display could be more closely integrated with the workflow, would this save time in the operating room?After publishing her findings, people raved over how voice narration rather than gesture aided imaging and collaboration in surgery. Consequently Mentis asked, “If given the opportunity would doctors use voice or gesture?” The ensuing observations revealed that while doctors stated their preference for voice, gesture was more frequently used for shaping telestration images. While voice narration and gestures provided greater interaction with the image, surgeons actually spent more time in surgery. Mentis reasons, “There is more opportunity for collaborative discussion with the information.” Interestingly, this did add time to the overall operation, but it also yielded greater opportunities to uncover and discuss critical information.

About Helena Mentis, Ph.D.

Assistant Professor, Department of Information Systems
University of Maryland, Baltimore County

Helena Mentis, Ph.D., is an assistant professor in the Department of Information Systems at the University of Maryland, Baltimore County. Her research contributes to the areas of human-computer interaction (HCI), computer supported cooperative work (CSCW), and health informatics. She investigates how new interactive sensors can be integrated into the operating room to support medical collaboration and care. Before UMBC, she was a research fellow at Harvard Medical School, held a joint postdoctoral fellowship at Microsoft Research Cambridge and the University of Cambridge, and was an ERCIM postdoctoral scholar at Mobile Life in Sweden. She received her Ph.D. in Information Sciences and Technology from Pennsylvania State University.

Helena Mentis is the director of the Bodies in Motion Lab at University of Maryland, Baltimore County (UMBC) with research spanning human-computer interaction (HCI), computer supported cooperative work (CSCW) and medical informatics. During a recent visit to the Design Lab at UC San Diego, Mentis talked about her research on surgery in the operating room.

She examines the medical world through surgical instruments and the workflow inside the operating room. Mentis hones in on minimally invasive surgery and the reliance toward images.  She is particularly interested in how medical professionals see and share visual information in a collaborative fashion, which has grown over the past several years. She asks, “What happens if surgeons were given greater control over the image? What would happen to the workflow? Would it change anything?”

In one study at Thomas Hospital in London, surgeons were using a lot of pointing gestures to direct the operation. Confusion would arise and the surgeon would need to repeat his exact intention with others. This break in the workflow inspired Mentis’ team to ask: what if we were to build a touchless illustration system that responded to the surgeon’s gestures? Her team set out to build what she calls “telestration,” which enables surgeons to use gestures to illustrate their intentions through an interactive display.

During another operation, the surgeon encountered a soft bone and had to stop the procedure. As a result, the surgeon had take off their gloves to re-examine the tissue on the visual display. Mentis notes, “There is a tight coupling between images on display and feeling with the instrument in hand.” If the image on display could be more closely integrated with the workflow, would this save time in the operating room?After publishing her findings, people raved over how voice narration rather than gesture aided imaging and collaboration in surgery. Consequently Mentis asked, “If given the opportunity would doctors use voice or gesture?” The ensuing observations revealed that while doctors stated their preference for voice, gesture was more frequently used for shaping telestration images. While voice narration and gestures provided greater interaction with the image, surgeons actually spent more time in surgery. Mentis reasons, “There is more opportunity for collaborative discussion with the information.” Interestingly, this did add time to the overall operation, but it also yielded greater opportunities to uncover and discuss critical information.

About Helena Mentis, Ph.D.

Assistant Professor, Department of Information Systems
University of Maryland, Baltimore County

Helena Mentis, Ph.D., is an assistant professor in the Department of Information Systems at the University of Maryland, Baltimore County. Her research contributes to the areas of human-computer interaction (HCI), computer supported cooperative work (CSCW), and health informatics. She investigates how new interactive sensors can be integrated into the operating room to support medical collaboration and care. Before UMBC, she was a research fellow at Harvard Medical School, held a joint postdoctoral fellowship at Microsoft Research Cambridge and the University of Cambridge, and was an ERCIM postdoctoral scholar at Mobile Life in Sweden. She received her Ph.D. in Information Sciences and Technology from Pennsylvania State University.

Read Next

Design Lab Self-driving Nissan Toyota Ford Duke Stanford E-hmi

Design Lab Collaborates with Amgen to Explore Adoption of Medical Therapies

The Design Lab has recently embarked on an exciting collaboration with Amgen to explore the…

Seda Evis

Designer in Residence: Seda Evis

While Seda Evis participates as a Designer-in-Residence at UC San Diego with The Design Lab, and is VP of Strategy & Growth at FreshForm Interactive—an experience design and innovation consultancy—she also claims to have what she refers to as a superpower: her hybrid mind, which she describes as the combination of two worlds: the business side and the design side.

As a Designer-in-Residence at The Design Lab,  Evis enjoys working in an interdisciplinary setting, which she says enhances her existing skillset. “Academia tends to be quite separate from how the practice is done,” she explains. “I find ways of doing interdisciplinary work, as well as cross industry work, very important for innovation because that’s how you actually get seeds from different places.” As of now, most of Evis’s work at The Design Lab has been working with the Community Team on the now winning bid for the World Design Capital 2024 (WDC) designation. Her work dovetails with her role on the Board of  Directors for the Design Forward Alliance–a non-profit organization started by the Lab in partnership with the regional design community and one of the partners co-leading the HOME 2024 WDC efforts alongside the Design Lab, Burnham Center for Community Advancement, the City of San Diego and the City of Tijuana.   The designation puts the San Diego-Tijuana region in a prestigious international cohort recognized for the “effective use of design to drive economic, social, cultural and environmental development.”  Even the proposal theme is significant, Evis says. “Home” not only refers to the immense and diverse communities of San Diego and Tijuana that form one, but also serves as an acronym for Human-centered, Open, Multi-disciplinary/Multi-cultural, and Experimental. For Evis, participating in HOME2024 signifies her career “coming full circle.” 
Olga McConnell

Olga McConnell, Project Specialist and Executive Assistant to the Director of The Design Lab

As the Executive Assistant to the Director of The Design Lab, a project manager for the Lab’s special projects and annual events, and a lifelong learner who holds a M.A. in English Linguistics and Translation, and a M.B.A. in Business Administration and Management, Olga McConnell’s zest for knowledge is palpable. She is currently on track to complete a Project Management Certification at UC San Diego Extension at the end of 2021, and she is planning on obtaining her Project Management Professional (PMP) Certification after that. “I’m kind of addicted to getting degrees,” jokes McConnell. “I even thought the other day, maybe I’ll go to law school. And then I was like, no, enough, enough.” 

For nearly five years, McConnell was Executive Assistant to Don Norman, the Founding Director Emeritus of The Design Lab. She is now the Executive Assistant to the new Director of The Design Lab, Mai Thi Nguyen. It is Nguyen’s vision of human-technology-community interactions, along with her JEDI (justice, diversity, equity and inclusion) approach that has McConnell excited about this new chapter in the Lab’s legacy, saying, “I see how great she is as an efficient leader, so I’m really looking forward to working with her, supporting her administratively, as well as taking charge of certain projects that she has in mind.”
Design Lab Cat Hicks Signalio

Cat Hicks Q&A: A Conversation about Google & her new start-up Signal IO

The Design Lab has long lasting impacts. Catherine Hicks has seen the Design Lab since its…

Mobile Health

Focus on mobile health: Scientists develop app to diagnose, treat leishmaniasis

Photo courtesy of Centro Internacional de Entrenamiento e Investigaciones Medicas

Cutaneous leishmaniasis - caused by bites from infected sandflies - produces skin lesions that leave behind both scars and stigma that last a lifetime. Up to 1.2 million new cases are diagnosed each year across the 90 countries where the disease exists, including Colombia.

“Leishmaniasis happens where the medical system isn't," says Dr. Eliah Aronoff-Spencer, a Fogarty mHealth grantee at the University of California, San Diego (UCSD). He's been working in rural Colombia to bridge the access gap between remote communities and the public health system, using a mobile tool that empowers community health workers to identify new cases of the disease and monitor treatment.
Design Lab Remanufacturing Circular Economy Law

Spotlight on Peggy Zwolinski

The Design Lab welcomed Peggy Zwolinski, a visiting scholar from France who is currently working…

Back To Top