Research and Rescue: Engineers and First Responders Join Forces

Homa Alemzadeh, assistant professor in the Charles L. Brown Department of Electrical and Computer Engineering, is lead researcher on the new technology. Her collaborators are John A. Stankovic, BP America Professor of Computer Science and director of UVA Engineering’s Link Lab for cyber-physical systems, and Ronald D. Williams, associate professor of electrical and computer engineering.

They are developing a system which listens to conversations of first responders during emergencies and then offers helpful prompts to improve the responders’ situational awareness and support them as they make decisions. The artificial intelligence technology is a cognitive assistant specific to emergency medical services, or EMS, and is called CognitiveEMS.

The responders of North Garden Volunteer Fire Company signed on to help with emergency scene simulations for data collection and for evaluating CognitiveEMS. Richmond Ambulance Authority joined the effort as well and provided emergency response transcripts and other data for the testing.

Stephens, who also happens to be a 42-year employee at Northrop Grumman, leads a team of electrical and mechanical engineers in marine navigation and guidance systems. He understands the value of collaboration.

“It’s hard to simulate a real-world environment in the lab and even harder to account for factors that come into play on the ground and in public-facing situations,” he said. “I am glad we can lend our front-line expertise to the project.”

Homa Alemzadeh, assistant professor in the Charles L. Brown Department of Electrical and Computer Engineering, is the lead researcher on CognitiveEMS. (Contributed photo)

The responders have been working with UVA on the prototype since the researchers earned a $1.1 million grant from the U.S. Department of Commerce’s National Institutes of Standards and Technology in 2017. The project was one of 33 funded under the Public Safety Innovation Accelerator Program, focused on key technology areas that have the potential to transform the future of public safety communications and operations.

Alemzadeh became principal investigator on the project just a few short months after joining the UVA Engineering Link Lab, where more than 40 faculty members and more than 200 graduate students combine efforts to develop and deploy cyber-physical systems. One of the Link Lab’s research focus areas is smart and connected health care.

“Link Lab promotes research that cuts across traditional boundaries and creates partnerships with the stakeholders who will benefit from the cyber-physical systems we are developing,” Stankovic said. “The CognitiveEMS project is a powerful example of how this approach leads to solutions that broadly benefit society.”

“I enjoy thinking about the real-world implications of research,” Alemzadeh said. “A device that assists responders while they are delivering medical care at the scene could reduce stress and cognitive overload and improve outcomes for both the patients and responders.”

John A. Stankovic, BP America Professor of Computer Science and director of the Link Lab, is a researcher on CognitiveEMS. (Photo by Dan Addison, University Communications)

Alemzadeh and her team shared their progress and a CognitiveEMS demonstration at the 2020 National Institute of Standards and Technology Annual Public Safety Broadband Stakeholder Meeting, held virtually in July.

CognitiveEMS listens to a first responder talking at the scene, gathering vitals, symptoms and facts; it then converts the audio to text. Trained through machine learning, CognitiveEMS extracts key semantic concepts from the text to pinpoint corresponding EMS protocols for the situation.

By modeling the important steps in the EMS treatment protocols and real-time monitoring of their execution by responders, CognitiveEMS continuously generates what it has determined to be the most helpful prompts – additional factors responders might consider as they make decisions about interventions. The system also takes into account the level of uncertainty in the extracted information from the scene and shows evidence and confidence scores for each prompt to build trust with the responders.

The objective is to reduce the cognitive load on the responders, helping them to focus on the highest-priority tasks throughout the process.

- Ronald D. Williams
associate professor of electrical and computer engineering

In developing CognitiveEMS, the researchers first had to create a vocabulary and grammar unique to EMS concepts. They also had to build labeled datasets of simulated EMS audio transcripts of responders based on models of treatment protocols used by those responders.

The ontology and the behavioral models of EMS protocols connect the things responders do and under what conditions they do them – as identified by the words spoken at the scene – to the most relevant EMS protocols. The team has made these available to the research community through an online public repository.

Williams, in addition to being a researcher on the project, is a 25-year veteran first responder and life member of the Charlottesville-Albemarle Rescue Squad. He stresses that the CognitiveEMS prompts are not intended as guidance, but as input for responders to consider.

For example, an emergency medical technician might be responding to a call where the patient has difficulty breathing. Because fever could be a distinguishing factor between chronic disease or infection for that symptom, CognitiveEMS might prompt the responders by asking if they had considered taking a temperature.

“We were not looking to tell a responder what to do. We always want to leave the decision-making with the responders, so that they are always in control,” Williams said. “The objective is to reduce the cognitive load on the responders, helping them to focus on the highest-priority tasks throughout the process.”

Another way CognitiveEMS reduces responders’ burden is through its automatic forms feature. During the incident, the system extracts timestamped information that is used to generate real-time incident reports that can be edited prior to submittal.

Williams understands why responders have indicated this feature is one of the most helpful.

Ronald D. Williams, associate professor of electrical and computer engineering, is a researcher on CognitiveEMS. Williams is also a first responder and life member of the Charlottesville-Albemarle Rescue Squad. (Contributed photo)

“Documentation is difficult to do in the field when you are focusing on keeping your patient alive. The only practical means for filling out the forms is after the patient is in the hands of hospital staff,” Williams said. “I remember having to go down the hall to another room and fill out a form that documented everything you did, everything you saw, and the times these things happened. That’s just tough.”

Stephens appreciates an additional advantage of the system’s automatic data capture.

“Time is irreplaceable in emergencies. If you can shorten the timeline to that first medical intervention by even 10 minutes, then that’s a real game-changer,” he said. “CognitiveEMS is continuously relaying what is happening with the patient in real time. Ambulance medics enter the scene with a much better understanding of the problem, so they can start life-saving interventions in transit to the hospital.”

Thinking back to the fire that led him to volunteer in the first place, Stephens is thrilled to be an active participant in developing such an advanced, next-generation tool. “As a rural fire department, getting the opportunity to partner with UVA – by lending our services to help take tech into the future – well, that’s really neat,” he said.

Next steps on the project include fine-tuning the system for better human-cyber-physical system interaction that will transform CognitiveEMS into an at-the-scene agent. Researchers are working to integrate the CognitiveEMS with wearable hands-free interfaces, such as smart glasses, for seamless transmission of situational data to responders.

They also aim to house the processing of CognitiveEMS locally on the wearable device, so that it is stand-alone and doesn’t need internet. “In order for responders to rely on CognitiveEMS, they have to know that it will always be there,” Williams said. “We want to go the extra mile and ensure that a loss of communication does not take the system away from the responder.”

The UVA team will continue to test their system using realistic datasets and scenarios during simulated training experiments, in collaboration with the emergency responders who have been pivotal in developing CognitiveEMS from the outset.

“Working with the responders who would use the technology in the field has been critical for translating research into actionable results,” Alemzadeh said. “We are grateful to the first responders who are helping us make CognitiveEMS more accurate, functional and reliable.”