S4E21: Improving Patient Engagement with AI Chatbots (ft. Kathy Mazza, Northwell Health)

Jordan Cooper   1:17
Mm.
Mhm.

Kathy Mazza   1:36
So people may be familiar that the Medicare star ratings, you get quality dings if your patients are readmitted within 30 days. You know, it's expensive. You know, you're not getting additional reimbursement. You're kind of giving a warranty for that 30 days to the payer. And so we have a really robust program for managing that.
And, but it's expensive. It's a large team of nurse practitioners, physician assistants, administrative support, and figuring out who really needs attention, you know, and who needs level one attention and who maybe could be reached less frequently and is probably going to be okay in the community. We started using chat bots to communicate with patients.

Jordan Cooper   2:16
Okay.

Kathy Mazza   2:22
to determine were their symptoms the same, improving, worse? Did they have a follow-up appointment? Did they fill the prescriptions, things like that? And depending on their replies to that, it was determined, you know, this one needed a call within 15 minutes, could be the, you know, for a

Jordan Cooper   2:39
Oh, wow.

Kathy Mazza   2:40
a red or, you know, a high alert, a heart failure patient who was having worsening symptoms, didn't have a follow up appointment, all those kind of red flags may perhaps gain some weight. And we could reach them quickly where patients who had all those things kind of covered and were improving, we might wait till tomorrow to call so that you used your

Jordan Cooper   2:42
Mhm.
Yeah.

Kathy Mazza   3:01
expensive human resources more efficiently. The product we were working with there, we're not currently involved with. We've recently begun our Epic transition. We started in November, and we're, you know, with a very large implementation, as you might imagine, across

Jordan Cooper   3:03
Hmm.
Mm-hmm.
Right.

Kathy Mazza   3:21
over 100,000 employees and almost 30 hospitals. So we're learning now what Epic capabilities we purchased and understanding how we might use some of that and also looking at some external ones because this is a space that's exploded since 2018. What we used was really kind of

Jordan Cooper   3:37
The.

Kathy Mazza   3:42
It was called conversational AI, but I would say it was more algorithmic, more clinical decision tree of if the patient says yes to this question, ask them this question, where if these newer models that we're evaluating really can be, you know, do you have a ride to the appointment? And instead of yes, no, the patient can

Jordan Cooper   3:53
Mmh.

Kathy Mazza   4:03
free text or speak their reply that my sister's taking me. That's great that your sister helps you like that. You know, it may snow Tuesday. You know, understand, you know, understanding, you know, is that going to be a problem? You know, things like that. So the generation that's coming right now is really exciting what's possible.

Jordan Cooper   4:05
Right.
Yeah.
So I think that Northwell's journey with these chatbots is maybe of great interest to listeners because many health systems across the United States have begun their journey much more recently. And instead of 2018, some have started incorporating Gen. AI tools in the 2020s.

Kathy Mazza   4:36
Uh-huh.
Mhm.

Jordan Cooper   4:43
I'm wondering, so as you've seen kind of more of an algorithmic AI tool evolve and then the array of tools evolve, what kind of reactions have you seen from patients, from providers? And then how have you been adapting this sort of kind of resource allocation tool
As you've been getting more advanced AI resources.

Kathy Mazza   5:08
you know, you're spot on there. We were early and nobody really, I did a research project on our pilot for dissertation. It was my project. And it was, I couldn't find articles about the use of chatbots in healthcare. It just, you know, it was a very innovative thing in 2018.

Jordan Cooper   5:10
Yeah.
Mhm.
Yeah.

Kathy Mazza   5:29
We were really fortunate that we did that because when the COVID surge hit in New York in 2020, we had this tool up and ready and a crisis that frankly delayed the implementation of some of these tools for many health systems because budgets were constrained.

Jordan Cooper   5:34
Mm-hmm.

Kathy Mazza   5:48
resources were constrained. It was not the time to be introducing something new. We would probably never have kicked that pilot off in 2020. But because we already had the tool, we used it. Honestly, we used it to send patients home from the emergency room who might have been admitted under other conditions, but they were sent home with a chat bot.

Jordan Cooper   5:57
Right.

Kathy Mazza   6:08
a pulse oximeter, a thermometer, and we monitored them and we could dispatch an ambulance if we thought we needed somebody to come back, or we could know that they were safe in the community. And honestly, those chat bot questions included things like, do you have food in the house? You know, do you need a delivery from the pharmacy? How can we help you stay

Jordan Cooper   6:08
Mhm.
Mhm.
No.
Mhm.

Kathy Mazza   6:29
home in that. So getting a head start really positioned us well in that crisis. And what I think what we've, some of the lessons we've learned from that pilot, and it recruited thousands of patients, when I say a pilot, it was a very large pilot.

Jordan Cooper   6:30
Yeah.

Kathy Mazza   6:50
We now, we're much more discerning consumers now. We have, we know what we want from this product now. The Medicare population is, by definition, generally 65 and older, except for some, you know, other unique circumstances. I don't,

Jordan Cooper   6:54
Yeah.
Mhm.

Kathy Mazza   7:10
I don't find that patients 65, 75 are at all reluctant to use technology specifically to support their health. The literature shows they will. But when you start getting to a patient who's 90, 95, there can be challenges in some of this kind of technology.

Jordan Cooper   7:17
Mhm.
Mhm.

Kathy Mazza   7:30
adoption. And so now some of the models that we're looking at are more voice activated rather than text activated. We'd like to explore that where it comes in sounding quite natural where the phone call comes in and it may say, you know, this is a Northwell chat agent calling.

Jordan Cooper   7:37
Mhm.
Yes.

Kathy Mazza   7:50
you know, identifying it, but asking the questions and taking advantage of some of the natural language processing that we really didn't find effective in 2018. You specifically asked about our patient experience. And, you know, I have one anecdote I've shared

Jordan Cooper   8:05
Yeah.

Kathy Mazza   8:10
before, but we had one rather high-powered executive. You know, it's New York City is our metro, where that's where we are. And he was used to a certain level of performance from everything around him. And he had been discharged home with a heart failure diagnosis. And

Jordan Cooper   8:15
Mhm.
Mhm.

Kathy Mazza   8:30
in the chat bot, he put in his weight and he had gained a few pounds from discharge to the time he was doing this. And a couple of minutes later, a nurse called and he was confused by the phone call. He was not understanding why someone was reaching out to him. And when she said, well, I see that, you know, your weight is 203 and you were 199 on Friday,

Jordan Cooper   8:35
Mhm.
Hmm.

Kathy Mazza   8:52
and I'm concerned that we may need to adjust your medications or is there something going on with your diet? How's your shortness of breath? And he kind of got flustered and he said,

Jordan Cooper   8:53
Yeah.
Mhm.

Kathy Mazza   9:02
wait a minute, you're calling me because of the weight I put in my phone 5 minutes ago? And she said, yeah, he goes, I was in the hospital last week and I rang a call bell and I couldn't get a nurse that fast. And so, you know, patients can be delighted by this technology if you make it meaningful.

Jordan Cooper   9:07
Yeah.
Yeah.

Kathy Mazza   9:23
If you send somebody a 40 question questionnaire every morning and nobody does anything with it, they drop out. After one week, they're done. But when they feel that it really impacts their care or somebody takes action, or even if the chat bot says,

Jordan Cooper   9:27
Mhm.
Mhm.
Mhm.

Kathy Mazza   9:41
You're doing great. Keep it up. Would you like to see some education about cardiac rehab or about, you know, and take them down paths that maybe they've expressed an interest in learning how to exercise with heart disease or better nutrition with heart failure? And you can use these opportunities.

Jordan Cooper   9:48
Right.

Kathy Mazza   10:02
I don't know if you used the V-safe chat bot that the CDC had in COVID. It was a, if you got vaccinated, you could answer a few questions about how you were tolerating the vaccine, what symptoms did you have to it, any reactions to it. And I did it every day when they sent it to me because it was so easy. It was

Jordan Cooper   10:15
Mhm.

Kathy Mazza   10:24
a minute, less than a minute. I never like said, oh, this thing's going to take a while. I'll do that later and put my phone down. I knew I could answer two or three questions and they would have the data they needed. And I think that's where we have to find the sweet spot. It's very tempting to ask patients lots of things. But if you're never going, if no one's going to take action on it, don't waste their time.

Jordan Cooper   10:26
Mm-hmm.
Yeah.

Kathy Mazza   10:44
It would be a tidbit I would offer to others.

Jordan Cooper   10:44
So...
It sounds like there are a lot of different data sources feeding into this tool. Obviously, there's clinical record, there's patient-generated responses. What other sort of data sources did you need to integrate in order to enable this tool to not only be a data ingestion mechanism from the patient, but also to be

Kathy Mazza   10:53
Mhm.
Mhm.
Mhm.

Jordan Cooper   11:09
a data provision mechanism to send resources and information and direct calls to the patient.

Kathy Mazza   11:15
So the vendor that we worked with had a dashboard, but we integrated fully into our ambulatory EMR. So that the, and to the, we had a homegrown platform for care navigation for that group of nurse practitioners, PAs, nurses that were doing these care navigation

Jordan Cooper   11:19
It is.
Mhm.

Kathy Mazza   11:37
models post-discharge specifically, but also for our own employee plan for chronic diseases like diabetes or COPD. We provide those kind of programs for employers, for lots of different groups. And so they all documented in this one platform. And we made it very visually

Jordan Cooper   11:41
Mhm.

Kathy Mazza   11:58
there were color, it was color-coded and they could look very easily to see who had a red reply today, who was all green, who had a, it was green, red and yellow were our color levels of alert. And you know, who was green moving into yellow looked different than who was red moving into yellow because

Jordan Cooper   12:11
Mhm.
Mhm.

Kathy Mazza   12:20
That's a very different message. If somebody's doing great and now they're moving into a medium zone, it's very different than somebody who's really been in trouble, but now they're in medium. They're kind of moving in the right direction. And so putting that in the user workflow that they got their alerts in their usual electronic medical record, they didn't have to go

Jordan Cooper   12:21
Mhm.
Mhm.
Mhm.

Kathy Mazza   12:40
remember a password to log into another place, because that is a real barrier. People change departments, change employment, somebody else has to get access. You really want to make sure that if there's potentially a dangerous situation out there, that someone is monitoring and can see this very quickly, not even

Jordan Cooper   12:45
Mhm.
Mhm.

Kathy Mazza   13:04
a few times a day, depending on the use case. Now, if it's about, you know, a rather low risk use case, that may be fine. But we used some of this for OB-GYN and postpartum depression screening was part of it. One other anecdote, we had a patient who

Jordan Cooper   13:05
Mhm.
Mhm.

Kathy Mazza   13:24
had her first postpartum visit with the doctor, brought the baby into the visit. Everybody was ooing, ahhing off the baby, you know, checked out fine, went home and answered the questions in her postpartum screening and expressed suicidality. She was really struggling with postpartum depression. And

Jordan Cooper   13:31
Mhm.
Yeah.

Kathy Mazza   13:43
In conversation later with the doctor, the doctor said, you know, I...
I thought you were okay. I'm so sorry I had missed it. And she said, well, you know, I was embarrassed to say anything because everybody, I know the baby's a blessing and lots to be thankful for and a healthy baby is a gift. But she said, when I got home and then the chat bot was there, it was a little more anonymous. It felt easier to have the conversation there.

Jordan Cooper   13:52
Mm.
Mhm.
Mm-hmm.
Mhm.

Kathy Mazza   14:11
And that was a real eye-opener to us that for certain use cases, patients actually prefer this. And listen, we all know, like if my dentist office calls me to remind me I have an appointment, I'm like, oh, I'm in a meeting. Can you just text me? Like, can't we just do this more easily? And so I think

Jordan Cooper   14:26
Yeah.
Yeah.

Kathy Mazza   14:30
where we're coming to now is the chat bots are becoming more sophisticated to where they can know how do I prefer to be addressed? When do I prefer? Well, you know, some patients will answer at 7 in the morning. Some will answer at 7 in the evening. Some want natural language processing. Some want just

Jordan Cooper   14:46
Right.

Kathy Mazza   14:50
a couple of questions with multiple choice. How do we target to get rich data from all of them? You know, and now the addition of devices, wearables, sensors, medical devices, it's, you know, it's unlimited. It's a little overwhelming.

Jordan Cooper   15:02
Yeah.
Let's talk about those devices. I know that you've integrated apps and medical devices and wireless devices. How, and I guess all of that is into this AI chatbot type workflow. Can you explain, can you elaborate on what kind of apps, device, and wireless devices have been integrated and

Kathy Mazza   15:19
Mhm.

Jordan Cooper   15:28
how have that, how has that affected workflows for providers and just the general experience for Northwell.

Kathy Mazza   15:35
Sure.
Interestingly, with our initial vendor, we did a pilot just to do hypertension monitoring. We didn't really want to start with something like glucose monitoring or any kind of obstetrical application, like some high risk things where somebody could really be in a dangerous situation. We said, let's do this with blood pressure where we can kind of have a

Jordan Cooper   15:45
Mhm, mhm.
I.

Kathy Mazza   16:02
you know, an explanation at the beginning, like when you call your doctor's office and they say, if this is an emergency, hang up and dial 911. We felt with blood pressure, we could provide some, you know,

Jordan Cooper   16:09
Right.

Kathy Mazza   16:14
grow barriers to people and say, beyond this, call us. Don't wait for us to call you. Make sure you get in touch with us. And so, but that pilot, we learned a lot about how data did not flow well on the internet through the blood pressure cuff to the

Jordan Cooper   16:20
Yeah.

Kathy Mazza   16:34
manufacturer of the blood pressure cuff on their app and then over through Validic and into the vendor. And so we actually put it on pause because we felt we were going to create a patient safety issue where patients were safer with their notebook and a pen.

Jordan Cooper   16:48
Mm.

Kathy Mazza   16:53
in 2021 that was. And so, you know, we're now that we're then shortly after that, Epic was announced. And so now we're beginning to understand what are what are their capabilities for incorporating this data. But we have physicians

Jordan Cooper   16:55
Mm.

Kathy Mazza   17:12
whole specialties lined up waiting for us to offer glucometer monitoring. Hypertension is, you know, such a common problem. And there's revenue tied there for remote monitoring if it's done effectively and you meet certain qualifications with patients. So
There's lots of potential there. The thing I will say about providers that we need to be mindful of is they don't want to know every blood glucose and they don't want to know every blood pressure. The volumes of data are way too much. And so what we need to be careful is not to alert

Jordan Cooper   17:35
Mhm.

Kathy Mazza   17:52
them with every blood pressure that's over 140 over 90 because my goodness, like that's why somebody is on hypertension monitoring. They're, you know, and somebody who's 140 over 90 who was 160 over 100 last week is very different than somebody who was 120 over 80. So

Jordan Cooper   18:01
Yeah.
That's great.
Right.

Kathy Mazza   18:12
There's a lot to be learned from small pilots. And what I tell people is don't do nothing. It's a little overwhelming. Nobody's really sure what to do. But if you just do one small thing, you really will learn more than you might imagine from that small project.

Jordan Cooper   18:22
Mhm.

Kathy Mazza   18:31
And so what we're trying to say with the projects we're planning now is how do we let a doctor know that this patient who's diabetic is, there is 6%, has better blood glucose control within, you know, 6% better this quarter. This one,

Jordan Cooper   18:49
Mhm.

Kathy Mazza   18:49
spikes on Fridays, their blood glucose goes up Fridays at 530. Maybe we have a conversation about happy hour. What's happening on Fridays? You know, and this, you know, this hypertension patient, you know, is trending this much better or is it within range 80% of the time where it was only 50% of the time six months ago?

Jordan Cooper   18:55
Mhm.
Yeah.

Kathy Mazza   19:13
so that we give them insight, not numbers. If we don't help them manage the volumes of data that are coming at them, they'll never buy into these projects. But I wouldn't.

Jordan Cooper   19:24
So Kathy, I love the anecdotes. I love the illustration. We are approaching the end of this podcast episode. And I think I'd like to ask you one final question about your outcomes over time. As we mentioned, Northwell got started with AI pilots a little bit before the rest of the country. And one of the things that AI does very well is identify trends over time.

Kathy Mazza   19:34
Certainly.
Mhm.

Jordan Cooper   19:47
You're mentioning these meaningless alerts and how to identify what is a meaningful alert and when to notify that, embed it within the provider workflow. So the question I'd like to pose to you is, what has been kind of the result over time of what has been the impact of having these AI
chat bots and new iterations of manifestations of AI at Northwell Health and any words of advice to other organizations who are earlier in the process in Northwell.

Kathy Mazza   20:20
Sure. So we, you know, I've mentioned this robust care navigation post discharge that Northwell offered. Our readmission rates were already good. Like in 2018, we were, you know, doing pretty well on that measure. And with this technology, the patients who participated in it,

Jordan Cooper   20:30
Mhm.

Kathy Mazza   20:40
had a reduction. And I remember my academic advisor at the time saying to me, looking at the numbers of my data to start and saying, boy, if you can impact this number, you're really getting blood from a stone at this point because it's really, you already have good reduction of readmission. So

Jordan Cooper   20:48
Mhm.
Yeah.
Right.

Kathy Mazza   21:01
this technology would have to really do something special to make it a measurable difference. And it did. So I think that what I would tell people is that, you know, these technologies are there. Be careful in

Jordan Cooper   21:06
Mhm.

Kathy Mazza   21:21
setting your thresholds. If you create a lot of false alarms, you really reduce the effectiveness of them, ironically. You can use that data certainly to know that somebody is trending 10% higher over a month or in a value that you want lower or whatever.
But if you alert nurses and doctors, it just becomes alarm fatigue and it becomes less meaningful. So working with providers to understand when would you, what could we tell you at your patient's office visit that would be helpful for you to know versus when do you want us to call you on Sunday morning?

Jordan Cooper   21:54
Yeah.

Kathy Mazza   21:59
for about your patient and let you know that. Because if you trigger them constantly, and we found we pretty quickly reset our yellow alerts because, you know, one patient said within a few weeks of going on the program,

Jordan Cooper   21:59
Mhm.
Mhm, mhm.

Kathy Mazza   22:18
When he got a call from the nurse, he was like, oh, am I yellow again? And so, you know, just understanding that, wait a minute, we're starting to over monitor in some ways because you have so much ability and such scalability. So, you know, set the thresholds where you really would do something.

Jordan Cooper   22:22
Yeah.

Kathy Mazza   22:37
thing, not just to know. And then use the data about that you know to identify the trends for the provider so that they can say, okay, you've really been in good control on this number. Let's look at this other thing now and get insight rather than data.

Jordan Cooper   22:37
Right.
Well, Kathy, I appreciate everything that you've given to us today. Thank you for joining Healthy Data Podcast.

Kathy Mazza   23:02
My pleasure.