S4E22: ROI-Driven AI Business Use Cases (ft. Stephen Weber, UChicago Medicine)

Jordan Cooper   1:15
Mhm.No.Yeah.

Stephen Weber   1:36
We really tried to develop programming and a focus that would say, yeah, this is core to our business mission and what we're doing, but at the same time, too, required some very specific attention before we just turned it loose.

Jordan Cooper   1:48
So I think a lot of organizations across the United States in 2026 are encountering AI and developing their strategy and trying to figure out what their governance model is, what the security implications are. How do they even clean up their data so that they're feeding it with appropriate models without risking the release of PHI? What are some of the primary

Stephen Weber   1:58
Personally.

Jordan Cooper   2:08
concerns and business drivers of Northwestern, of New Chicago Medicine's adoption of Gen. AI.

Stephen Weber   2:14
With.Well, I think it's actually the same for all of our competitors and colleagues here in Chicago and across the country too. And you hit on some of the same themes. You know, the work that we might do around AI, whether it's with foundational models or in terms of things that we might be building ourselves, it is predicated on the nature of the data that we have and also the, you know, the importance and the sanctity of that day, of those data, right? So

Jordan Cooper   2:38
Yeah.

Stephen Weber   2:40
So the work that we had built in, even before we had heard of the first large language model, was really around data governance and also cleaning up our data. So we made a lot of our investments in quality and safety. We were never a place during that, hired a lot of kind of past their prime clinicians who said, I want to get into quality at this point. We've always been very quantitatively oriented. And what we built our quality and safety programs on

Jordan Cooper   2:45
Mhm.Mm-hmm.

Stephen Weber   3:03
were around data and analytics and really establishing an analytic source of truth for our operations, for our quality, and certainly into our strategy and finance as well. So that was the foundational piece to it. And it's not a step that can be skipped. So if you're going to, you know, we're hearing these pitches from big companies, existing vendors and startups as well,

Jordan Cooper   3:06
Uh-huh.Mhm.Okay.

Stephen Weber   3:26
But it really is going to be predicated to have we done the due diligence in terms of cleaning up our own data and having it ready to serve us in these models.

Jordan Cooper   3:33
So I think, again, a lot of listeners can identify, sure, we have these, you know, ambient listening scribes. We have a lot of other vendors, sometimes large EHRs, sometimes independent startups, all promising a lot of value through AI. But I think what you're talking about is the less sexykind of prerequisite of selecting A vendor, which is how do we clean up our data? So I'd love to ask you to just delve into, okay, we know that we're going to have those conversations maybe 6, 12 months down the line. Maybe we're even having them in parallel. But what are we doing right now to improve data quality? What are we doing with the data governance? What are we doing with...

Stephen Weber   3:57
Thank you.Yeah.

Jordan Cooper   4:13
with aggregating, normalizing, deduplicating the data, ensuring that it's high quality data, define that, how do we go down that journey?

Stephen Weber   4:21
Well, the answer about what we're going to do is yes to everything that you said. We need to make sure we have the right expertise on the ground to be able to do it. And we have to realize too, not everyone who's listening to the or watching the podcast right now will necessarily have the same set of resources and expertise to apply. So really understanding what you need to invest in in terms of either the personnel and expertiseor the partners with whom you're going to work, who are going to help clean up the data. It is a lot of work around, like you said, you're looking to pull out redundant data. You're looking to make sure that what you think you're measuring is actually being measured. And that is the, you know, I also hate to frame it that way. It is the unsexy work, but it really speaks to the issue that's applied going forward too, which is the discipline that it takes.

Jordan Cooper   4:52
Yeah.

Stephen Weber   5:08
To recognize that each of these steps, you need to have some discipline about what you're doing. So in terms of the governance, you want to have a broad representation of data governance. You don't want just a bunch of, with all due respect to them, technology hacks around the table when you're thinking about data. You need clinical, you need operational, you need strategic leaders. That way, you're also sending a message to the organization that

Jordan Cooper   5:09
Yeah.The.

Stephen Weber   5:29
This is really important. This is a key asset, a key tool, and essential for our success going forward. So that governance becomes the first piece of it. It's investments in understanding that your systems are, again, delivering what you think they are in terms of the analytics and doing that validation. But the discipline really picks up with that part that you talked to next. And again, I'm still not sure it's getting into the sexy part,

Jordan Cooper   5:31
Mmh.

Stephen Weber   5:51
But it's the idea of looking at what are we going to solve with these new tools. And I think for a big piece for us was we realized there were a lot of people who were willing with good intentions to come and sell to us products and solutions that are important and that may or may not deliver on the value that we expect. But we needed to start with the idea of prioritizing ourselves

Jordan Cooper   6:00
Uh-huh.Yeah.Oh.

Stephen Weber   6:12
around what are the problems that we're facing today. And for us, we built in with our data governance and then our prioritization schemes in IT and in the clinical services to say, we start everything by saying, what's the problem that we're trying to solve? And then we move into, and this is built into our meeting agendas and how we're talking about it.

Jordan Cooper   6:14
Mhm.Okay.Mm-hmm.

Stephen Weber   6:33
What's the capability we need to address that problem? And only then do we dig into the solution. And I think for a lot of people who are taking part in this right now, we can all appreciate that despite our best efforts to do it that way, without discipline, we'll often start with a solution. We'll hear a compelling vendor story. We'll hear from a colleague at another institution.

Jordan Cooper   6:38
Yeah.

Stephen Weber   6:53
We'll listen on a podcast and hear about something that was terrific. And then we'll try to come back and say, what problem can we solve with this? But that's just for me, management malpractice. I think we need to start by saying, what's the problem we're trying to solve?

Jordan Cooper   7:06
So Stephen, I think a lot of the listeners to Healthy Data Podcast really do appreciate when we dive deep into a specific example. Could you select one priority? It doesn't have to be the top priority, but one priority that UChicago has addressed in the last few years that you've been involved with. And once you identified that priority, what was the problem you were seeking to solve? How did you identify a solution that you thoughtAI might be able to support in addressing that problem? And then finally, how are you able to kind of implement and evaluate the value of that particular solution?

Stephen Weber   7:42
I love, and that's exactly the framing for it, right, Jordan? And look, let's take something that, again, it's become ironically tried and true at this point, but could be familiar to a lot of people. And that's how we thought about Ambient Scribe. We were very much an early adopter with our vendor. We actually looked at it as the problem of saying,

Jordan Cooper   7:56
Mmh.The.

Stephen Weber   8:02
We have capacity and access problems. Again, I think that's probably familiar to a lot of people listening right now. That's true in ambulatory and an inpatient, but we'll focus on ambulatory right now. And we were seeing that we were not, we were really having difficulty asking of our clinicians to really come up to the volume that we need to see in order to meet the demandfor new patients, existing patients. And we were like everyone else, trying every system we could do around this. So we actually looked at Ambient Scribe as solving the problem or helping to address the capacity problem. So as we looked at that, we thought about, here's the problem. We can't get patients in to see us in an ambulatory setting. I'm in practice. I'm an infectious disease doctor and epidemiologist. So

Jordan Cooper   8:34
Mm-hmm.

Stephen Weber   8:44
I was living this too. We said, what are the capabilities that are going to help expand that access? And so the ambient scribe was one of a number of measures that we deployed. Some of them technologically oriented, some more based on scheduling, some other technology oriented, though, that included things like patient self-scheduling, whatnot.

Jordan Cooper   8:45
Mhm.Yes.Uh-huh.

Stephen Weber   9:04
But what we saw was that, all right, we saw this opportunity around the Ambient Scribe, and we said, all right, that's a solution we're going to go for. When we looked at the existing vendors at that time, we were looking for a partner with co-development. We were looking to say, who's somebody going to grow with us? This is ironically over two or three years, right? It's a different environment now. We see the established players.

Jordan Cooper   9:08
Uh-huh.Correct.No.Uh-huh.

Stephen Weber   9:25
We know who's doing what, but at that time, we were really trying to pick a horse that we thought was actually going to grow with us or race with us to stick with the analogy. And as we move forward with it, though, we set up a very, very complex set of measures that go with it. You mentioned that in your question. And this becomes absolutely critical because there is more, we as an organization spent a lot more time

Jordan Cooper   9:31
Right.

Stephen Weber   9:47
looking and anticipating return on investment and return on value when we were signing contracts. We were not always as good at coming back and really tracking the value later on. So we built up a scorecard around what we expected to see, and that included throughput on patients. It included measures around provider burnout. We used actuallya rating that truly originated with the space program that looked at cognitive load for the clinicians. We looked at patient experience, and we said, we're going to track these. We're going to look at same store over different visits, if you will. We're going to look between and amongst clinicians. And we committed in a way, though, that was actually a commitment and not a pilot. We really did not want to be

Jordan Cooper   10:15
Huh.

Stephen Weber   10:31
death by pilot as we do this. So there were a lot of organizations at the time who were getting a lot of play for having gotten 10 license and rotating them around, you know, different practices. We went in with about 200 licenses as our first take. We wanted to get beyond the early adopters and innovators and penetrate in some of the areas where folks might struggle a little bit more with the technology.

Jordan Cooper   10:33
Mhm.Mm.Mm-hmm.

Stephen Weber   10:52
And we went live with a focus on driving adoption of saying that we want to make sure we have the resources in place so that you can actually make use of this tool. So this all becomes important because as we started to look back at the day and accumulate experience, we were getting warm and fuzzies from people. This is great. We're glad you're doing this.

Jordan Cooper   11:02
Mmh.

Stephen Weber   11:11
But there were a couple of things that went with it. We expected from the beginning, we were very transparent with our medical staff and providers. We said, if you're adopting this tool, we're going to expect your template to expand. And I know that was really, really, we had a lot of pushback.

Jordan Cooper   11:11
No.Yeah.Okay.

Stephen Weber   11:28
externally from that. We had a lot of our folks who were going out and talking about this, saying, no, you really can't do that. That's unfair. People don't want to take on. And look, and we get it. I understand clinicians can feel really overburdened. But at the same time, too, I think for us, we were fortunate that people were seeing how difficult it was to get patients in. And so for those earlier, those 200 folks who took the licenses on,

Jordan Cooper   11:43
Mhm.Mm-hmm.

Stephen Weber   11:50
Their templates expanded and they added, I think, one or two patients a week. So A modest number, but still the idea of saying, if the problem we're trying to solve is access, we need to expand access. But the results themselves, Jordan, which I'll come to in a moment, I think were really surprising and really interesting to us and point to this idea of a disciplined approach.

Jordan Cooper   12:00
Right, I.And what were those results?

Stephen Weber   12:11
So we saw, we did see that slight increase in the patient, in the capacity. For all the talk of pajama time, for all the talk of beleaguered physicians, what we actually saw was a discrepancy between the measure time that we were saying, we're an epic shop, and so we're able to look

Jordan Cooper   12:14
Uh-huh.Mhm.

Stephen Weber   12:31
and understand how much folks are looking at the medical record outside of their clinical practice. And we saw that there wasn't a dramatic change. It might have been a few minutes here and there. There were some clinicians for whom it was more pronounced, but across all of them, it wasn't huge. What did change, though, was the perceived time in the medical record, that folks who envision that they might be spending upwards of an hour or two in the medical record

Jordan Cooper   12:45
Mhm.Yeah.

Stephen Weber   12:54
outside of clinic, were now saying that they were only spending half an hour. It was really hard, if not impossible, to validate that. But the perception was there, and that was important. We were getting enormously positive feedback about it. But the one that really blew us away and that we suspected, and certainly the vendors said it would be important, we saw dramatic changes in patient experience.

Jordan Cooper   13:01
Huh.

Stephen Weber   13:14
Our patients notice the difference. They notice the engagement of the clinicians. I was not someone by the nature of my clinical schedule, I'm not hiding behind a keyboard in clinic, but I have to say, even with now using the ambient scribe technology, I felt more present. I felt less cognitive load and my patients really noted it. And we could actually compare not just between

Jordan Cooper   13:24
Mhm.

Stephen Weber   13:37
ambient scribe users and non-users, but even ambient scribe users who from time to time would forget to turn on the microphone, we could see differences in terms of the patient experience. So for us, we started, we want to expand access. We succeeded to a small degree, but we had a comprehensive measure set that actually allowed us to detect a broader range

Jordan Cooper   13:46
Mm-hmm.

Stephen Weber   13:59
And then we went live. We went broader and broader from there. And now we see this on the inpatient side. We've extended this to our trainees. And we think of it as a real success story beyond just the kumbaya feel that people are talking about with ambient scribe.

Jordan Cooper   14:12
Stephen, I love how you use data to tell a story about not only the return on investment in this ambient scribe thing, but the surprises in what the data tells you, which is that perception is different from reality, but there is a lot of value in that perception. There's additional patient volume. You did it slightly address and improve patient

Stephen Weber   14:27
Tumba.

Jordan Cooper   14:32
access, but the patient experience and the provider experience and the improved probably retention rate, but definitely kind of the sense of engagement is something that could be very attractive for capturing market share. Now, that said, I would like to talk about data quality in another kind of vein.which is we spoke earlier in this episode about improving the quality of data for AI tools so that it ingests quality data. But of course, Ambient Scribe is ingesting auditory data. So I guess we may have to adapt the question to, you know, how much, what is the process forimproving what processes do providers need to go through to review the data output from Ambient Scribe and make sure that it's accurate and that it's placed into discrete data elements that are useful and incorporated into the EHR and how do you avoid inaccurate data?That could causeway problems down the road, especially when you're trying to do your analytics.

Stephen Weber   15:35
Yeah.Yeah, and again, you framed the question around the value, and we started this conversation around the value of having high quality data, obviously independent of the implications for model building that we all want to have the best possible data on our patients that leads to better care. You know, we started with a, it can be a little bit heretical and maybe a little provocative view that when we look

Jordan Cooper   15:44
Mhm.Mhm.Mhm.

Stephen Weber   15:59
at this issue, whether around ambient, scribe or otherwise, is understandably, we all want to be sensitive and we always smart about the risk of hallucinations and bias and whatnot in models and just lack of validity. But we also have to appreciate that our starting point is not an ecosystem that's free of hallucinations, that's free of bias.or free of invalid data. And so we look right now, you know, in terms of hallucinations, we don't think of our clinicians hallucinating, but we know that 10s of thousands of Americans die every year because of misdiagnosis. That's settled science. And while that's not technically a hallucination, it's still smart people getting things wrong or smart systems getting things wrong.

Jordan Cooper   16:22
And then.Mhm.Mhm.

Stephen Weber   16:40
Bias? Anyone wants to argue that there's no bias in the American health system right now, despite the best intentions of clinicians, leaders, and otherwise, is deluding themselves. So we're starting from a system that's already brokering in, at times, invalid data bias and hallucinations. That's not to lower our standards, but it's also to understand that this presents a huge

Jordan Cooper   16:41
The.Uh-huh.

Stephen Weber   17:00
huge opportunity for us to actually look at ourselves and say, how are we doing about this? How are we thinking about it? So back to your original question, is we do have, there's the expectation, and it's part of the process and the workflow around the ambient squad that the notes are being reviewed. Are they being reviewed more or less than what was historically being cut and pasted from notes and between patients?

Jordan Cooper   17:06
Yeah.Hmm.

Stephen Weber   17:23
That's a little bit harder to get at. We're trying to dig in and understand that a little bit more because if we're going to look at the quality of the data and the work that we're doing right now, it's important to look at what was the previous model before we reject anything out of hand. But I think that human in the loop right now becomes very, very important. But even as we go forward, we're going to see that evolve.

Jordan Cooper   17:24
IfUh-huh.Mhm.

Stephen Weber   17:42
because we're going to see the models, especially as they're starting to build on in, and our vendor has functionality so that you're building on previous experiences with the same patient. Each episode is not treated independently. It's going to become better and smarter around that. So yes, so we have the individual providers are looking at it and that.

Jordan Cooper   17:54
Mhm.

Stephen Weber   18:01
And we have systems built in for people to report, both through the actual vendor platform and then through our risk and safety platform, where they're seeing departures from the expectations in terms of the quality of the data that are being generated.

Jordan Cooper   18:12
So Stephen, we are approaching the end of this podcast episode. I'd like to remind you that the topic we came to discuss today is the importance of implementation and adoption in the application of digital health and AI to improve safety and quality. And again, there's a million directions we could have gone with that. We could have focused, we focused a bit on implementation and adoption and the implications of that.

Stephen Weber   18:26
There are.

Jordan Cooper   18:32
Digital health can be so many more things than just the ambient listening scribe. And safety and quality, again, there are many options. There's industrial Internet of Things solutions. There's many different ways you can take this. But let me ask you, as we are wrapping up this episode, which direction would you like to take it? Is there any kind of parting words of advice to your peers across the United States

Stephen Weber   18:43
Absolutely.Yeah.Sam.

Jordan Cooper   18:54
who are in similar positions, are similarly needing to evaluate new AI solutions, or speaking to vendors, or trying to ensure there's the right quality of data. They want to improve patient experience, provider retention, burnout is an issue everywhere. The same problems are almost universal. The solutions is a smorgasbord that different variations are available everywhere.What words of advice do you have to your peers?

Stephen Weber   19:18
I think you outlined well, and I'll answer in 30 seconds or less, right? But you outlined well the notion of just how broad the issues are and the implications. And that's why I find, and we found ourselves drawn to the horizontal interventions, if we will, that really affect all of these. Whether you're talking about a new ambient scribe, you're talking about a revenue cycle tool,you're talking about a diagnostic aid. In each of those cases, and as well, including existing cases, scorecards, dashboards, pop-ups, and information, decision support tools that we're using, is the last mile is critical, Jordan. The idea of we're generating and creating data, those data need to be acted upon.

Jordan Cooper   19:48
Yeah.Yeah.A

Stephen Weber   20:17
and turn that into practice to recognize that as useful, actionable information is really key. And we have not done enough as an industry to incorporate everything from, and this goes well beyond UX, human-centered design, implementation science, it literally is a science, to understand how we can take thetechnical output, the analytic output, and turn it into practice to actually change behavior. That's where the difference is going to be made. Because if you show me a great AI or digital tool, whether it's coming from the commercial world, from one of our scientists here, where the drop off in value in terms of quality, safety, and economics and growth are going to be,is going to be how it's deployed and how we actually extract that value in the last mile from when the output comes from the model to when practice actually changes. And so focusing on practice change and implementation is absolutely the critical piece. The model is going to keep getting better. If you and I do nothing else, the people around us are going to keep building better and better models. What's on all of us and the people who are listening is health system weave, thosefor health system leaders is to dig in and say, how are we going to make sure that our teams have access to those data, understand those data, and then are affecting their practice with those data, whether at the bedside as a clinician or working in another part of the organization. So that, I think, is going to be a real goldmine for all of us, with the benefit showing, yes, economically,But in terms of quality and safety, for sure, doing the right thing needs to be the easiest thing to do.

Jordan Cooper   21:44
If I could summarize, Stephen, I think what you're telling our listeners is that information needs to not only be high quality, but actionable. You need to understand exactly why it is you're implementing something. And it can't be because everybody else is doing it. It's the hottest, coolest, newest thing. You need to understand what are the business drivers, what are the clinical drivers, what are the operational drivers,of your organization, and then how can these tools, you know, it almost seems too simplistic to state, but how can the tools help you achieve those aims? And even if it's the best tool in the world that could theoretically achieve those aims, if it's not driving human behavior change, then it's not delivering the ROI. So.

Stephen Weber   22:23
You got it.

Jordan Cooper   22:24
Steven, I'd like to thank you so much for joining us today.

Stephen Weber   22:27
Thanks, Jordan. I really appreciate it.