S4E10: GenAI Referrals Extraction (ft. Cori Pace & Chaitanya Vempati, Memorial Hermann)

Jordan Cooper   3:06
Yeah.So Cori, you noticed the problem at Memorial Hermann that was facing your clinical admissions team. Eventually, Gautic Chaitanya and he began formulating a solution. But before we get there, I think our listeners may be interested in understanding a bit more. Is there a system, a process at Memorial Hermann whereby AVPs are empowered to come up with ideas and askfor budget or solutions from the health system leadership team? And how does that process work? Because I think a lot of our listeners are saying, yeah, I am encountering problems, but I don't know what to do about it. Maybe I don't have a personal relationship with the system CTO or COO. How did you go about saying, I'm hearing from my team that there's a problem and I want to escalate and see if we can find a solution? Did you just...

Cori Pace   3:47
Yeah.Yeah.Yeah.

Chaitanya Vempati   4:02
Yeah.

Cori Pace   4:06
Yeah.

Jordan Cooper   4:07
hear about Gen AI, how did that process work?

Cori Pace   4:10
So we actually have a really great platform for anybody within our system to submit ideas. It's called iGenerate. You can put in anybody from, I mean, any level of our organization can submit an idea, hey, this seems like a really cool thing.submit it, and then we have a fun little thing called Shark Tank that once, you know, they all get vetted through our system. We have hundreds of ideas and then take kind of the top ideas that look like they might be really successful for multiple patient experience, safety, revenue.And then it and it goes to this Shark Tank style thing where you present it to to our executive level team that has representatives from, you know, our ISD finance, kind of the spectrum of our C-suites. And that my idea got taken to that to then be able toget the system level support needed. I will also say each, if you are not going through that process, we all have, each campus has their own process where you can take things up through your own leadership team. And then we have individual campus level meetings with our ISD teams. Shatanya, you can

Jordan Cooper   5:12
Mhm.

Cori Pace   5:29
talk more to about other projects in our AI council.

Chaitanya Vempati   5:29
Yeah.No.

Jordan Cooper   5:33
So, Chaitanya, can you talk about the intake process? How did you become aware of this project? Did you sit on the Shark Tank pitch session? And if not, who did and how did they filter this project down to you?

Chaitanya Vempati   5:45
Yeah, I think the team presented it to the Shark Tank. And we have, I mean, we refer to our Chief Digital Officer. The Chief Digital Officer typically sits on that Shark Tank. We also have people representatives, project managers who sit on the Shark Tank and so on. So this was one that was highlighted as some things

Jordan Cooper   6:06
Mhm.

Chaitanya Vempati   6:09
as we are building up our maturity in AI, especially agentic solutions and using large language models internally, was something that was passed down to me from those teams who were sitting on those platforms saying, hey, is this something that is even possible or even feasible to build out internally? Because most health systems today, when they...

Jordan Cooper   6:14
Mmh.

Chaitanya Vempati   6:31
When they think about agentic solutions, a lot of them are not thinking, "Hey, can we even build that first?" Like a lot of them are utilizing vendors, utilizing either startups or established EMR systems and tools within them to be able to use them for these types of solutions.And this was something that was passed down because of the novelty of the particular solution and how useful it was going to be for the system. So that's how the project came about to at least our group to be evaluated.

Jordan Cooper   7:05
So you decided, how did you decide to build this solution instead of purchase it from a vendor? And second, what Gen AI model, are you leveraging an open source Gen AI model to build your own solution? Or are you leveraging a commercial clot or Open AI or Gemini?Product.

Chaitanya Vempati   7:26
Yeah, let me talk to the first part of that particular question. So I think constantly Cori and I go back and forth as soon as we see anything in the industry, like, hey, is this something that can replace, you know, our argument what we are doing, right? Like, so that is a constant search. Cori has done tremendous research. I've done research to see if there is anything out there.or even off-the-shelf AI systems like Claude or Copilot or anything like that, can they do the same thing that we are trying to do through a series of prompts, right? So after that evaluation, it became fairly evident that this is a very specific use case. TIRR is a very specialized hospital and the workflow that they're looking at for the rehab intake process.

Jordan Cooper   7:48
Mhm.

Chaitanya Vempati   8:12
is a very specific workflow that requires some thought process to be built in. Once we've decided it, and because of the novelty of the solution, we chose to think of it like an internal lean startup, right? Like an agile team. And one of the first things we did was that, and Cody remembers this,

Jordan Cooper   8:16
Mhm.

Chaitanya Vempati   8:33
We said, hey, Cori, we can't take on this yet. Let us do some discovery internally and think this process through. Let's look at the architecture and think it through. Out of that came out the answers to your question, Jordan, which is like, hey, what model should we use? Should it be an open source model? How can we pitch this?

Jordan Cooper   8:39
Mm.

Chaitanya Vempati   8:52
technologies to view. So to put things in perspective, like so we use AWS Bedrock as one of the, you know, secure LLM orchestration platforms, along with internal AIML tools that we've built up. So we use for this particular intake a mix ofopen source models, some things from OpenAI, and primarily the big chunk of heavy lifting is done by Anthropic and Claude's model coming out of AWS Bedrock in a secure instance hosted by Memorial Herman. So those are the types of things that we use to orchestrate this. AndA lot of work we do, we did here is not sort of like 1 prompt, like take this, run it through one prompt and do it. Yeah, when we first looked at it, you know, the volume of the pages, the types of faxes we are getting in for these were so complex.

Jordan Cooper   9:40
Mhm.

Cori Pace   9:51Mm.

Jordan Cooper   10:03
Mhm.

Chaitanya Vempati   10:03
several iterations to be able to even architecture it. And then we decided like, hey, let's bring on the nurses. Let's work together. We think we can accomplish this.

Jordan Cooper   10:12
So I'd like to delve now, and I think our listeners would like to hear you delve now into a dual explanation. How did you collaborate between both of your teams? I think there's a lot of times solutions break down when technology and the business owners are in silos.and the business owner makes a request, and technology runs off and builds a solution, and then a year later comes back and says, voila, here we are. And they're like, oh, that's a little wrong, or that's not a thing, and you didn't get the buy-in, and people aren't sure about it, and then it just kind of fails. But how could you put, I think our listeners would like to hear,

Chaitanya Vempati   10:33
Yeah.

Cori Pace   10:41
Mhm.

Chaitanya Vempati   10:41
The.

Cori Pace   10:51
Yeah.

Jordan Cooper   10:53
Exactly how, once, Chaitanya, once your team decided to go ahead and build something, how did you collaborate with Cori and her team to ensure that there is a sufficient amount of clinical buy-in and that you were addressing the actual needs of how do we make a patient visit, I think the metrics was turnover.from admission to next stage of clinical care more efficient and more successful.

Cori Pace   11:18
No.

Chaitanya Vempati   11:21
I can start and then hand it over to Cori and you can basically fill in the missing pieces if I may. So it was clear, like this problem was fairly challenging, right? And it was also clear that the data science and AI group does not know

Cori Pace   11:26
Yeah.

Chaitanya Vempati   11:40
the clinical nuances that Cori and her team at TIRR know, right? So we decided like this is not going to be something that's going to be successful if we want to follow it. If we can go take this as a requirements document and go inside and build something and come back, it was never going to work. So what we said was that we're going to meet biweekly.and weekly sometimes. And we need people like nurses, actual users on the call weekly. So we said we'll do it in an agile fashion. The objective would be to bring something of useful nature to each of these meetings. So let's work on for some, identify 3, 4 things to fix.small things that should only take a week or two to fix, identify those, fix them, and show up with the working prototype at these meetings. And the key was for us to start with the working prototype right from the beginning. We had something rudimentary working almost weeks into the project. And since then, it was all about constant iterating.And it is imperative and I think I want to reiterate what you said.Solutions such as this will not be successful if they are built in silos. And it is actually to be talked through that the business is building the solution, utilizing the technology and the engineers behind the scenes. They are just listening to the constant feedback and iterating. And that's the only way to be successful in this domain. Cori, you might want to add something.

Cori Pace   13:06
No.Yeah, I think that was great summary. We definitely have the weekly, bi-weekly meeting, bi-weekly and then sometimes weekly. We have a Zoom channel that the nurses are also included on where every single, like I get things popping up all the time. Hey, this file is in, I'm not seeing it yet. And thenthe engineer, oh, yep, here's what's going on. And so they'll fix it real time. In these meetings, it's been really helpful to just pull up a few summaries and the nurses will be like, this is amazing. What would be even more amazing is if we had X, Y, and Z and the team will either say, yes, we can do that immediately or that might be ana more of an optimization phase. They have worked very closely with our nursing team to really kind of fine tune the information. The nurses have really spent a lot of time validating the information that is being pulled together to make sure that it's not missing things oror giving incorrect information. And it's been really from the beginning, pretty accurate from the very beginning.

Jordan Cooper   14:23
Cori, what fears did your nurses, physicians, and other clinical staff have at the beginning of this process when they first started talking to Chaitanya?

Cori Pace   14:34
I think I probably had the most fears of we take a lot of really specialized patients at this hospital that a lot of other rehab facilities don't. And so what I really wanted to make sure of is that as we were building this model,that we weren't, and we don't to this day, have it say, yes, they're appropriate for rehab or no, they're not appropriate for rehab. Because what I don't want is for our clinical teams to lose that decision making and that clinical judgment themselves. I want it to pull the information together for them.and then then be able to make that decision of, yes, this is an appropriate patient. It may not be a typical patient that as of right now, our tool might say would be more appropriate for another level of care because we do take those patients here. But that may be down the road as it learns more and as we are feeding it more information,it does start picking up those nuances for those patients that are appropriate but not typical for rehab. So I think that was kind of my fear is.Does our team lose that clinical decision? Do they rely on it too much? Do they, as we have new people come in and they are relying solely on this, I mean, I think this is true for all of AI, myself included in how I use it, but do we lose our own clinical decision?judgment because we're relying so much on it.

Jordan Cooper   16:08
Could you, we've talked a lot about getting to the solution. Just for the benefit of our listeners, can you provide an overview of what the solution actually does from the perspective of an end user?

Cori Pace   16:21
Yeah, it's really cool. Essentially, it gives us the very first page is a checklist of all of the things that we have said is required for an admission or for a referral. And it will tell us immediately out of these 800 pages, all of these elements are here and these three things are missing. And so our intake team can quickly go back to the referral source and say,hey, we've got your referral, we're processing it, but we also need these three data elements so that we can finalize the process. And then, and then it just goes, we have given it our pre-admission screen data, what we're looking for, and it goes head to toe. Here are, here's the head to toe summary of the patient present, you know, summary of present.

Jordan Cooper   16:49
Mhm.

Cori Pace   17:04
hospitalization, any lines, tubes, drains, labs, imaging, vitals, kind of just the whole summary in about 3:00 to 5:00 pages, maybe, Shatanya, instead of hundreds of pages.

Chaitanya Vempati   17:18
Yes.

Cori Pace   17:23
So then our team is able to look through that, quickly see it, and then it is not integrated into Epic at this time. And so they have a summary, and then they're using that summary to help complete their documentation within Epic.

Jordan Cooper   17:24
Um...How does the model's output compared to the manual summary of that same 800 page PDF that you used to do? Is there like an 80% overlap or are there, are there, you know, is there any hallucination? WhatAre you seeing, how does it compare to the manual process?

Cori Pace   18:03
I think for the most part, it is much more comprehensive. What we've heard feedback from the team is it has caught things that they may have missed because it was buried in that. I have not heard, and Shatanya, I don't know if you have of really any hallucinations. It has been

Chaitanya Vempati   18:20
Billions.

Cori Pace   18:23
pretty accurate from the get-go.

Chaitanya Vempati   18:25
Yeah, I mean, I think a lot of feedback that we've got, I mean, and we have a spreadsheet that tracks every feedback and our manager over it is very **** about tracking it because we want to make sure that we don't miss anything. So it's not, there's a lot of things that are.sort of buried in this, you know, 800 page faxes. It's not always in a standard format or it's not always even in the same order, right? So people tend to miss things and it has caught things in some page in the last few pages of the whole fax. So we get routinely feedback like, hey, how amazingly it has caught things that they would have probably missed.

Cori Pace   18:55
Mhm.

Chaitanya Vempati   19:09
Now, you asked a very, very specific question, and an interesting one on hallucination, and you asked about the fears of what this model might do. And joining both of them, when we first started on the project, the AI team was tremendously fearful of hallucination, because that was a big thing back then, right? Like, because it's a large context of data,and we are asking specific information, is it hallucinating from this large? Like all of the data cannot fit in one context. It's not like you could upload this document new cloud or Chad GPD and say, summarize this, like it's too big. So one of the things, the techniques we used was just chunking and breaking it into pieces and trying to make sure that each piece onlyextracts information about its piece rather than thinking through the big picture and all of that, and then combining all of that into a big picture. So we run through various layers of AI, not just one AI, and that has almost eliminated hallucination. The biggest concern and the trick that we had to build out is that...it missing things. Like we were very careful about, like we should not have a situation where it is present in the document and the AI thinks it's not there. We initially had a few cases where that was happening and we quickly addressed it by looking at various techniques, changing models, etc.

20:18
Yeah.

Chaitanya Vempati   20:29
You know, doing the AI techniques that we are routinely used to, but we never we were worried about hallucination, and I think we engineered a solution to engineer out the hallucination and make sure it is accurate most of the time.

Jordan Cooper   20:44
I think our listeners might be interested in hearing, just be asking themselves, is the primary data source, this PDF, included in the chart as a reference in case someone does have a question and wants to look at page 621?

Chaitanya Vempati   20:56
Yeah.

Cori Pace   20:57
Yes.

Chaitanya Vempati   20:58
Yeah.

Jordan Cooper   20:58
Okay, and okay, and then I want to ask, and you say it's not integrated into Epic. How is this information being integrated into the EHR? How is that information, like what is the data source from Epic's perspective?

Cori Pace   21:16
So our nurse liaisons complete a pre-admission screen that has fields that they have to fill in and complete. And then so they are using this summary and the like they're validating that the summary is accurate from the references that it gives for the for the actual record. And then they are

Jordan Cooper   21:23
Mhm.

Cori Pace   21:35
basically transposing some of that information into our pre-admission screen.

Jordan Cooper   21:40
And what's the degree of, is there copying and pasting or is it really just retyping and transposing information and to what extent has that led to errors of any kind, even typos?

Cori Pace   21:53
Yeah.

Jordan Cooper   21:55
Not a big problem.

Cori Pace   21:55
Um, no, it hasn't really been a problem that we've that we've run into.

Chaitanya Vempati   21:59
Yeah.

Jordan Cooper   22:00
A quick question I think we can easily address. Any difficulty in applying the LLM to a PDF as opposed to another data format?

Chaitanya Vempati   22:08
Ohh.Yeah, I think, I mean, I would say like there is a there is a difficulty here that you mentioned. I'm glad you are asking this question. It is a technical challenge that we spend, you know, nights and weekends over. It's not a question about the format of PDF. It's just the fact that the PDF are essentially, you know, faxes like...files converted into PDFs, and they come in all kinds of different formats. Sometimes they have images embedded as text, sometimes that is text and images, sometimes that is only text, right? So it's the mixture of formats that is handwritten notes sometimes.

Jordan Cooper   22:31
Yeah.

Chaitanya Vempati   22:50
There is all these different permutations and combinations, imaging results, and so on. The challenge here, the technical challenge, is to be able to have an LLM configured in such a way that it's able to look at all of these contiguous things as one context, right?

Jordan Cooper   22:57
Mhm.

Chaitanya Vempati   23:09
You need to take in a bunch of text, an image in the middle, and a bunch of text all into one context. And the image may not be like an X-ray. The image may be still text scanned in as an image, right? So that was the big technical challenge, not particularly PDF per se.

Jordan Cooper   23:21
Mm-hmm.Mm.

Chaitanya Vempati   23:29
It's all of these various elements embedded and mixed into one grouping. That was the biggest challenge to solve.

Cori Pace   23:37
Ohh.

Jordan Cooper   23:37
So it sounds like your LLM has been able to accommodate a mix of different types, sizes, and formats of data sources without needing to normalize, aggregate, or deduplicate. Is there any, would there, is there any, are you leveraging your integration engine at all to do that sort of process before feeding the data into the LLM?

Chaitanya Vempati   24:01
No, I mean, we are basically, this particular bot essentially looks at one fax as its own entity. So, you know, when we get multiple faxes on the same patient, we typically they're all combined into one sort of document.

Jordan Cooper   24:10
Mm-hmm.

Chaitanya Vempati   24:19
And we somewhat purposefully designed it this way as we are building this out, because, I mean, we didn't want to build out like huge integrations and, you know, we want to see that the confidence in the output it produces is there before we go integrate into a bunch of systems. And routinely, these are external referrals.coming into the organization. So we may not have all of the patient data internally to normalize anyways.

Jordan Cooper   24:39
Mhm.

Cori Pace   24:44
Yeah.

Jordan Cooper   24:47
So we are approaching the end of this podcast episode, and I think this is a very interesting conversation, as I'm sure our listeners do as well. I have a few more questions I want to throw at you, but I'll throw them at once and let you kind of pick and choose which ones you'd like to respond to since we don't necessarily need to address all of them. One, I think some of our listeners may be just...thinking very generally, how has this impacted clinical workflow and revenue? And the second thing I think some of our listeners may be thinking is there are tight margins at healthcare delivery systems across the country, and they're only becoming tighter since recent changes at the federal level with legislation.

Cori Pace   25:25
Mhm.

Jordan Cooper   25:27
I'm wondering to what extent, many organizations are wondering, if we build a new software solution at our organization, to what extent can we package it and resell it as a product to other organizations with the same problem, since software obviously has a higher margin than delivery of healthcare services,And to what extent can we transform Memorial Hermann not only to a healthcare delivery organization, but to a software product vendor?

Chaitanya Vempati   25:56
Yeah, I think I'll let Cori at least address the first part of the question, which is like, how is it impacting clinical workflows? And I think you did mention patient experience. We might want to address patient experience because that's why we did this all of things to begin with, Cori.

Cori Pace   25:58
Yeah.Yeah, I think patient experience was definitely one of the top drivers. And one of the things that we frequently hear is that, especially for these complex patients, it just takes us a while to get to a yes or a no. And so we are frequently looking at how do we speed that up? How do we get?give them an answer one way or the other either way. And so that is definitely that piece. The other component is that...We are not able to just add stuff like our referrals are going up, which is great, but we are not really able to just throw more and more and more staff at them based, you know, exactly on what you just said with margins being tight. And so it's how do we how do we look at other alternatives and technology is there and.Why wouldn't we use that to help supplement, not necessarily replace? I think we have so much more that our team can do if they are not bogged down into digging through 800 pages. Now they can spend more time talking to the patients, talking to the families, being in front of the patient,doing an actual visual assessment of the patient sometimes when that has been harder. So I think taking some of this tedious work off of them will, and again, we're at the very beginning of this, like our full team really just got access to this last week. We went from just partial team to full team.having access. And so I think the full impact is yet to be seen as they really learn it and as we continue to modify it and optimize it. But that is my vision and my hope is that taking this tedious work off of them will actually help them work at the top of their license more and be able to be in front of the patient and having more conversations with the families.

Chaitanya Vempati   28:08
Yeah, I will add to your second question. I mean, can we commercialize this and should we, and I think there is a model for all healthcare systems to sort of collaborate in tools such as this and have a licensing model where, you know, you are compensated for all of the work that you've kind of put in, but also they are able to reap the benefits as opposed to us starting from scratch.And in general, I mean, I'd like to sort of appeal to the audience, whoever are listening, I mean, this is a day and time where AI is accelerating so, so fast that, you know, I think they would love to, I mean, they would want to invest in building out tools for their organization because not everything is solved by the vendors. AndSome of these are like very custom to your organization. Things are, tools are evolving so that building some of these things may be, I mean, a commodity in two years or so, right? So I think just an appeal to everybody to move fast and just start adding value because a lot of these technologies is, and is going to become a commodity in two to five years.I know so.

Jordan Cooper   29:15
I'd like to thank you both for walking us through this very interesting Gen AI use case and how it's expecting patient safety, patient experience, revenue, and allowing clinicians to operate at the type of their license. For our listeners, again, this has been Cori Pace, the AVP of BD and Patient Solutions forat the Institute for Rehabilitation and Research. Cori, thank you for joining us today.

Cori Pace   29:38
Thank you.

Jordan Cooper   29:39
And Chaitanya Vempati, the AVP of AI and Analytics for the Memorial Health Health System. Chaitanya, thank you for joining us today.

Chaitanya Vempati   29:47
Thanks, a letter.