S4E1: Healthy Data Podcast Chuck Christian Franciscan Health & Jordan Cooper InterSystems

Jordan Cooper   1:21
From an on on Prem to a Azure hosted Epic instance in your cloud strategy.

Chuck Christian   1:27
Sure, I'd be happy to. Well, and the thing about it is, is that the our cloud strategy started out a little bit different. We actually were private hosted for five years and then we're going to re-up that private hosting in a private cloud.
And that that vendor, which was a Dell Virtustream, decided to get out of the hosting business so that it kind of forced us to move to the public cloud a little sooner than we we had originally planned. We knew we were going to get there one way or the other.
But we had some other things going on, but it just upped our timeline. So and the Bill Laker, who was the previous CIO, Charles Wagner is the current CIO head. That was one of the gifts he gave us.
Before he retired was he signed the the hosting, the first private cloud hosting agreement and they took about 18 months to do that. I joined the company in April of 2019 and in May of what, 30 days later we.

Jordan Cooper   2:25
OK.
Yeah.

Chuck Christian   2:34
Turned it on in the Virtustream data centers. And so we were there for five years and then we spent another 18 months moving it, building it inside of Azure and moving it. But we had already moved some workloads to Azure prior to that.

Jordan Cooper   2:34
No.

Chuck Christian   2:52
So it's it's been an interesting journey to say the least. But the one thing that I did back in 23 was I created a position called Cloud Cost Analyst.

Jordan Cooper   3:06
Hmm.

Chuck Christian   3:06
Is because I'd heard too many stories about people who really were not watching their spend in the cloud and just thinking it was gonna take care of itself. And they were getting hundreds of thousands of dollars worth of surprise billing. And so I decided that I would.
From the very beginning, start watching that very carefully and we had an analyst that was on my team that reported up to one of my directors. He was very detail oriented.

Jordan Cooper   3:30
Yeah.

Chuck Christian   3:38
I've been told not to use the word ****, but I and Eric knows I talk about him a lot like that. He is outstanding. He's learned more about Fin OPS than he ever thought he would. And every every Wednesday morning at 7:30 like clock.

Jordan Cooper   3:48
Mhm.

Chuck Christian   3:58
Work. I get a report in my e-mail of what our spend is for the month, how we compared to this same period last year, what's trending, if it's anything that's worrying him about the spend that it's going up higher than he thought because he's plugged into as that infrastructure changes so.

Jordan Cooper   4:18
Mhm.

Chuck Christian   4:18
And we have a managed services company that manages all of that for us in Azure, but my team monitors all of that and we manage from the application out, they manage everything from the application.
Below.

Jordan Cooper   4:34
Well, Chuck, I have so many questions based on what you just said. I think the most pertinent one to our listeners would be, has there been any point when your cloud cost analyst since he's been hired has actually alerted you to some?
Elevated spending that's been of concern to him and if so, what was your intervention at the time? I think that would be of great interest to our listeners.

Chuck Christian   5:03
Well, as you can well imagine, there's a lot of things you can do inside of Azure very quickly. You can spin up environments, you can increase storage. And so our managed services partner was doing some moves of storage from one storage tier to another.
It took them longer than they had anticipated and so we our storage costs jumped up $20,000 in one month and so that that's a red flag that's Erie's going to let us know. So we immediately reached out to the managed services partner and asked him what are you doing?

Jordan Cooper   5:25
OK.

Chuck Christian   5:36
And so then they mitigated it very quickly. And so and then we've had some other ones, Epic and requires a great deal of storage and they use BLOB storage and that kind of stuff. And Microsoft didn't used to charge for BLOB storage.

Jordan Cooper   5:37
Hello.

Chuck Christian   5:53
They had it in preview mode. I don't know exactly what that means, but it means you get to use it. You don't have to get charged for it. But then all of a sudden, you know, we got noticed that next month you're gonna get charged for it. And what we saw was that BLOB storage, the bill for that was over $200,000 in one month. And we look, we had a lot of conversations at Microsoft.
And said that cannot possibly be correct. And so sure enough, they went back and said, yeah, you're right, we the billing is entirely wrong. We've shut it off. We're, you know, we're going to give you a credit for all of that. And you have our deepest apologies and we won't do it again.
Well, it's Microsoft. They're going to make mistakes just like all of us. But as long as you watch it, they've been very, very good about if we can show them the era that that we're seeing, they're very good about applying a credit back to us and removing that expense at a later date.

Jordan Cooper   6:52
Sounds like if a peer organization without a cloud cost analyst proactively monitoring your spend were to have elevated costs, it might be difficult to retroactively identify where there were inaccurate charges. So I suppose.
Having your analyst was able to proactively avoid what could have compounded into significant additional erroneous charges. Is that right?

Chuck Christian   7:21
Right. Well, it's kind of looking at your credit card statement and going, damn, I didn't buy that. And so, but then you have to, you know, go through the trouble of getting it taken off and disputing it and that kind of stuff. It's better to do it as you see it. So you're absolutely correct. And so that those are the things that Eric does and.

Jordan Cooper   7:24
Mhm.
Mhm.

Chuck Christian   7:40
I will tell any of the viewers that are out there, if they want the job description for my cloud cost analyst, I'll be happy to share it. I have shared it out on probably two dozen times already. Go ahead.

Jordan Cooper   7:42
I.
Now um.
I do have a question that Eric probably would not like me to ask, but nonetheless, Gen. A I is rapidly making headway into every organization in America, especially large health systems.

Chuck Christian   7:58
Yeah.
Mhm.

Jordan Cooper   8:08
Is there any planned use for Gen. AI in your future road map to do some of the to automate some of the work that Eric is manually doing now and running these reports?

Chuck Christian   8:18
Well, I mean, we're we're already doing some of this stuff. Microsoft has a lot of tools, but we're not going to depend upon that. It's kind of like the Fox watching a hen house. I mean, we have a tendency to take Ronald Reagan's approach of trust but verify. And I'm not sure that yet an A I could do what Eric does.

Jordan Cooper   8:25
Review.
Mhm.

Chuck Christian   8:37
If I had my guess, if we have some new tools that come in, it'll augment his ability to catch things earlier on down the road. I mean, I've had some conversation with some folks that they have an AI tool that will look around the utilization of the hardware.
If they're seeing that it's being over provisioned, I mean, you've got more than you really need based upon what's being used. It'll resize it automatically. And I'm going, well, I'm not sure if I'm ready for the Tesla.

Jordan Cooper   9:04
Mm-hmm.

Chuck Christian   9:12
Opportunity inside my Epic environment to tell you the truth because you could resize it based upon you know a limited data set and be and make a terrible mistake that you have to undo and that would cause a performance issue for your clinicians.

Jordan Cooper   9:21
OK.
So I'd like to pivot for a moment, Chuck. Actually, there's a number of topics that we haven't yet addressed that you alluded to earlier, and I'd like to dive into those. So you did mention that Dell Virtue Stream didn't want to be in the data center hosting business anymore, and that originally is what prompted your move to Microsoft.

Chuck Christian   9:34
Yep.
OK.
Run.

Jordan Cooper   9:51
Microsoft Azure and you said that you initially Franciscan Health had some hesitancy about going into the public cloud as quickly as you ended up doing. My question is how were you able to overcome that hesitancy about going to the public cloud from the private hosted solution and why did you end up deciding to go in that direction with?
In this case, Microsoft Azure instead of seeking to move from Dell versus Stream to another competitor's fully hosted solution.

Chuck Christian   10:20
Well, has a lot to unpack in that that question. So I'll, I'll, I'll do my best. And so Dale made some business decisions about the time that we the the cost of running Epic in a private cloud or the public cloud.
Was not inexpensive. And so our purchasing policies required us to go out to RFP before we just automatically renewed an agreement. So we did that. We sent it out to a lot of different consulting firms, managed services firms. We sent it out to AWS, Amazon, Azure.
We talked to Rackspace and of course Epic, it's their self as well as Virtustream because we didn't know at that point in time that Dell was going to Virtustream was a separate business unit inside of Dell. It was autonomous.
But then they made the business decision to move it into inside of Dell Technologies and the leadership of Dell Technologies decided they did not want to be in the data center business anymore. And so they they let their customers know and we were given an off ramp to move and so.
We knew that that gave us that limited our choices. We could either go to the public cloud, we could, you know, turn to Epic for their their hosting or we could, you know, go to Rackspace or AWS and.
Or Microsoft. And so we did not have any workloads in Amazon AWS at that point in time. We'd already started and had built our networking in two different data centers. We were mostly everything that was in production was in North Central, which is in Chicago.

Jordan Cooper   12:03
OK.

Chuck Christian   12:11
And we had done some work out in Central, which is in Iowa related to virtual desktops and those type of things. And so it wasn't an easy decision to make. Microsoft came to the table with some good partnerships.
Willing to make some investment in our adventure. There had not been a health system our size that had actually moved to Azure at that point in time. And so we were kind of breaking ground on this and that was some of the hesitancy.

Jordan Cooper   12:44
OK.

Chuck Christian   12:48
You know, Franciscan is not considered a organization that is pioneers and many things. Sometimes we're fast followers. But you know what? You know, you know what pioneers get. They get the arrows from the from the Indians. And we don't really want to do that because there's a risk associated with.

Jordan Cooper   12:56
Mhm.

Chuck Christian   13:08
But we felt pretty confident with the partners that we had and the managed services group that had already moved some workloads of size and had had that experience. And so we decided to not really take a chance.
But it was a calculated risk of moving out there. We and that the full backing of Microsoft, they weren't going to let us fail. Epic was not going to let us fail. Our managed services partner that were doing the bill was not going to fail because they were.

Jordan Cooper   13:39
Mhm.
Yeah.

Chuck Christian   13:45
Reputationally, it was there was too much at risk for them. We had, they had to have us succeed and we didn't want to fail either. And so, you know, and I will tell you that, you know, I had a meeting with Microsoft on Wednesday.
With a lot of people that were involved in this process and we are all in agreement that it was a grand learning opportunity for all everybody that was involved. So it's it turned out really well.

Jordan Cooper   14:15
So you mentioned your implementation partner a few times. You know, I know we've spoken about first. I'd like to secondly address architecture in light of, but but first let's just talk about financial and general risk tolerance. You said you need to pick a good implementation.

Chuck Christian   14:27
Yep.

Jordan Cooper   14:34
Partner and a good integration engine that can migrate data to the cloud environment. How did you go?

Chuck Christian   14:38
Not an integration engine, an integration partner. It's a it's humans. It's not done by machines.

Jordan Cooper   14:41
In an integration part, so how?
So, well, how do you pick the right implementation partner? And also how do you pick the right kind of well, also the integration platform in which you're building those interfaces? How do you pick both of those?

Chuck Christian   14:59
Well, so here's The thing is Epic to Epic is not that big a deal. Epic does a really, really good job. They have what they call their cloud configuration guide. They'll tell you exactly how that environment needs to be stood up.

Jordan Cooper   15:04
Mm-hmm.
Mm-hmm.

Chuck Christian   15:14
And built down to, you know, they'll give you a, you know, Chinese menu of servers and things to put it on. So they have a good fundamental underlying architecture of.

Jordan Cooper   15:27
Hello.

Chuck Christian   15:30
You put it on this, we know it's gonna run because they've got all the monitoring tools, internal and external for that. So weren't worried about the architecture because Epic had that nailed down. Do they have everything perfect? No, but it's a partnership that.

Jordan Cooper   15:32
Mhm.

Chuck Christian   15:50
But I think Epic got right from the very beginning is that if you make your customers successful, they're gonna stay with you to do that. And that's exactly what they do is they work really hard to make sure that we're successful and we're referenceable to anybody that wants.

Jordan Cooper   15:52
Mhm.

Chuck Christian   16:10
To talk to us. And so the thing about it is that it's a true migration. We were running our system in Virtustream's data center in Sterling, VA. That's where our production environment was. And we migrated that data to Chicago. But during the process, Microsoft came back and said.

Jordan Cooper   16:14
OK.
3.

Chuck Christian   16:30
You know, we've learned something about our environment. In Chicago, we have one data center, so we have availability zones. That mean you can spread compute over multiple racks of servers, but you can't spread it over multiple data centers.
And so they recommended that we we not go live in Chicago, but we moved to San Antonio, which is South Central. They have multiple data centers down there and we have spread our compute over two of their three data centers.
Mostly on the presentation layer because we can't split our database, it's not splittable. So but everything's replicating back up to Chicago where our Dr. is. And so you need people that are certified with Epic technology.
And know how to migrate that data. I have a technical engineer on my team who's absolutely outstanding. She has been with Franciscan from the first implementation of Epic in 2011. She was a technical analyst before.

Jordan Cooper   17:30
Mm-hmm.

Chuck Christian   17:40
For that and she's extremely detail oriented and she manages that environment from the application and the architecture side and is the true interface with our managed services partner.

Jordan Cooper   17:41
Mhm.
Quick, quick question there. When you're talking about Sterling, VA and Texas and Chicago and you're in Indiana, are there any concerns about lag due to geographic distance?

Chuck Christian   18:02
Yep.
Yeah.
No.
Well, no. In my network architects, we always have a conversation about the speed of light and how fast it takes to propagate. There's we're almost at the same network latency.

Jordan Cooper   18:15
Mhm.

Chuck Christian   18:26
Between our data centers and our hospitals as we are from our colos because we designed it that way. We have 10 gig circuits that our topology of our network is we have one of our colos is in Equinix in Chicago. The other colo is here in Indiana.

Jordan Cooper   18:44
Mhm.

Chuck Christian   18:46
Indianapolis at Henry St. And so everything connects back to the colos. It doesn't connect through my data centers connect to the colos. And so we're not everything's not home running back through one of the data centers. So we have a data center in Indianapolis and a data center in Lafayette IN.

Jordan Cooper   18:52
Mhm.
I'd like to wrap up this podcast episode in the next few minutes, but I would be remiss to do so without first asking about, and you did allude to this, your disaster recovery strategy. You mentioned it's the same size as production. Also, you mentioned an interesting story. I think our listeners.

Chuck Christian   19:15
Yeah.
Yep.

Jordan Cooper   19:22
Would be intrigued to hear about Black Friday. So can you give us some insight into your Dr. and HA disaster recovery and high availability strategy, how you chose to make sure that it was the same size as prod instead of just making it smaller, cheaper, but scaling to scale up later and how that's relevant to?

Chuck Christian   19:27
Yeah.
Yep.

Jordan Cooper   19:43
Thanksgiving.

Chuck Christian   19:43
Well, everything is relevant. You know, The thing about it is, is that when you're using public cloud, everybody's using the public cloud. And what we found is it had nothing to do with my epic environment, but it had everything to do with the data lake house that we were building at that time.
We needed to add to that environment and we this was the week the Monday before Thanksgiving and so we my server team went to provision additional hardware in our environment in our subscription and those servers were not available.

Jordan Cooper   20:14
Mhm.
Google.
OK.

Chuck Christian   20:37
We'll do the, you know, kind of the pilot light where you'll provision all that at one time. But if you can't get the server you want, you have to look for the next server and the next one, the next one. You can do it with, you know, with orchestration and automation and make it less painful. But what we did is.
We have the same number of Citrix servers or VMS in both of our data centers. And then in October of last year, we did something we've never been able to do before. We actually flipped from South Central, which is in San Antonio, to Chicago.
In 45 minutes we we were operational in Chicago running and we ran there for 30 days. And so in November we flipped back and we got, we flipped back in 42 minutes. So we now know that it's it's a viable environment.
And it's all we have to do is change a lot of DNS entries that we're working on automating and then ensure all the integrations to all the things that are still running in our data center, which is some third party applications are connected properly.
The the the moral of this story is if you if you want to make sure your doctor works, you really really have to test it and not test it, but running it.

Jordan Cooper   22:03
So Chuck, we are are at the end of this podcast episode. We've really gotten in the weeds here of Franciscan Health's cloud strategy over the last few years. And I'd like to ask one final question if you were to speak to a peer, another VP of tech and CTO at another.

Chuck Christian   22:17
Sure.

Jordan Cooper   22:22
Health system in the United States listening this episode right now. And what would you, what lessons learned would you share about cloud strategy or what advice would you give? What would you want to tell that person as they consider how to navigate these obstacles as they determine their own cloud strategy?

Chuck Christian   22:38
Sure. When I joined Franciscan, I was told that Franciscan had a cloud first strategy and that meant they were going to put everything in the cloud. And so I looked at my team and I said, so we do know, we know that everything can run in the cloud. They said, well, no, I said. So then you don't have a cloud first strategy, you have a workload placement strategy because I've been in healthcare long enough to.
There are certain applications that are absolutely not tolerant at all for network latency, and there are things that need to be on Prem that if everything else is gone, it needs to work and that's like infant safety for the nurseries.
You know and and you know cardiac monitoring, you need to have edge servers for packs and all those type of things. And so you need really have to understand your environment and go through the app rationalization about what can and cannot live in the cloud and come up with an appropriate plan.
That you can mitigate those those risk in the event that you're gonna you're gonna have a ADR event and you know it's in like cyber cyber events. It's not a matter of if it's a matter of when and so you just you have to work every day to make sure that.
You're going to have an environment that's going to work. We strive to have ours back in full operation within 7072 hours. Are we there yet? No. But if our designs work the way we think it will and we find the appropriate partners to help us build it, it will.

Jordan Cooper   23:51
OK.
I really appreciate our conversation today, Chuck, especially because I think when you are in your position as CTO of a health system, you do have to think about things that normally you wouldn't think are particularly relevant, like a Cyber Monday sale affecting retail. How does that affect cardiac monitoring in the?

Chuck Christian   24:22
Yep.

Jordan Cooper   24:26
NICU, but in fact there are potential implications if you take processing from servers on Prem and move them into commercial cloud and there's no compute, no servers available because they're all being used for other applications in the commercial cloud. So I think the cross area, the many areas we covered today.

Chuck Christian   24:42
Yep.

Jordan Cooper   24:46
Think are are a lot of of great interest to our listeners. I appreciate you joining us. For those who've been following, this has been Chuck Christian, the VP of Technology and CTO of Franciscan Health. Chuck, thank you so much for joining us today.

Jordan Cooper stopped transcription