LCMC Health Enhances Clinical Resiliency with EHR Alternate Production running on AWS for Disaster Recovery.
Executive Summary
Sapphire Health, an AWS Partner, worked with LCMC Health to migrate their Epic[1] Alternate Production to Amazon Web Services (AWS). LCMC Health’s production EHR database is replicated in the organization’s secure healthcare landing zone on AWS, serving as a disaster recovery environment. If the primary EHR production system is degraded or offline, the alternate production environment can be activated in under 20 minutes, providing clinicians, administrators, and patients with continuous access to all data and services available through the electronic health record database, at the same or better performance level as the on-premises production systems. Sapphire Health leveraged infrastructure- as- code using Terraform and Ansible to script and build the entire environment in AWS. This method created long term operational benefits with shorter implementation times. Additionally, by leveraging a strategy of dynamic scaling for web service and presentation, LCMC Health projects significant savings with Alternate Production in AWS.
Migrating EHR Alternate Production to AWS to improve resiliency against old and new scenarios
LCMC Health is a nonprofit network of healthcare providers in Southern Louisiana. LCMC Health operates eight hospitals and employs more than 14,000 people, making it one of the largest healthcare providers in the New Orleans metropolitan area. LCMC Health is committed to providing all its clinicians and patients with uninterrupted access to critical patient information to provide the best care possible for everyone it serves. “Extended system downtime is a huge disruption to our mission of providing the best care for our patients,” says David Singer, Chief Information Officer at LCMC Health. “As the infrastructure becomes more complex and dependent on so many outside factors, we need highly adaptable solutions that can scale to meet our evolving needs. Taking advantage of cloud hyperscale gives us an extremely flexible solution that can address a multitude of disruption scenarios, that simply can’t be matched with only on-premises datacenters.”
To mitigate disruption risks, LCMC Health wanted to find a cost-effective solution that offered a reliable alternate production environment for their electronic health record system. “All our data centers including our disaster recovery data center are housed within our hospitals” Singer says. “Since all our hospitals are in the greater New Orleans area, which meant both our primary and secondary data centers are in the same geographic region.” LCMC Health had previously looked into remote colocation datacenter options, but they provided little added value except for being remote and proved to be very expensive for what it actually provides. “We need more than a remote datacenter. The supply chain constraint for technology during the pandemic made it very clear that we also need to be able to tap into hyperscale resources whenever needed.”
“Working in partnership with Sapphire Health on an EHR Cloud Read only solution on AWS two years ago gave us a high degree of confidence that the AWS platform would scale and perform at a high level for EHR Alternate Production as well.”
David Singer
Chief Information Officer, LCMC Health
LCMC Health turned to Sapphire Health, an AWS Partner that had already been providing managed services for the LCMC Health EHR infrastructure. “We have worked with Sapphire Health since 2016, and we trusted them to find the right technology,” says Singer. Sapphire Health helped LCMC Health deploy an AWS-based Cloud read-only solution to mitigate against ransomware attack in 2022. With an isolated read-only environment on AWS, LCMC Health knew it could gain additional business resiliency in the event of a ransomware attack where production, reporting, and disaster recovery were inaccessible. Now, by migrating Alternate Production (DR) to AWS, LCMC Health will build the foundation for a disaster recovery strategy that can withstand almost all regional disasters in an extremely cost-effective manner using similar AWS resources.
Sapphire Health designed the alternate production environment in AWS to take advantage of cost savings strategies using dynamic scaling for Web and Presentation tiers of the architecture; and ensured the operational database was built on the suggested performance tier to meet production requirements. “There’s been an incredible increase in cloud performance by the major cloud providers in the past few years” said Tyler Jordan, Director of Cloud Solutions at Sapphire Health. The growing performance capabilities of cloud providers like AWS means that LCMC Health will be able to meet future growth needs that normally accompany rapid growth or acquisitions for its Disaster Recovery strategy in AWS.
The industry-specific solution is architected to align with AWS best practices and conforms with global compliance frameworks such as the Health Insurance Portability and Accountability Act (HIPAA). “We have a secure Landing Zone for Healthcare on AWS which gave us the ability to turn on the appropriate auditing tools and data protection mechanisms to ensure we had a sound strategy to continue to leverage AWS securely,” says David Gates, information security officer at LCMC Health. “But what I think is most exciting development for LCMC Health with this project, is the introduction of cloud as a strategic digital transformation platform.” says David Gates, “it’s changing the way we think about technology scalability and availability with the continued use of declarative scripts to manage the Alternate Production environment build and changes. The downstream effect is enabling faster transformations in clinical applications and integration of data in a meaningful way, unhindered by traditional infrastructure constraints.” LCMC Health worked closely with Sapphire Health to prepare modular declarative scripts in Terraform for every aspect of the environment build. Combined with additional configuration automation through Ansible and Kuiper, Sapphire Health was able to demonstrate how the entire environment can be recreated over and over, in precisely the same manner with zero configuration drift.
Sapphire Health completed the migration of Alternate Production to AWS for LCMC Health in early spring of 2024, with two DR activations tests. In both DR activations, the activations were performed well under the 20-minute Recovery Time Objective set by LCMC Health. It also exceeded all performance metric expectations including user workflow exception rates. “Our previous DR activations to our on-premises secondary datacenter were very fast since Sapphire Health had the entire process scripted out.” Said David Gates, “So, we wanted to make sure we didn’t add significant downtime when activating DR in AWS. We were very pleased to see that we could perform the activation with less than 5 minutes of technical downtime.”
As part of the project, Sapphire Health implemented AWS services such as Amazon Elastic Compute Cloud (Amazon EC2) for on-demand compute, Amazon CloudWatch for monitoring the environment, and Amazon GuardDuty and AWS Security Hub to ensure environment security. The new Alternate Production environment combined with the Isolated Recovery Environment already configured at AWS will give LCMC Health a powerful resource to respond to a wide range of potential disruption scenarios from system failures to security threats.
Advantages of migrating Disaster Recovery from on-premises to AWS
The new Alternate Production environment in AWS represents another step in one of LCMC Health strategic objectives to ensure business continuity through almost all disruption scenarios. “Our goal is to get to near zero minutes of unplanned system unavailability. A commitment we are making for all our users.” Said David Singer, “That means we need to invest in the technologies and strategies that give us the most resiliency and recoverability.” David Singer continued by saying, “Legacy infrastructure lack the flexibility as well as the speed of change needed to address modern day disruption scenarios which often involve security or software related issues.”
LCMC Health Alternate Production environment in AWS is designed to do far more than just serve as Disaster Recovery if the primary data center fails. It is also designed to address sudden or temporary scale requirements even if the primary data center is still functional. With the presentation and web service layers load balanced across multiple availability zones, it can dynamically scale up at incredible speed, using available resources in multiple AZs. LCMC Health made this an important strategic objective for the project. “It was a lessen we learned the hard way during the pandemic when we faced severe supply chain constraints while trying to integrate in a new hospital we had just acquired.” Said David Gates. During the pandemic in 2021, facing severe delivery delays, LCMC Health had to resort to over-provisioning through its existing resources on both primary and secondary datacenter when server and storage orders failed to deliver on time for critical projects after waiting more than 9 months. “It was not a good situation as our users faced temporary performance issues and other system disruptions as we sacrificed years of resiliency design improvements to borrow more scale from existing equipment we had within our datacenters.” Said David Gates, “It is important that our long-term digital strategy with AWS ensures we are never in that position again.”
Sapphire Health completed the Alternate Production in AWS project in eight months which included 2 months of contracting and approvals with AWS and other vendors. The project timeline synced with the normal upgrade and maintenance cycles at LCMC Health. This ensured the project was well planned and introduced minimal additional planned downtime. “Electronic Health Record Alternate Production migration projects to the cloud usually take between 6 and 12 months but can go faster or slower depending on other operational factors.’ Said Tyler Jordan, “In LCMC health case, it could have gone much faster since much of the foundational work was already in place, however we wanted to match the operational calendar already in place to minimize disruption during the migration.”
The Alternate Production environment in AWS is forecasted to save LCMC Health about $3.5 million over 5 years as compared to the normal capital purchase and maintenance of on-premises equipment in its own secondary data center that supported the EHR DR environment. But additional potential savings in datacenter related costs and software subscriptions could increase the savings by an additional $1.6 million over the same timeframe.
LCMC Health plans to keep working with Sapphire Health to expand the use of AWS services. “As healthcare technology evolves, particularly in areas like artificial intelligence and machine learning, having a robust and flexible cloud infrastructure becomes increasingly critical,” Singer says. “Our AWS foundation doesn’t just solve today’s disaster recovery needs – it positions us to rapidly adopt emerging technologies that can enhance patient care. The infrastructure-as-code approach we’ve implemented means we can deploy new capabilities in hours rather than months, while maintaining the highest levels of security and compliance.”
David Gates adds, “What makes this implementation particularly powerful is how it combines operational agility with enterprise-grade security. Our infrastructure-as-code approach means every deployment follows security best practices by design, and our integration with AWS’s native security services gives us unprecedented visibility and control. This architectural foundation will be crucial as we explore AI-assisted clinical decision support, automated workflow optimization, and other innovative healthcare technologies that require both robust performance and ironclad security.”
Security is a key strategic concern at LCMC Health. In addition to the new Alternate Production environment, LCMC Health is also making improvements to its previously implemented Isolated Recovery Environment. LCMC was the first organization to create an Isolated Recovery Environment for their electronic health record operational database on AWS back in 2022. “When we first started working with Sapphire Health on the recovery environment in AWS, we were mainly concerned about a ransomware attack.” Said David Gates, “and AWS offered the perfect isolation architecture to maintain a mirrored copy of our Epic™ ODB database. But back then, we were limited to only a Read Only environment that could be presented to our clinicians.” In an effort to improve resiliency overall, LCMC Health is working to evolve their Isolated Recovery Environment to offer limited read/write capabilities that balance additional clinical value with immediate security concerns.
The combination of these strategies leveraging AWS for disaster recovery as well as Security Incident mitigation are all designed to dramatically improve system resiliency. “After all,” Singer concludes, “at the end of the day, what’s most important is that IT at LCMC Health can meet its mission of serving the people that serve our patients in the best way possible – whether that’s through ensuring system availability, enabling new capabilities, or protecting sensitive information. Our AWS DR implementation with Sapphire Health helps us deliver on all these fronts.”
About LCMC Health
Based in New Orleans, LCMC Health is a nonprofit network composed of eight hospitals with 1,826 inpatient beds, 10 emergency rooms, and 8 urgent care facilities across Southern Louisiana. The healthcare system focuses on providing the best possible care for every person and parish in Louisiana and employs over 14,000 people and 2,538 physicians and providers.
[1] Epic is a registered trademark of Epic Systems Corporation.