Experian Achieves 100% Uptime with Amazon DynamoDB and Amazon Aurora

Inspired by a change in vision in 2014, global information services company Experian had big plans for its business. “We didn’t want to be known as just the credit bureau,” says Jeff Scanlon, senior vice president of platform operations for Experian consumer services business. “We wanted to be known as a place where consumers can go to really understand and take control of their overall credit, identity, and financial portfolio.” Poised to reinvent the way it presented credit and financial data to consumers, Experian realized its current on-premises database architecture didn’t provide the kind of functionality the company needed to grow.

Experian made the decision to move away from the monolithic application architecture of its early days, instead using a cloud-first approach by turning to a microservices-driven architecture that would offer scalability, flexibility, and security. To achieve that, Experian migrated off of Microsoft SQL Server to Amazon DynamoDB, a database solution from Amazon Web Services (AWS). Later, when Experian acquired CSIdentity (CSID) to further enhance its product offerings, it again enlisted AWS to facilitate a modernization effort that would replace its legacy relational database with Amazon Aurora to reduce administrative tasks and to improve application performance for end users.

As Experian considered how it wanted to modernize its architecture, it realized that its initial on-premises database platform didn’t have the capability to support the company’s fundamental shift in thinking. Migrating from the legacy system to the cloud, however, would allow Experian to improve performance, efficiency, and scalability by refactoring its monolithic applications into microservices.

Experian wanted a cloud-native database that supported a microservices architecture in which each service is supported by a dedicated database instance. This has multiple advantages over a monolithic structure, including increased security capacities with role-based access controls, table segmentation by microservice, and integrated AWS Key Management Service (AWS KMS) encryption—a big draw for a company that handles sensitive financial data. Amazon DynamoDB—a fully managed, multiregion, multimaster, durable database with built-in security, backup and restore, and in-memory caching for internet-scale applications—is particularly well suited for this type of architecture. Experian was able to build flexible and reusable microservices to segment data using Amazon DynamoDB as a serverless data store for consistent and fast performance. “We felt that the Amazon DynamoDB platform was going to give us the most credibility and flexibility in this new platform,” says Scanlon.

The Experian development team got to work building the core of its new customer services platform. This would become the main platform used for all of Experian’s consumer-facing product lines; consumers would interact with it to access their credit score, credit report, educational products, identity protection, credit match marketplace, and other product lines. Experian was building something new while migrating and ensuring that product availability and capabilities remained the same, if not better, than they were with the legacy system. By migrating from Microsoft SQL Server to Amazon DynamoDB, a serverless service, Experian would be able to scale up without worrying about managing physical database servers, even with the new platform workload. “Scalability was another big, big benefit for us—the fact that we’re scaling up to multiterabyte platforms without having significant overhead or latency issues,” says Scanlon. Administratively, DynamoDB allows for an automated self-service deployment, including features of high availability, backup, scalability, and encryption—all of which previously had to be purchased and integrated individually in the data center. Since Experian launched its consumer services platform on Amazon DynamoDB, the company has handled 50–75 percent volume growth in its data layers each year—and with a more flexible cost model, buying on demand instead of making large capital investments in hardware, software, storage, and networking.

The fluid structure of Amazon DynamoDB made way for unprecedented teamwork between application developers and data engineers. The new model helped them use common resources to build data layers and overall deploy the system more efficiently. “Something that might’ve taken 60, 90 days in the past to get servers configured and deployed we could now do in a matter of hours, enabling our cloud and data engineers to focus on product innovation,” says Scanlon.

The velocity of the product release afforded by Amazon DynamoDB brought further benefits to consumers. Experian Boost, for example, helps consumers raise their credit scores by using nontraditional data. And Experian CreditLock, which lets consumers lock and unlock their credit report in real time, is just one of many tools Experian has expedited to offer customers fraud protection. “The performance of our websites and consumer experiences has increased dramatically with Amazon DynamoDB because of our ability to scale quickly on demand,” says Scanlon. “And the information we can provide is much more detailed.” 

Experian again streamlined its architecture with AWS when it acquired CSID in 2016. The acquisition enabled the company to expand its identity protection and credit-monitoring solutions to consumers through enterprise partners around the world. But CSID was limited by its on-premises data center in a way that reminded Experian of its former database problems. It turned to AWS to bring the acquisition into the cloud.

The Experian team started the migration by picking up CSID’s existing platform database engines that were running on MySQL in an on-premises environment and running them as a MySQL instance on Amazon Elastic Compute Cloud (Amazon EC2)—a web service that provides secure, resizable compute capacity in the cloud. 

Then the entire system was moved to Amazon Aurora, Amazon’s MySQL-compatible relational database built for the cloud; the conversion required very little rearchitecting because of Aurora’s MySQL compatibility. This allowed for a modernization of CSID’s technology stack with minimal changes, keeping important features such as dark web monitoring while instantly making the applications more scalable, reliable, and flexible. It also provided a more advanced level of performance and was better suited for enterprise workloads than an open source database management system in terms of reduced database calls and zero downtime. “By moving to Aurora, we got the benefits of a managed service provider without the headaches,” says Scanlon. “The environment in Aurora is so consistent and reliable.”

Since relocating to Aurora, the CSID team has indeed seen an increase in performance. Compared to the self-managed MySQL systems, response times have improved 10 percent, replication lag has been cut in half, and backup runtime has been reduced by 95 percent. Managing the operating system layer used to require regular patching and updates, during which workloads had to be moved between primary and secondary servers, impacting uptime. With Aurora, AWS handles patching and security updates, all with no system downtime.

Uninhibited by the previous on-premises system, which had forced the company to maintain a narrower focus mostly on credit, CSID now uses the fluidity of backend functions with Aurora to offer customers daily refreshes of credit reports as well as additional noncredit data and dark web data that give end users a much more well-rounded view of their financial identity. 

Due in part to the migration from the on-premises data center into Amazon DynamoDB and the CSID migration to Aurora, Experian was able to boost performance, increase scalability, and ultimately provide more features to its end users. The experience has left Experian eager to get more value from its data by exploring other opportunities with AWS, including machine learning at scale with Amazon SageMaker and more robust analytics with Amazon Simple Storage Service (Amazon S3) data lake capabilities.

“AWS has done an amazing job supporting us and helping us through the process,” says Scanlon. “I feel like AWS is truly my definition of a strategic cloud service provider. They’re not transactional. They’re not here just for the order of the day, but rather they’re focused on how they make sure our experience is successful and that we continue to operate the best way we can.”