Goldman Sachs Streamlines Third-Party Data Consumption Using AWS Data Exchange

To meet Goldman Sachs’s need for more efficient and scalable technology—including the ever-growing demand to find, procure, process, and use data from third-party parties—the company's leadership has mandated a migration to the cloud. Goldman Sachs’s Market Data division is executing on that vision and is supported by AWS Data Exchange to streamline its third-party data consumption. This migration will enable Goldman Sachs teams across the company to build applications and analyze data more efficiently, and also help clients using products, such as Goldman Sachs Financial Cloud for Data. Goldman Sachs Financial Cloud for Data is a cloud-native financial data management and analytics solution built on AWS, which enables clients to seamlessly and securely integrate their proprietary data with Goldman Sachs curated financial markets data and third-party data products, including data consumed from AWS Data Exchange.

Goldman Sachs provides expert insight for investors who are looking to make the right choices in fast-changing markets. But getting the third-party data it needs for analysis—and ingesting it, parsing it, evaluating it, and delivering results—was taking too long and using too many resources. Staying a leader meant changing how the company operates. 

New York-headquartered Goldman Sachs traces its roots back to 1869. Over the last 150 years, it has grown into a global investment banking, securities, and asset management organization that provides a wide range of financial services to clients including corporations, financial institutions, governments, and individuals. All of those clients depend on timely, expert analysis and advice to remain competitive, and Goldman Sachs depends on 400+ vendors supplying terabytes of data to perform that analysis in support of its clients.  

And just like its clients, Goldman Sachs needs the data to be delivered in a timely manner, and in a way that can efficiently handle the size and complexity of its ecosystem. That is why Goldman Sachs leadership mandated that the company migrate to the cloud. “The demand for market and financial data is increasing and we’re seeing growth in new use cases, new vendors, greater coverage, and deeper history,” says John Chappell, Managing Director at Goldman Sachs. “But we can’t continue to effectively support our clients just by continuing as is and adding more people to find, ingest, and process this growing list of data. We need to improve our scalability and efficiency, and that’s where the cloud comes in.” 

With the volume and variety of data increasing, the process of getting raw data into analytics and machine learning tools was growing more complex. Maintaining massive numbers of data sources and feeds, importing, processing, and preparing the data, and setting up and processing payments was adding up to a lot of costly, duplicated, and undifferentiated work. By migrating third-party data consumption to AWS Data Exchange, Goldman Sachs eliminated these challenges. The company can find and subscribe to a diverse range of data in one place, compliantly license third-party data at scale, manage entitlements across the organization, and simplify access to data by consuming it directly in the cloud where it gets analyzed. And that strategic migration was recognized by the company’s senior leadership.  

Marco Argenti, Co-Chief Information Officer at Goldman Sachs, says “AWS Data Exchange is a key component of Goldman Sachs’s financial cloud strategy because it reduces friction for sourcing financial data from new and existing third-party providers, and allows us to focus on delivering our core services and differentiated data analytics to better serve our clients.”

Goldman Sachs has seen demand for financial market data increasing exponentially, typically doubling year-on-year. And with so many employees trying to keep up with demand, Chappell explains that there is often “a huge amount of operational toil” with acquiring vendor data. “We have over 10,000 engineers responsible for more than 1.5 billion lines of code,” says Chappell. “That’s more than a quarter of the firm’s staff. Core data engineering, which is led by our Chief Data Officer, Neema Raphael, is a team of over 500 engineers and we’re building data platforms, services, curating data models, and providing a data governance framework for all the divisions in the firm.”

Developers and data analysts expect modern tooling to build applications and analyze data in a cloud-native environment, and Goldman Sachs is adapting to support that across the organization. Data is largely generated in the cloud, so continuing to ingest it using non-cloud native mechanisms only to re-upload it back to the cloud for analysis are extra, unnecessary steps. 

Because AWS Data Exchange is a cloud-native managed service that allows these processes to be centralized, streamlined, and automated wherever possible, migrating existing market data feeds improves the overall efficiency of consuming third-party data. Goldman Sachs estimates that streamlining workflows through AWS Data Exchange saves the company hundreds of engineering hours each year—hours that are better spent building new features for clients.

“We’ve built an automated data pipeline centered on AWS Data Exchange that allows us to ingest and process third-party data much more efficiently,” says Chappell. “This means that even as we add additional data feeds, whether for our internal processes or our customers, we have cut down on the time our teams spend on ETL and other data preparation.” 

Removing the need to Extract, Transform, and Load (ETL) third-party data, and streamlining ingestion through AWS Data Exchange, has also helped Goldman Sachs expand the company’s software offerings.

To help clients eliminate the need to develop and maintain foundational data-integration technology and lower the barriers to entry for accessing advanced quantitative analytics across global markets, Goldman Sachs and AWS have collaborated to launch Goldman Sachs Financial Cloud for Data. This suite of cloud-based data and analytics solutions for financial institutions helps clients discover, organize, and analyze data in the cloud to quickly gain insights and make better-informed investment decisions. Through this collaboration, institutional clients benefit from decades of Goldman Sachs experience to address data management and analytics challenges in financial markets. 

And just as Goldman Sachs and its clients look to the cloud to modernize their infrastructure, its key vendors are doing the same. FactSet, a leading provider of financial data and an AWS Partner, has been providing services to the sell side, corporates, asset managers, and asset owners for more than 40 years.  

Since its founding—back before most companies had the computer power and access to collect and prepare data for analysis—FactSet has grown from using rigid data delivery methods to providing instant anytime, anywhere access to financial data and analytics that investors use to make crucial decisions. It has continually looked for better, faster ways to get customers the data they need and in a form that’s ready for use. And the next stage of that evolution is AWS Data Exchange for Amazon Redshift. 

AWS Data Exchange for Amazon Redshift lets users find and subscribe to third-party data in AWS Data Exchange, and then query that data in minutes in an Amazon Redshift data warehouse.

This means that firms like Goldman Sachs don’t have to create custom solutions to work with the data set they need. That quick access to ready-to-use data without ETL processes helps investors with their pursuit of ever-moving targets. “We’ve made all 40+ FactSet datasets available on AWS Data Exchange for Amazon Redshift,” said Peter Dorsey, Vice President, Cloud Strategy, Content and Technology Solutions at FactSet. “The integration means that our clients like Goldman Sachs can spend more time on the value-add analytics of the data and less on wrangling the data to get it ready for use. Whether they want to use one dataset, a handful, or all of them, they’re all there, ready for immediate access.”

Today both Goldman Sachs and FactSet are embracing cloud technology to stay ahead. “FactSet on AWS Data Exchange is a natural extension of our mission to provide a frictionless data experience to financial services across the discover, license, and use lifecycle,” says Dorsey. “Data catalogs accelerate the data discovery and vetting process. Once users have found the data they need, they can quickly license, access, ingest it, and analyze it, which gives them faster access and better control.”

Goldman Sachs agrees about the importance of a frictionless data experience to future-proof the company. “Data has powered the financial services industry for decades,” says Chappell. “Traditionally, quality of data, price, and usage rights were the key considerations for selection. Now, how that data is delivered to us is as important, if not more so. Whenever we have a new data request or we’re talking to a new vendor, we now look to understand if the data can be delivered through AWS Data Exchange.”