The Financial Industry Regulatory Authority (FINRA) is a not-for-profit organization authorized by the U.S. Congress to protect investors and ensure market integrity through effective and efficient regulation of broker-dealers. It writes and enforces rules governing the activities of more than 3,800 broker-dealers representing more than 600,000 brokers, examines firms for compliance, fosters market transparency, and educates investors.
Every day, FINRA oversees up to 75 billion market events—99 percent of equities trades and 65 percent of options trades in the United States—applying data analytics to uncover insider trading and other strategies used to gain an unfair advantage. FINRA recently moved a key part of its technology infrastructure to Amazon Web Services (AWS), adopting AWS Lambda serverless computing to make data validation more efficient.
FINRA’s Order Audit Trail System (OATS) is part of an integrated audit trail of order, quote, and trade events for all National Market System (NMS) stocks and over-the-counter (OTC) equity securities, and is used to monitor the trading practices of member firms. FINRA uses OATS data, along with other market data, to create the life cycle of each order—from receipt or origination through execution or cancellation—and monitor trading practices of member firms. Broker-dealers must submit daily electronic OATS data to FINRA, adding up to more than 50,000 files. As soon as data is received, FINRA validates it to ensure it is complete and correctly formatted according to a set of more than 200 rules. The system performs up to half a trillion validations each day. Processing demand varies significantly over time and can double or triple in response to market conditions that drive higher trading volumes.
To arrive at the right solution to host OATS, FINRA needed to perform an in-depth comparison of potential architectures. Three proofs of concept were built to test various options: Apache Ignite running on Amazon Elastic Compute Cloud (Amazon EC2); Apache Spark with Amazon Elastic MapReduce (Amazon EMR); and AWS Lambda. Data demonstrated that AWS Lambda was the right choice for the validation application based on criteria including scalability, data partitioning, monitoring, performance, cost, and maintenance requirements. “We have an overall organizational goal of moving to a real-time processing model,” says Tim Griesbach, senior director of technology at FINRA. “In addition to meeting our other criteria, AWS Lambda was a good fit for that long-term vision.”
Security was a critical component of the selection process. “We have very stringent data-protection requirements,” says Griesbach. “Our security team worked closely with AWS to review infrastructure, software, and services and found we could build our system in a way that complied with those requirements.” These standards include encryption of data in motion—which prevents FINRA from using plain HTTP connections to transfer information—as well as at rest using server-side managed key encryption.
FINRA developed the solution in only three months, including testing to ensure the system could handle peak loads. Data is ingested into Amazon Simple Storage Service (Amazon S3) via File Transfer Protocol (FTP). AWS Lambda functions perform the validations. FINRA repurposed part of the original validation architecture as a controller for AWS Lambda processes. The controller runs on Amazon EC2 and manages both data feeds coming into AWS Lambda and notifications going out of it.
The controller also manages external data sources. In addition to data submitted by broker-dealers, the validation system uses reference data, such as stock symbol reference files. The controller ensures the data queue continues running even when some tasks are paused pending the retrieval of external sources, and the company takes advantage of AWS Lambda data-caching abilities to reduce processing time whenever possible.
The system relies heavily on messaging to coordinate services, using Amazon Simple Queue Service (Amazon SQS) for input/output notifications. In addition, FINRA uses Amazon Virtual Private Cloud (Amazon VPC) to partition the system into separate test and production accounts to protect the live-validation process from errors.
FINRA architecture digram
“Using AWS Lambda, we’ve increased cost efficiency by a factor of two,” Griesbach reports. “We only pay for what we use, and we don’t have to manage on-premises server infrastructure.” The organization has achieved significantly faster processing times. Rohit Malhotra, technical project manager at FINRA, says, “With our previous system, validation results were returned to broker-dealers within three minutes on average, and up to seven minutes at peak times. With serverless processing based on AWS Lambda, we are returning results in less than one minute regardless of volume.” Additionally, the organization is able to provide more detail about data errors to broker-dealers.
FINRA plans to adopt AWS Lambda serverless computing for the entire solution by replacing the controller with microservices. It is also looking to move other applications to the cloud such as web applications, data processing, and analytics. Using AWS Lambda means FINRA can work more effectively with broker-dealers and more efficiently fulfill its mission of keeping markets transparent and fair for all.
Learn more about building serverless applications with services like AWS Lambda.