Alpha Vertex Case Study

Alpha Vertex was founded in 2016 to provide advanced analytical capabilities to investors, brokers, fund managers, and others in the financial community. The company uses artificial-intelligence tools to identify, ingest, and analyze traditionally studied factors—including exchange rates, interest rates, and fundamentals—and non-traditional factors such as media content, SEC filings, weather, and litigation information.

One of Alpha Vertex’s data products, PreCog, uses machine-learning models to forecast returns on both short- and long-term investments for about 20,000 of the most liquid equities worldwide. PreCog’s machine-learning models are regularly retrained and then deployed to deliver the product’s live-forecasting model, a mission-critical process that must conclude within 48 hours to avoid impinging on production resources. During the initial job launch, input retrieval places especially high stress on the system and cache performance is critical. Alpha Vertex needed a file-storage solution with strong scalability and multi-gigabyte throughput to hold data and interim models for a compute scenario that includes:

• The ingestion and analysis of millions of data points, in hundreds of categories, affecting the targeted equities.
• A Kubernetes cluster of almost 300 Amazon Elastic Compute Cloud (Amazon EC2) instances.
• 150 Kubernetes Pod launches per minute, with a steady state of 1,500–2,000.
  

Alpha Vertex tested Amazon Elastic File System (Amazon EFS) for data caching and interim model storage under full-load conditions for two weeks, and compared its performance to both a leading proprietary parallel network file system (pNFS) solution that is often deployed in high-performance computing (HPC) environments, and a traditional network file system (NFS). These were the results:

• Amazon EFS was 66 percent faster than the traditional NFS and 18 percent faster than the proprietary pNFS.
• Amazon EFS provides a common data source for workloads running on an elastic Kubernetes cluster federated across multiple public cloud providers.
• Amazon EFS scales up or down automatically as files are added or removed.
  

• Fast deployment: Alpha Vertex can stand up Amazon EFS for model retraining in less than 60 seconds.
• Massive scalability: From a cold start, Amazon EFS was able to expand to hold 45 TB within the 48-hour training window.
• Fast throughput: Even connected to 1,500–2,000 clients, I/O was clocked at 3–7 GB per second.
• More affordable: Amazon EFS not only outperformed the traditional NFS and the proprietary pNFS it was tested against, it also cost less than either one.
• Hiccup-free: Writes to Amazon EFS were as fast as saving on-premises, with zero timeouts and file corruptions. “Amazon Elastic File System can eat gigabytes per second without a hiccup,” says Michael Bishop, chief technology officer for Alpha Vertex.