Upstox Saves $1 Million Annually Using Amazon S3 Storage Lens

As Upstox’s business grew, so did the amount of data it stored in Amazon S3. Upstox needed to find a way to optimize data storage as part of an initiative to reduce cloud infrastructure costs across the company. It began to improve data storage efficiency by following Amazon S3 best practices and lowering costs despite increases in its storage usage. Upstox started using AWS in 2016 for its scalability and elasticity. It was fast, easy, and more cost effective for Upstox to migrate existing applications to AWS compared to maintaining an on-premises data center.

In June 2021, the company used Amazon S3 to build its secure and scalable data lake solution—Upstox Data Platform, a solution to facilitate comprehensive, custom, and secure data delivery to business users. “We chose Amazon S3 as the foundation of our data lake architecture because of its meaningful storage innovation, performance, scalability, availability, security, and affordability. We could break down data silos, unlock the value of our data, and increase innovation across our organization,” says Indranil Chandra, principal machine learning and data engineer at Upstox. This migration away from its on-premises data lake helped to reduce time to insight from days to hours.

Upstox focused on ingesting data from many data sources, including relational database management systems and APIs, into its data lake to create a workflow for queries to use the Upstox Data Platform. In early 2022, Upstox ingested a few petabytes of data into Amazon S3. Data is ingested into Amazon S3 buckets, then queried for various use cases, like getting insights on adoption rates of new features and uptime trends of services using Amazon Athena, which analyzes petabyte-scale data where it lives with ease and flexibility. Upstox uses Amazon Athena as an in-house data-querying tool across the entire firm to answer business queries and give visibility into daily business and product-feature performance.

The significant rise in the volume of stored data in Amazon S3 also meant a rise in Amazon S3 storage costs. To lower the Amazon S3 storage costs and optimize its economic and operational advantages, Upstox started a three-phase plan. The first phase involved optimizing the data in its Amazon S3 buckets where Amazon Athena query results are stored. Starting in May 2022, the company implemented a retention policy for Amazon Athena data queries to warrant query results for a finite time. In just 2 days, it had reduced daily Amazon S3 storage costs by 62 percent. By using Amazon S3 best practices, Upstox built a highly available and durable data lake architecture.

The second phase of optimization focused on using Amazon S3 Storage Lens to get organization-wide visibility into object storage usage and activity trends. Using the Amazon S3 Storage Lens advanced metrics, Upstox had access to 35 additional metrics and 15 months of historical data. Upstox gained extensive insights into which buckets and prefixes were growing and at what rate, the health of data operations, and how to identify cost savings and performance improvements by using the most optimal Amazon S3 storage class. Upstox used Amazon S3 Storage Lens advanced metrics to identify existing buckets with insufficient lifecycle rules in place. It then set up lifecycle rules to migrate infrequently accessed data to Amazon S3 Glacier Instant Retrieval, which provides low-cost archive storage with milliseconds retrieval for rarely accessed data. Upstox uses Amazon S3 Glacier Instant Retrieval to store previous snapshots of its data lake for debugging, compliance, regulatory, and audit activities. “Amazon S3 Storage Lens advanced metrics are extremely important in our daily operations because we use them to drill down into our data costs so that we can make quick, well-informed, and impactful business decisions,” says Chandra. “We can now understand and visualize our storage usage and analyze it to detect outliers and anomalies, which consistently helps us optimize our storage at scale.” Within 2 months, Upstox had reduced its daily cost for Amazon S3 by 93 percent.

The third phase of Upstox’s optimization involved upgrading its data pipelines to ingest data from high-throughput source databases, like NoSQL databases, into the data lake in near real time. Within 5 months, Upstox achieved a 96 percent cost reduction. “The path forward is to continue using Amazon S3 Storage Lens to maintain a closer pulse on our storage usage and further improve upon cost efficiencies,” says Chandra. “With the Amazon S3 Storage Lens interactive dashboard, we can easily locate the Amazon S3 prefix hot spots where increases in cost happen and optimize them with the right retention policies and Amazon S3 storage class to further improve cost efficiencies.”