Pinterest Scales Daily Log Search and Analytics from 500 GB to 1.7 TB and Reduces Costs by 30% Using Amazon OpenSearch Service

The Pinterest observability team relies on Elasticsearch to monitor and issue alerts for new software deployments on the main Pinterest site. With Elasticsearch, the team can identify problems during software deployment, then quickly analyze logs to troubleshoot root causes. “We use metrics to detect all incidents when they happen, but log search is the main tool we use to find out what’s causing them,” says Wei Zhu, a staff engineer for Pinterest’s observability team.

Pinterest’s search for a cost-effective, scalable solution for log analysis began in 2016. Initially, Zhu’s team was self-managing open-source Elasticsearch software to monitor log data and troubleshoot issues with software deployment. But the costly operational overhead and the ever-increasing volume of daily data necessitated a change. Pinterest signed a 3-year contract with a third-party vendor, but that arrangement presented new challenges: the vendor’s high licensing cost drove up the overall cost of the solution, and its unique query language produced a steep learning curve for Pinterest’s software engineers.

In the second half of 2019, with the third-party vendor’s contract expiration approaching, Zhu’s team began migrating its data to Amazon OpenSearch Service. The move made sense: Pinterest was already using a variety of AWS services to scale its processing, storage, and data analysis workloads, and Zhu’s team determined that the move would reduce costs. Also, Amazon OpenSearch Service’s support for Kibana—a free and open user interface that lets users visualize Elasticsearch data—made it more readily accessible for Pinterest engineers. “The cost is much less because we don’t spend resources to manage the infrastructure,” says Zhu. “And Kibana is open source, which makes using the tool much, much easier—if we have questions about using it, we can find the answers online ourselves.” Overall, Amazon OpenSearch Service enables Pinterest to quickly find and resolve issues as part of its continuous integration and deployment of software, helping Pinterest ship new features to Pinterest users faster.

In just 1 year after the migration to Amazon OpenSearch Service, Pinterest’s observability team went from ingesting 500 GB of data per day to 1.7 TB per day. By the end of 2020, the team expects to be able to ingest over 3 TB of data per day. Currently, Pinterest is able to monitor and issue alerts for 20 new software deployments a day. “Using [Amazon OpenSearch Service], we can see that the scale has grown tremendously,” says Zhu. “And since we only have one person managing the pipeline, that shows immense resource efficiency.” With only one engineer overseeing its Elasticsearch deployment, the team can turn away from low-value work to focus on innovation and other business-critical tasks, such as exploring new use cases and adding more resources to the log search.

Security was also an important factor for Pinterest’s observability team as it was considering the migration to Amazon OpenSearch Service. “We’re really careful with our proprietary information and private user data, so we set up a whole pipeline with that in mind,” says Zhu. In addition, the team plans to adopt Amazon OpenSearch Service’s fine-grained access control, which enables different roles to access data at the index, document, and field levels. “We are looking to move more fine-grained access control for the search capability,” says Zhu. “Better control of personally identifiable information data would be a big win for us.”

Most recently, Pinterest started using UltraWarm for Amazon OpenSearch Service to save costs for Elasticsearch clusters. This new low-cost storage tier, which provides fast and interactive analytics on up to 3 PB of log data at a fraction of the cost of the current Amazon OpenSearch Service storage tier, has helped Pinterest reduce costs by 30 percent—savings Zhu expects to rise to 40–50 percent. UltraWarm is not only saving costs but also creating opportunities to pursue new use cases. “We’re onboarding a new use case that ingests about 2.2 TB of data per day,” says Zhu. “Without UltraWarm saving costs, I don’t think the use case would be possible. It’s eye opening to see how AWS puts customers first by innovating services to reduce costs for us.”

Zhu anticipates his team will adopt Amazon OpenSearch Service for analyzing all its Hadoop cluster logs by the end of 2020 in order to save costs on the more than 2.2 TB of data being ingested each day. “In the second half of 2020, we’re going to focus on investing more into log search because we think there’s a lot more value to discover here with Elasticsearch,” says Zhu.

Amazon OpenSearch Service helped Pinterest quickly scale its data-ingestion capabilities and reduce the burden of low-value work on engineers. As a result, team productivity increased. Now, Zhu and his team can quickly and efficiently monitor and issue alerts for new software deployments, helping Pinterest deliver quality features to hundreds of millions of daily users.