The Orchard Builds High-Performing Data Synchronization Solution Using Amazon MSK

The Orchard is a leading distributor in the music industry, reaching digital and physical retailers worldwide. The Orchard offers a suite of services to support independent artists and music labels, providing the solutions needed to manage music distribution, marketing, and accounting. As the distributor grew, it wanted to improve analytics quality, time to insights, and the synchronization and sharing of data across its teams. To reach these goals, The Orchard needed a way to centralize its siloed data systems and to simplify sharing information across applications for employees and customers.

After engaging Amazon Web Services (AWS), The Orchard built a high-performing data synchronization solution using Amazon Managed Streaming for Apache Kafka (Amazon MSK), a fully managed Apache Kafka service that makes it easy to ingest and process logs and other forms of streaming data. Using this highly available solution, The Orchard provided access to near-real-time data for employees, improved accuracy of search queries, and reduced time to synchronize metadata from 10 hours to 30 minutes.

The Orchard collects, processes, and distributes music from thousands of labels to Spotify, Amazon Music, and over 200 other streaming providers. Using The Orchard’s suite of applications, artists and their managers can perform crucial activities, like tracking streams and collecting royalties. The Orchard lets teams build out the stack needed for their applications fully, from database to front end. However, its siloed infrastructure made it difficult to query and share data across multiple teams. “When one team needed data from another system or team, it had to build a custom script to query the database,” says Farouk Umar, lead manager of engineering at The Orchard. “Managing these scripts becomes very difficult when you need to pull large amounts of data.” It would take a long time to fulfill queries, and by the time the process was completed, the data would sometimes be inaccurate and outdated. 

Managing so many scripts became a time-consuming, cumbersome process for The Orchard, and this infrastructure was preventing the company from developing innovative new services for customers. Seeking a highly available, simple-to-implement solution, The Orchard turned to AWS. The company realized it could improve data querying and synchronization capabilities by migrating to Amazon OpenSearch Service —a fully managed service that makes it easy to perform interactive log analytics, near-real-time application monitoring, website searches, and more—and by using Amazon MSK to collect near-real-time events from different sources. “We ran a proof of concept on Amazon MSK and realized that it was incredibly easy to use,” says Umar. “This initial project gave us confidence that we could maintain this solution in production.”

The Orchard began implementing its data synchronization solution in December 2019 and completed the project in December 2020. The company is now able to share and exchange near-real-time data to all of its applications, which has accelerated its time to insights. Using serverless technologies, The Orchard can scale on demand to support a large number of users. The foundation of its architecture is AWS Fargate, a serverless compute engine that lets developers focus on building applications without managing servers. 

Using Amazon MSK, developers can quickly access relevant data for their projects without needing to write custom scripts, saving significant time for the company. Data is synchronized to MSK, helping employees access the most up-to-date information in near real time. “It would take 10 hours to synchronize our metadata from its main database to our analytics data warehouse,” says Umar. “We would have to paginate and download large amounts of data to our analytics system to pull the relevant information. But now, using Amazon MSK, this process takes 30 minutes.” With faster data querying speeds and less time spent on managing infrastructure or writing scripts, developers at The Orchard can focus their time on innovation. 

The Orchard has improved its development and experimentation speeds on AWS. The company now has access to a highly available environment where it can develop and deploy new applications and upgrades at a faster pace. As a result, employees can quickly devise solutions that generate value for The Orchard and its customers. For example, a developer used a Kafka Connector, an open-source framework to run generic copy workloads, to load data from spreadsheets into Amazon MSK and The Orchard's analytics databases. The developer used Amazon MSK to quickly prototype and set up the connector, making spreadsheet data visible across dashboards and available for teams to query and run analytics. “This project took less than 1 day’s work for this employee to complete,” says Umar. “Now, this solution is widely used across our company.” 

The Orchard’s customers also experience faster, more responsive search performance using Amazon OpenSearch Service and can gain relevant information even when using qualitative search queries, which were limited under the previous solution. “Our search capabilities are much more accurate,” says Umar. “We’ve seen a significant reduction in support queries since implementing this solution.” Using AWS services to power its applications, The Orchard can now provide its customers with valuable new features, such as access to audit reports so that administrators can track any changes to their accounts.

The Orchard vastly improved its speed of innovation using its data synchronization solution powered by Amazon MSK and Amazon OpenSearch Service. Its team can deliver valuable new services to customers at a faster pace and will continue to use AWS services to enhance the solution. “Working alongside the AWS team has been really valuable,” says Umar. “It’s nice to know that, as a customer, you always have the support of AWS developers when you need it.”