Snap Inc. Uses Amazon EC2 G4 Instances to Deliver Bitmoji TV to Millions

Since Snap acquired Bitstrips—later rebranded as Bitmoji—in 2016, the cartoon avatars have become a distinctive feature of the app. On average, more than 70 percent of Snapchat's daily active users have their Bitmojis linked to their Snapchat accounts. Snap began tinkering with the concept of user-personalized content with Bitmoji Stories, and in 2019 the company gave the green light to Bitmoji TV, tasking engineers and animators with overcoming the associated compute obstacles.

Snap’s initial testing of Bitmoji TV found that every second of video playtime required about 1 second of processing. This was a significant delay for a 3-minute video—and one that would drive most users elsewhere. Snap developers needed an efficient, cost-effective way to generate real-time graphics quickly and consistently. “The initial architecture was all CPU rendering and encoding,” says Snap software engineer Brad Kotsopolous. “It would have been expensive to stand up that many instances, so we were exploring some GPU acceleration technologies for our rendering system,” continues Kotsopolous.

Snap engineers worked with graphics processing company NVIDIA to develop a unique rendering system. “We were doing both the frame rendering and the video encoding on the GPU,” says Kotsopolous. “We found that it was really promising—at least a 10 times performance boost over the previous CPU implementation.” Bitmoji had run on the AWS Cloud since before being acquired by Snap, so the team was leaning toward Amazon EC2 G3 Instances, which deliver a powerful combination of CPU, host memory, and GPU capacity. But when AWS released the next-generation Amazon EC2 G4 Instances—which provide the latest generation NVIDIA T4 GPUs—Snap knew it wanted to experiment with the new technology.

The key to creating Bitmoji TV’s real-time rendering of unique videos was finding a balance between power and price. “We benchmarked pretty much all of the Amazon EC2 G4 Instance sizes,” says Chirag Gada, also a software engineer at Snap. “So starting from Amazon EC2 G4 large and then up to XL, to 4XL, even 8XL. We were trying to find the sweet spot on the maximum throughput,” notes Gada. Snap found that sweet spot at 4XL.

Snap also worked with AWS to arrive at the right instance configuration and set up the framework required to render user-specific videos. The avatars were all stored in Amazon DynamoDB, a key-value and document database that delivers single-digit millisecond performance at any scale; and the video frames were stored in Amazon Simple Storage Service (Amazon S3), an object storage service that offers industry-leading scalability, data availability, security, and performance. “For every episode and for every segment of that episode,” says Kotsopolous, “we needed to pull data from Amazon DynamoDB portraying what a particular user looks like and from Amazon S3 looking at how to position that user and that particular frame; then we put it all together and passed it to the renderer. We pushed them into Amazon S3, and then we served them to the user through Amazon CloudFront”—a fast content delivery network service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds.

Amazon EC2 G4 Instances offered more than 10 times performance improvements compared to CPU-based solutions to meet the massive demand of new video releases that would reach millions of unique users per day. In fact, with Amazon EC2 G4 Instances, 1 second of video now renders in only about 100 ms.

With Amazon EC2 G4 Instances implemented, Snap saw peak uses of 1,500–3,500 GPUs when a given episode of Bitmoji TV aired each Saturday morning, with a floor of around 100 GPUs at low-traffic times during the week. Handling this kind of workload on premises would have been prohibitively expensive, notes Kotsopolous: “These GPUs themselves are over $1,000. If we had to buy all the hardware, it would be $3 million.” Using Amazon EC2 Instances, Snap optimizes costs by paying only for what it uses—spinning up and spinning down instances as needed—and the Amazon EC2 G4 Instances in particular turned out to provide the right balance of cost and performance.

“With Amazon EC2 G4 Instances versus Amazon EC2 G3 Instances,” says Kotsopolous, “we were able to get a 50 percent boost in how much traffic we could handle at the same time. The instances themselves cost a bit more. But we were getting a 50 percent boost for a 10 percent higher cost”—a cost savings of 36 percent. This boost in traffic capacity, together with a 45 percent decline in latency, meant that Bitmoji TV viewers were much less likely to see the dreaded spinning wheel.

With Amazon EC2 G4 Instances, Snap was able to create a fun, bold kind of personalized entertainment that rendered quickly enough for millions to enjoy simultaneously while keeping compute costs down. The tenth and final episode of Bitmoji TV season one aired on April 4, 2020, leaving Snap excited to build on its success. “Bitmoji TV tweets seem to be overwhelmingly positive,” says Kotsopolous. “A lot of people are talking about how they enjoy it.” After all, who doesn’t want to be the star of their own show?