Sway Uses Amazon EC2 G4 Instances and ML to Get People Dancin’

The Doritos’ Super Bowl 2020 commercial showcased a dance off between 75-year-old actor Sam Elliott and 21-year-old rapper Lil Nas X, a competition made possible using artificial intelligence technology developed by a startup called Humen, the parent company behind Sway. This tiny company was able to earn a partnership with megacompany Doritos for a Super Bowl campaign and launch Sway, an app that topped the charts in the App Store, with the help of Amazon Web Services (AWS). To take on the colossal workload of launching a graphics-intensive application during the Super Bowl, Humen used Amazon Elastic Compute Cloud (Amazon EC2) G4 Instances—and achieved super results.

The story begins in 2018, when Tinghui Zhou was a PhD student of computer science at UC Berkeley. For his final research project, he cocreated a video called “Everybody Dance Now” in which the movements of expert dancers were transferred to amateurs. That project went on to garner nearly 800,000 views on YouTube. After graduating, Zhou turned that video—and the artificial intelligence technology behind it—into a full-blown company called Humen and its flagship app, Sway.

The basic function is ingenious: the user moves in front of a camera for about 30 seconds, during which time the Pytorch-backed machine learning (ML) model maps out the user’s digital “skeleton.” The user can then choose a dance from a fixed library of dance filters, and the app applies the professional-quality dance to the user’s skeleton, creating a short clip of the user busting impressive dance moves. Even with Pytorch’s ability to handle quick research and prototyping and to array models efficiently, such a sequence of events took quite a while to train with on-premises capacity: “Training a model for the user usually would take hours on a high-end GPU,” says Zhou, now CEO of Humen, “which is really not feasible for a massive consumer use case.”

When the ad agency representing Doritos saw “Everybody Dance Now,” they approached Humen with an idea for a Super Bowl spot. “It was a natural fit but definitely a leap of faith for them—using a tiny startup for such a high-profile, visible campaign,” says Zhou. Humen realized that an event as flashy as the Super Bowl was an opportune time to launch the Sway app. However, this meant that the seven-person Humen team had mere months to build an ML infrastructure that would amplify the tech behind “Everybody Dance Now” for Sway’s anticipated incoming tidal wave of users.

Projecting a spike in Sway users from prelaunch to postlaunch, the Humen team knew an on-premises solution wasn’t going to work. So the company looked to the cloud. At first, the team tried Amazon EC2 P3 instances, which use NVIDIA Tesla V100 GPUs to significantly accelerate ML and high-performance computing applications. Though the Amazon EC2 P3 instances are fast, Humen needed to consider the business reality of spinning up enough instances for a Super Bowl–scale campaign: “We did a cost analysis and reached a conclusion,” says Zhou. “Amazon EC2 G4 Instances would make a lot more sense from a cost perspective—a roughly three times cost reduction—and wouldn’t compromise the user experience.”

And so Humen proceeded with Amazon EC2 G4 Instances, the industry’s most cost-effective and versatile GPU-based instance for deploying ML models in production and graphics-intensive applications. And with the model innovations and engineering efforts that Humen made throughout the year, the company saw its ML training time shrink from hours to minutes, with Amazon EC2 G4 Instances handling the workloads to maintain that rate even at Super Bowl scale.

Beyond providing Humen with reserved instances, AWS had a hand in helping it prepare for the launch. “I had a really good relationship with the Infrastructure Event Management team during the weeks leading up to it,” says Humen engineer Jesse Myers. “They were able to help us find a couple of bottlenecks in the system.” During the Super Bowl, the AWS team helped to identify and solve database IOPS bottlenecks. And when AWS and Humen pinpointed a specific AWS API call that was being rate limited and realized it could not be increased to suit Humen’s needs, AWS helped the company to architect a new solution that Myers implemented in less than a day.

When the big day came, Humen was ready. “With Amazon EC2 G4 Instances, I was able, in about 6 months, to make Sway capable of handling Super Bowl scale,” says Myers. “A big part of that is the fact that I could use AWS Cloud resources and had good automation using AWS APIs.”

The commercial aired. After the dance off, a surge of users heeded Lil Nas X’s call to action—“Who’s got next?”—and downloaded the app as prompted. Users generated videos of themselves performing the same moves as Lil Nas X and Sam Elliott and posted them to their social media accounts using the hashtag #CoolRanchDance. At one point, Sway received 2,000 API requests per second, which required the compute capacity of more than 3,000 GPUs, and AWS delivered with Amazon EC2 G4 Instances.

In the wake of the Super Bowl, Sway hit number two on the entertainment list of the App Store and number three overall, beating out social media giants and drawing unsolicited celebrity attention from the likes of Diplo and Ryan Seacrest. Of course, every day isn’t the Super Bowl. As the need for compute capacity stabilized, Humen was able to save money and energy by spinning down the number of Amazon EC2 G4 Instances they were using—something that wouldn’t be possible with a traditional data center.

Even while enjoying the afterglow of a successful launch, Humen cast its eye to the future. Beyond the dance videos that Sway currently offers, Zhou wants to expand to sports, TV, gaming, and movies. Additionally, the team wants to provide a way for users to share their created content within the app for collaboration. Humen is also trying to improve the photorealism of its output, including bringing 3D-based perception into its skeleton and scene-analysis ML models. “AWS provides a very scalable infrastructure for us to experiment with all these new features easily,” says Zhou. This infrastructure includes more containerization, which will allow Humen to split its loads into smaller pieces that can take advantage of the different instance types for different kinds of compute.

Speaking about the suitability of AWS for startups like Humen, Myers says, “If I want to have all the different pieces work nicely together and have control of the network and really be able to think about all the things you need to operate a system, there was really only one answer that made sense.” Humen was able to pull off an incredible feat with Amazon EC2 G4 Instances, showing how AWS makes a great collaborator for startups looking to grow quickly.

To learn more, visit aws.amazon.com/ec2/instance-types/g4/.