Nearmap Brings the World to You with Amazon Web Services

Australian imaging company Nearmap is changing the way we look at the world—literally. Founded in Perth in 2007, Nearmap has made a name for itself by capturing high-resolution images of urban landscapes and offering customers access to its extensive library of aerial imagery and integrated geospatial tools. “Essentially, what we aim to do is provide remote presence,” says John Corbett, Nearmap director of vision systems. “This stops people from having to get into their cars and go out to sites” and allows visualization of areas that are hard to reach in any other way. From oblique and top-down 2D images to large-scale 3D maps of landscapes from all over the world, the company offers its customers very sophisticated products that require a huge amount of compute capacity to reconstruct.

Nearmap has long relied on Amazon Web Services (AWS) for its substantial workloads, storing its data with Amazon Simple Storage Service (Amazon S3) and taking advantage of the Amazon Elastic Compute Cloud (Amazon EC2) G2 Instances. But as its raw data intake expanded and fluctuated, so did its need for a scalable, elastic solution for computing capacity—a solution it found in Amazon EC2 G4 Instances.

Nearmap has designed a high-resolution camera system for efficient image capture across large areas. On a typical day—and up to 364 days a year, depending on the weather—each of Nearmap’s chartered airplanes uses this camera system to capture up to 10 TB of raw data on 5 solid-state hard drives.

The data, once processed, is then stored on Amazon S3, a simple storage service that offers an extremely durable, highly available, and infinitely scalable data storage infrastructure at very low costs. The processed imagery is “stored in Amazon S3 for many, many years,” explains Corbett, giving Nearmap “the ability to just keep on scaling effectively, infinitely.”

But before the data can be turned into usable images, it must first be processed. To do so, Nearmap deploys two pipelines, one for 2D and one for 3D. Data for 2D images can be processed and added to the Nearmap online library as soon as they’re captured, but the fluctuation in the number of images captured throughout the year means a huge variation of workloads. The 2D pipeline’s use is sometimes near zero and sometimes enough to necessitate the use of 8,000–10,000 CPU cores in a day, whereas the 3D pipeline stays fairly fixed at 340 GPUs. “Having the flexibility to ramp up and down on a day-to-day basis is very important for our business,” says Corbett.

Nearmap’s original solution, Amazon EC2 G2 Instances, features high-performance NVIDIA GPUs and is designed to handle substantial graphic workloads. But Nearmap found that its computing needs were reaching the upper limit of the older G-series instance capabilities, and it was beginning to experience a bottleneck in production.

What Nearmap needed was an upgrade, and it knew just where to find one: Amazon EC2 G4 Instances deliver the industry’s most cost-effective and versatile GPU instance for deploying machine learning models in production and graphics-intensive applications. For Nearmap, the impact of upgrading was immediate and profound: the company reduced costs by 67 percent and was able to run three times as much data for the same cost as on Amazon EC2 G2 Instances, in a process that felt effortless and whose integration felt seamless with the AWS tools Nearmap was already using. “When we do have more work that we want to push through,” says Corbett, “there are far fewer barriers to us ramping that cluster up, and really it’s just cranking a dial, and all of a sudden it runs faster.”

Additionally, the high quality of the final 3D maps processed with Amazon EC2 G4 Instances are important to Nearmap, but perhaps an even bigger attraction is the instance flexibility they offered over the Amazon EC2 G2 Instances. Being able to deploy the heavy workloads to instances with just the right mix of GPU, CPU, and RAM for each project was a big draw. The added benefit of being significantly more environmentally friendly is a point of pride for Corbett: “Amazon EC2 G4 Instances are a lot more energy efficient, so compute costs really plummeted, helping us preserve precious energy resources,” says Corbett. Not only did Nearmap’s kilowatt usage drop from around 112 kW to 23 kW, but Nearmap was also able to get more data processed per day. “It makes me sleep a lot easier at night to think that we’re now using a third or a quarter of the energy that we used to,” says Corbett.

As Nearmap continues growing, AWS keeps pace by releasing new innovations and solutions—and Amazon EC2 G4 Instances are a prime example. “It’s just a massive resource that we can count on,” explains Corbett. “We can deploy essentially the largest 3D reconstruction pipeline in the world. We’ll scale it up from 10 to 1,000 GPUs, and it just works.”

Nearmap revolutionized the aerial mapping industry with its proprietary camera system and software, which changed the way aerial maps were constructed. Until Nearmap, aerial maps were especially time consuming and costly to make and as a result were only accessible to very large enterprises and government organizations. But Nearmap has made 2D and 3D maps easier to access for a much broader range of customers with its high-quality, constantly updated library. “We have created a much lower threshold for people to be able to utilize imagery,” says Corbett. This means that 3D content is available for use for public safety and disaster recovery events as well as for planning purposes for activities like 5G telecom deployments. Other uses include envisioning developments before they are built, decreasing the costs of solar panel installations, and even virtual reality.

Now, with the workflow streamlined, Nearmap can look into automating other parts of the business and finding more ways to maximize efficiency. In 11 years, Nearmap has grown from a startup in a single country to covering most of the populated areas in Australia, New Zealand, Canada, and the United States. There’s still a lot of world to capture, and the company’s future calls for innovative tools—like those provided by AWS—that can rise to the occasion.