The Internet of Things on AWS – Official Blog

Part 1/2: Building Reliable IoT Device Software Using AWS IoT Core Device Advisor

This post was co-written by David Walters, Sr Partner Solutions Architect, AWS IoT, and Pavan Kumar Bhat, Sr. Technical Product Manager, AWS IoT Device Ecosystem.

Introduction

Internet of Things (IoT) devices that fail to connect to the internet reliably or are vulnerable to security threats can be catastrophic to IoT device makers. An unreliable IoT device can lead to customer dissatisfaction and loss of customer trust.

IoT device makers building reliable, secure, and scalable IoT devices and applications need comprehensive tools—such as cloud and device testing infrastructure—for testing IoT device software. Once customers deploy IoT devices in production, it becomes increasingly difficult to discover and fix problems such as connectivity or reliability issues. These issues have the potential to negatively impact the customer experience. Fixing the issues may lead to device down time because it will require physical access to the device or unplanned over-the-air updates. These issues can be compounded when you scale from hundreds of devices to millions of devices globally.

AWS IoT Core lets you connect IoT devices to the AWS Cloud without the need to provision or manage servers. AWS IoT Core can support billions of devices and trillions of messages, and provides customers with a fully managed and secure device gateway and message broker. With AWS IoT Core Device Advisor, now generally available, you can test your IoT device’s connection to AWS IoT Core to ensure the device’s software follows the best practices for connectivity, scalability, and security. Device Advisor provides pre-built Message Queuing Telemetry Transport (MQTT), Transport Layer Security (TLS), and security tests that are built based on real-life customer use cases and common errors found during the development of an IoT device.

In this two-part blog series, I will explain how AWS IoT Core Device Advisor provides a managed testing framework for IoT devices that can keep your devices secure and reliable. In this first blog, I showcase how Device Advisor works. In the second blog, I examine a real-world use case to demonstrate how Device Advisor can be used during development to debug an industrial condition monitoring IoT software application.

AWS IoT Core Device Advisor Use Cases and Benefits

Device Advisor test suites can be created and run from the AWS IoT Core console, AWS CLI, or AWS SDKs. Test engineers utilizing the AWS SDKs or AWS CLI can build AWS IoT Core Device Advisor test suites into a Continuous Integration/Continuous Deployment (CI/CD) pipeline to ensure each new version of the device’s software is robust and reliable before triggering an over-the-air (OTA) update or releasing a new product.

Using Device Advisor to spot common errors before deploying your IoT application can help accelerate your time-to-market and increase your customer’s confidence in your solution. Additionally, because AWS IoT Core Device Advisor is a fully managed service, using Device Advisor frees up your engineering team’s time to focus on core business logic and differentiation rather than building and maintaining their own testing capabilities.

Hardware partners that participate in the AWS Device Qualification Program can qualify their device for AWS IoT Core by running the AWS IoT Core Device Advisor qualification test suite with their device. After passing all tests in the qualification suite, partners can download the signed test report and upload it to a Device Catalog submission. The AWS Partner Device Catalog provides a list of devices that are qualified by partners for AWS IoT Core, FreeRTOS, AWS IoT Greengrass, and Amazon Kinesis Video Streams. Qualified devices have already passed core MQTT and TLS connectivity tests in the qualification test suite. This provides you with peace of mind and time to focus on building your IoT applications rather than ensuring your hardware works with AWS services

How it Works

IoT devices connect to AWS IoT Core Device Advisor via TLS1.2 using X.509 certificates and mutual authentication. Device Advisor operates on a separate endpoint than AWS IoT Core, and your production devices remain unaffected while the devices you are testing are connected to Device Advisor.

AWS IoT Core Device Advisor How it works diagram

Figure 1: AWS IoT Core Device Advisor system diagram

First, you configure custom test suites to suit your specific use case and testing requirements. Each test case is run sequentially, and Device Advisor manages the test setup for each test case. You are responsible for connecting your device to the Device Advisor endpoint and triggering the appropriate device-side behavior for each test case.

Device Advisor can test your device’s TLS implementation and AWS IoT policy for security vulnerabilities. Device Advisor evaluates your device’s connectivity to AWS IoT Core by testing the device’s MQTT protocol implementation. Scalability tests like MQTT Exponential Backoff ensure your device does not cause disruptive behavior such as edge network congestion (latency due to bandwidth constraints) in the event of a disconnection event. For a full list of test cases, and their configuration parameters, please see Device Advisor test cases.

The pre-built tests are designed to simulate failure scenarios, and Device Advisor validates that your IoT device responds appropriately to this behavior. There are several TLS test cases that evaluate the security of the device’s TLS implementation. For example, in the ‘Not Signed By Recognized CA’ test case, Device Advisor presents an invalid server certificate and tests that the device appropriately closes the TLS connection when it receives the invalid certificate.

After running a test suite, Device Advisor provides a detailed test report that shows passing and failing tests. Each test case is accompanied with detailed Amazon CloudWatch Logs that show the TLS and MQTT packets exchanged with the device under test. The CloudWatch Logs can be used to debug and fix any issues and test failures.

Setting Up AWS IoT Core Device Advisor

Let’s take a look at the detailed workflow to set up an AWS IoT Core Device Advisor Test Suite from the AWS Management Console. Before starting, create and activate an AWS account if you do not already have one.

  1. Set up the necessary AWS IoT resources for your device. You will need to create an AWS IoT Thing, register a Device Certificate, and attach an AWS IoT Policy. To complete this step, follow the instructions in Create AWS IoT resources from the AWS IoT Core Developer Guide.
  2.  Configure your device to connect to AWS IoT Core Device Advisor by setting the endpoint on your device to connect to Device Advisor’s unique endpoint. Your Device Advisor endpoint will differ from the AWS IoT Core endpoint, but utilizes the same server certificate and device certificate that you setup when connecting to AWS IoT Core.To view your Device Advisor endpoint from the AWS Management Console, navigate to the AWS IoT Core service console, choose Test and then Device Advisor. Choose Start walkthrough and the Device Advisor endpoint will be displayed under Getting started with Device Advisor. 
  3.  Configure an IAM Role for your device and copy the Role ARN. The IAM Role will grant your device privileges to perform actions during Device Advisor testing.
  4. Create an AWS IoT Core Device Advisor Test Suite on the Device Advisor console by navigating to the AWS IoT Core service console, choose Test, Device Advisor, and then Test Suites. On the Test Suites page, choose Create test suite and then Create a new test suite. You can also choose pre-configured test suites, such as the AWS IoT Core Qualification test suite, to qualify your device for listing in the AWS Partner Device Catalog.
  5. Configure your test suite. Choose ‘Test suite properties’. Enter a test suite name and enter the Device Role ARN created in step 3 and click on ‘Update properties’.
  6. Drag and drop the test cases that you wish to run onto the test group. Detailed instructions to set up your test suite and configure test cases is available in the Device Advisor documentation.
  7. Choose the test suite name from the Device Advisor console. 
  8. To run the test suite, choose Actions, and then choose Run test suite. On the next page, choose your AWS IoT Thing that you created in step 1, and then choose Run test.
  9. AWS IoT Core Device Advisor will manage all of the test setup and update the status within the console accordingly. When each test moves from Pending to In Progress, connect your device to the AWS IoT Core Device Advisor test endpoint obtained in Step 2. Some test cases may require the device to perform actions such as publish and subscribe to topics. Your device should perform those steps immediately after connecting to the Device Advisor endpoint.

Examine the test results. If any test resulted in the status of Failed, the full test case log is available in Amazon CloudWatch for closer inspection and debugging. Device Advisor provides links to each test case’s full log. If you selected and passed the AWS IoT Core Device Qualification test suite, you can download the signed test report to upload with your device listing in the AWS Partner Device Catalog. If you did not pass the AWS IoT Core Device Qualification test suite, fix any failed tests and run the test suite again.

Conclusion

In this post, I explained the importance of testing your IoT device software and how AWS IoT Core Device Advisor provides a fully managed testing framework for IoT devices. You can test your device’s connectivity, security, and configuration requirements using Device Advisor.  In the next and final blog post in this series, I examine a real-world industrial condition monitoring use case built with Eurotech’s Everyware Software Framework, and how to debug common IoT device errors using Device Advisor.

To get started using AWS IoT Core Device Advisor, please see the Device Advisor Developer Guide.