Automatically Create Machine Learning Models

In the AWS Management Console type SageMaker into the search bar, and then choose Amazon SageMaker.

Select Canvas from the menu on the left.

On the Canvas page select Open Canvas.

If you did not already have Canvas running, it will take around 5 minutes for Canvas to start.
Once you see the Canvas Home page, you’ll select My Models on the left.

Select My Models on the left pane.

Select + Create new model  

For Model name you’ll type Marketing Campaign, leaving the Problem type set to Predictive analysis, and before clicking Create.

On the Select dataset page you’ll select + Create dataset  

Name the dataset MarketingData and click Create.

Download the dataset from this link: https://sagemaker-sample-files.s3.amazonaws.com/datasets/tabular/uci_bank_marketing/bank-additional-full.csv

Since you downloaded the data to your local machine, leave Data Source as Local upload, Click Select files from your computer or drag the bank-additional-full.csv file on to the window, and once the file is processed, you’ll Click Preview dataset. While local upload is convenitent for this usecase, there are also many other data source options available to make it easy to access the data sources required for different use cases. You can learn more about the different options here: https://docs.aws.amazon.com/sagemaker/latest/dg/canvas-importing-data.html

The preview will display the first 100 rows of the data. If everything looks good, you’ll click Create dataset.

After the dataset is ready, select the radio button beside MarketingData and click Select dataset.

From the Target Column drop down you’ll select y and click the down arrow for the Quick Build button to select Standard build. While the defaults are appropriate for this use case, before clicking Standard build, explore the Configure model options.  

Under Objective metric you could change the metric AutoPilot uses to optimize while building your model. AutoPilot generally selects the best metric based on the model type, but this allows you to override that based on your specific requirements.   

Under Training method and algorithms, you can optionally, select Ensemble or Hyperparameter optimization (HPO) as the approach to develop the model. With Auto SageMaker AutoPilot will use Ensemble for datasets smaller than 100MB and HPO for datasets larger than 100MB.  

Under Data split you’ll have to option to specify the percentage of data used during model development vs testing of the model. 80% for training and 20% validation is a good starting point for most datasets.

Under Max candidates and runtime, you’ll be able to set the maximum number of model candidates AutoPilot is allowed to generate and the maximum amount of time that AutoPilot is allowed to run a build job. Note: Max Candidates is only available for HPO training mode. We highly recommend that you keep the maximum job runtime greater than 30 minutes to ensure that AutoPilot has enough time to generate model candidates and finish building your model.

SageMaker AutoPilot will begin to run through the phases of model building. In the Model overview window you can monitor the process. It should take around 45 minutes for your build to complete.

After model building is complete, SageMaker Canvas automatically switches to the Analyze tab to show the quick training results. The SageMaker Canvas model selected based on the F1 score optimization can predict the class accurately 91.296% of the time. Machine Learning introduces some stochasticity in the process of training models, which can lead to different results to different builds. Therefore, the exact performance in terms of the metrics that you see might be different.  

In the Overview section the view you see is called Column impact and represents the aggregated SHAP value for each feature across each instance in the dataset. The column impact score is an important part of model explainability because it shows what features tend to influence the predictions the most in the dataset. In this use case, the customer duration or tenure and employment variation rate are the top two fields for driving the model's outcome.

To view other models or change the selected model, click the Model leaderboard link.

If you would like to analyze a different model from the leaderboard, selecting the three dots on the right and selecting View model details will take you to explainability and performance details for that model or you can Change to default model if you wanted to work with that model. You can click the X at the top right to close the Model leaderboard view.

Now, click on the Scoring tab. You will see a visual representation of Predicted vs. Actual to help you understand the accuracy of the model in predicting the different classes, in this case yes or no. You will also see, Model accuracy insights to help you interpret the visualization. Highlighting that if the model predicts the marketing campaign will not be effective, it is correct 96.872% of the time and that if the campaign was not effective, the model predicted it would not have been 93.201% of the time.

Now, click on the Advanced metrics tab. You will find detailed information on the model’s performance, including recall, precision, and accuracy. You can also interpret model performance and decide if additional model tuning is needed.

Next, visualizations are provided to further illustrate model performance. First, look at the confusion matrix. The confusion matrix is commonly used to understand how the model labels are divided among the predicted and true classes. In this case, the diagonal elements show the number of correctly predicted labels and the off-diagonal elements show the misclassified records. A confusion matrix is useful for analyzing misclassifications due to false positives and false negatives.

Finally, look at the precision versus recall curve. This curve interprets the label as a probability threshold and shows the trade-off that occurs at various probability thresholds for model precision and recall. SageMaker AutoPilot automatically optimizes these two parameters to provide the best model.

To start generating predictions, choose the Predict button at the bottom of the Analyze page, or choose the Predict tab. On the Predict page, Batch prediction is already selected. To make a one time batch prediction choose Manual and then select the MarketingData dataset. Selecting Automatic would allow you to make batch predictions for a dataset every time the data set is updated. In actual ML workflows, this dataset should be separate from the training dataset. However, for simplicity, you use the same dataset to demonstrate how SageMaker Canvas generates predictions. Choose Generate predictions.

After a few seconds, the prediction is done. Choose View from the message window at the bottom of the page to see a preview of the predictions. You can also choose Download to download a CSV file containing the full output. SageMaker Canvas returns a prediction for each row of data.

On the Predict page, you can generate predictions for a single sample by selecting Single prediction. SageMaker Canvas presents an interface in which you can manually enter values for each of the input variables used in the model. This type of analysis is ideal for what-if scenarios where you want to know how the prediction changes when one or more variables increase or decrease in value.

After the model building process, SageMaker AutoPilot uploads all artifacts including the trained model saved as a pickle file, metrics, datasets, and predictions to an S3 Bucket in your account named sagemaker-<your-Region>-<your-account-id> under a location named Canvas/<you-user>/. You can inspect the contents and use them as necessary for further development.

While it is outside the scope of this tutorial, from the deploy tab you can easily deploy you model to a SageMaker endpoint so you can make predictions from outside of the Canvas application, test and monitor your model to proactively detect issues such as model drift.

From the deploy tab, select + Create Deployment. Set the desired endpoint configuration and select Deploy.

From My models you also have the ability to Add the model you created to Model Registry so Studio users can catalogue, review and deploy the model in SageMaker Studio.

Select the model you want to add to Model Registry then on the versions Tab click the three dots to the right of the version you would like to add to Model Registry and select Add to Model Registry.

Navigate to the S3 console and choose Buckets. Navigate to your bucket named sagemaker-<your-Region>-<your-account-id> and select the check box to the left of all of the files and folders. Next, choose Delete.

• On the Delete objects page, verify that you have selected the proper objects to delete. In the Permanently delete objects section, confirm by entering permanently delete in the text field and choose Delete objects. After completion and the bucket is empty, you can delete the S3 bucket by following the same process. A success banner appears after deletion is complete.

On the SageMaker Canvas main page, choose Models. On the right pane, the model you built is visible. Choose the vertical ellipsis to the right of the View option and select Delete model.  

After the model is deleted, click on Log out to end your Canvas session.  

If you used an existing SageMaker Studio domain in Step 1, skip the rest of Step 6 and proceed directly to the conclusion section.  

If you ran the CloudFormation template in Step 1 to create a new SageMaker Studio domain, continue with the following steps to delete the domain, user, and the resources created by the CloudFormation template.   

To open the CloudFormation console, enter CloudFormation into the AWS console search bar, and choose CloudFormation from the search results.  

 Open the CloudFromation console. In the CloudFormation pane, choose Stacks. From the status dropdown list, select Active. Under Stack name, choose CFN-SM-IM-Lambda-catalog to open the stack details page.  

On CFN-SM-IM-Lambda-catalog stack details page, choose Delete to delete the stack along with the resources it created in Step 1.