Azure Machine Learning Experiment Using Python

Azure ML studio provides 3 artifacts for conducting machine learning experiments.

1. Notebooks
2. Automated ML
3. Designer

In this article, we will see how we can use notebooks to build a machine learning experiment.

From Azure ML studio, Click on Start now button on the Notebooks (or, alternatively, click on create new -> Notebook)

I have created a new folder TestModel, and a file called main.py from the interface above.

Structure of Azure Experiment:

It’s important to understand the structure of an experiment in azure and the components involved in successfully executing one.

Workspace:

Azure Machine Learning Workspace is the environment which provides all the resources required to run an experiment. For example, if we were to create a word document, then Microsoft Word in this example would be equivalent to a workspace as it provides all the resources.

Experiment:

An Experiment is a group of Runs ( actual instances of experiments). To create a machine learning model, we may have to create experiment runs multiple times. What groups the individual runs together is an experiment.

Run:

A run is an individual instance of an experiment. A run is one single execution of code. We use run to capture output, analyze results and visualize metrics. If we have 5 runs in an experiment. We can compare the same metrics for 5 runs in one experiment to evaluate the best run and work towards anoptimum model.

Environment:

The environment is another important concept in Azure Machine learning. It defines

1. Python packages
2. Environment variables
3. Docker settings

An Environment is exclusive to the workspace it is created in and cannot be used across different workspaces.

Types of environments:

There are 3 types of environments supported in Azure Machine Learning.

1. Curated: Provided by Azure, intended to be used as-is. Helpful in getting started.
2. User-Managed: We( set up the environment and install packages that are required on compute target.
3. System-Managed: Used when we want Conda to manage Python environment and script dependencies.

We can look at curated and system-managed environments from the environment link in Azure ML Studio.

To create an experiment, we use a control script. The control script decides workspace, experiment, run and some other configuration required to run an experiment.

Creating Control Script

A control script is used to control how and where your machine learning code is run.

from azureml.core import Workspace, Experiment, Environment, ScriptRunConfig

ws = Workspace.from_config()

experiment = Experiment(workspace=ws,name="python-test")

config = ScriptRunConfig(source_directory='./', script='main.py', compute_target="Test-Compute-Manan")

run = experiment.submit(config)

am_url= run.get_portal_url()

print(am_url)

Running the Experiment

Our control script is now capable of instructing Azure Machine Learning workspace to run our experiment from the main.py file.  Azure ML studio automatically takes care of creating experiments and run entries in the workspace we specified. To confirm what our code did, we can head back to our Azure ML workspace. It created an Experiment and a run. Azure automatically creates a fancy display name for a run which in our case is strong malanga. My first few runs failed because of some configuration errors. Running it for 3^rd time marks a successful run for the experiment python-test.

Evaluating Azure Machine Learning Results

Evaluating Azure ML Regression Results.

In the earlier article, we used Azure ML Designer to build a regression model. The final output of the regression model is few metrics which we use to understand how good our regression model is.

There are two steps of interest in evaluating the efficiency of the model. The score model step predicts the price, and evaluate model step finds the difference between prediction and actual price which was already available in the test dataset.

Score Model

Before going to evaluation, it is pertinent to investigate the output of Score Model and what has been scored.

Understanding Scoring

When we were training our model, we selected the Label column as Price.

Training model for label price means in simple terms is :

Using 25 other columns in the data, find what is the best combination of values, which can predict the value of our Label column (price)

Scoring model

1. Used training model to predict the value of price
2. Used test dataset and provide a predicted value of price

Therefore, after scoring, we will have an extra column added at the end of the scored data set, called Scored Labels. It looks like this in preview

This new column “Scored Labels” is the predicted price. We can use this column to calculate the difference between the actual price which was available in the test data set and how the predicted price (Scored Labels) is

The lower the difference, the better the model is. Hence, we will use the difference as a measure to evaluate the model. There are several metrics which we can use to evaluate the difference.

Evaluate Model

We can investigate these metrics by right clicking on Evaluate Model > Preview data > Evaluation results

The following metrics are reported:

Both Mean Absolute Error and Root Mean Square error are averages of errors between actual values and predicted values.

I will take the first two rows of the scored dataset to explain how these metrics evaluate the model.

Absolute Errors

Mean Absolute Error

Evaluates the model by taking the average magnitude of errors in predicted values. A lower value is better

Using the table above, MAE will be calculated as :

854.76 for the above 2 rows is the average error. Let’s assume there was another model whose MAE will be 500.12. If we were comparing two models

Model 1 : 854.76

Model 2 : 500.12

In this case, model 2 will be more efficient than Model 1 as its average absolute error is less.

Root Mean Squared error

RMSE also measures average magnitude of the error. A lower value is better. However, differs from Mean Absolute Error in two ways :

1. it creates a single value that summarized the error
2. Errors are squared before they are averaged, hence it gives relatively high weight to large errors. E.g., if we had 2 error values of 2 & 10, squaring them would make them 4 and 100 respectively. This means that larger values get disproportionately large weightage.

This means RMSE should be more useful when large errors are particularly undesirable.

Using the table above, RMSE will be calculated as :

Relative Errors

To calculate relative error, we first need to calculate the absolute error. Relative error expresses how large the absolute error is compared with the total object we are measuring.

Relative Error = Absolute Error / Known Value

Relative error is expressed as a fraction, or multiplied by 100 to be expressed as a percent.

Relative Squared error

Relative squared error compares absolute error relative to what it would have been if a simple predictor had been used.

This simple predictor is average of actual values.

Example :

In the Automobile price prediction, we have an absolute error of our model which is Total Squared Error.

Instead of using our model to predict the price, if we just take an average of the “price” column. Then find squared error based on this simple average. It will give us a relative benchmark to evaluate our original error with.

Therefore, the relative square error will be :

Relative Squared Error = Total Squared Error / Total Squared Error of simple predictor

Using the two-row example to calculate RSE :

Calculate Sum of Squared based on Simple Predictor of Average

Mathematically relative squared error, Ei of an individual model i is evaluated by :

Relative Absolute Error

RAE compares a mean error to errors predicted by a trivial or naïve model. A good forecasting model will produce a ratio close to zero. A poor model will produce a ratio greater than 1.

Relative Absolute Error = Total Absolute Error / Total Absolute Error of simple predictor

Coefficient of determination

The coefficient of determination represents the proportion of variance for a dependent variable (Prediction) that’s explained by an independent variable (attribute used to predict).

In contrast with co-relation which explains the strength of the relationship between independent and dependent variables, the Coefficient of determination explains to what extent variance of one variable explains the variance of the second variable.

R (Correlation) (source: http://www.mathsisfun.com/data/correlation.html)