In this article we describe how to define, train, evaluate, persist, and use a ML.NET Recommendation model in a UWP app. The blog post is part of a series on implementing different Machine Learning scenarios with .NET Open Source frameworks and components such as
All articles in the series are supported by the same UWP sample app that lives here on GitHub. Since the previous article was published, this sample app was upgraded to the latest prereleases of ML.NET thanks to Pull Requests from the Microsoft ML.NET Team itself (thanks Eric!). This means that the syntax in the code snippets is quite different from the previous articles, but much closer to the imminent official release.
Here’s how the Recommendation page in the sample app looks like:
It builds a model to generate recommendations for hotels on the Las Vegas Strip for a selected traveler type (single, family, business, …):
- when you select a traveler type in the combo box, the top 10 recommended hotels appear in the diagram, and
- when you select a hotel in the second combo box, a predicted rating will appear next to it.
Recommendation in Machine Learning
Machine Learning recommender systems are highly popular in e-commerce and social networks. They’re used for recommending books, TV series, music, events, products, friends, dating profiles, and a lot more.
There are two approaches for generating recommendations:
- Content Based Filtering recommends items to a user that are similar to previously highly rated items by the same user. The advantage of this is transparency (the model can explain why it recommends the item). Unfortunately this approach does not scale well with large data.
- Collaborative Based Filtering will recommend the items to a user that were highly rated by other -but similar- users. In most real world scenarios not every user has rated every item, so the base data can be very sparse. This makes the approach unsuitable in some scenarios.
Matrix Factorization in Machine Learning
Matrix Factorization is a common technique to solve the sparsity problem with Collaborative Base Filtering that we just mentioned. In a nutshell its goal is to mass-predict the missing ratings. Matrix Factorization is entirely based on linear algebra which is something that your CPUs, GPUs, and/or AI Accelerators are pretty good at. If you want to dive into the mathematical details, allow me to recommend (pun intended) the article with the very appropriate name A Gentle Introduction to Matrix Factorization for Machine Learning.
Major advantages of this algorithm are that it scales very well with large data and it is very fast. You don’t have take my word for it, but there must me a reason why Amazon and Netflix are relying on it. The algorithm has the disadvantage that it cannot always easy explain why it recommends an item. You must have stumbled upon recommendations like this before:
Before we dive into the code, allow us to clarify something: Matrix Factorization does NOT answer the question “What items would you recommend for this user?”. Instead it solves the “Here’s a list of products and a (list of) user(s), please predict their ratings” problem. So when you use it in your apps, there is some preprocessing (selecting the products to evaluate) and some postprocessing (filtering relevant recommendation) to do. Basically the algorithm always has to deal with too much data. Don’t worry about that: Matrix Factorization is a real Weapon of Mass Prediction…
Matrix Factorization in ML.NET
For Matrix Factorization in ML.NET you’ll need the MatrixFactorizationTrainer class. It comes in a separate NuGet package (Microsoft.ML.Recommender):
Model Input and Output
For training and testing the model, we’ll use a 2015 dataset with 510 Las Vegal hotel ratings from TripAdvisor. Here’s how it looks like:
Matrix Factorization predicts the rating (“Label”) between only two fields (“Features”) . If you have to deal with more fields, then you’ll need the FieldAwareFactorizationMachine instead.
In the sample app we choose TravelerType and Hotel as features to respectively play the roles of ‘similar user’ and ‘recommended item’. The Score column contains the rating and will play the role of ‘label’ (the thing to predict). Since the prediction engine’s output column is also called Score, we renamed it to Label for the input.
Here’s the structure of input samples that we will feed the model with:
public class RecommendationData
{
public float Label;
public string TravelerType;
public string Hotel;
}
The prediction looks like this:
public class RecommendationPrediction
{
public float Score;
public string TravelerType;
public string Hotel;
}
Observe the lack of LoadColumn and ColumnName attributes on top of the fields – we had these in all the previous posts in this article series. We don’t need the attributes here because we’re not using a TextLoader to read the training and testing data sets. Instead we’ll create our IDataView with a call to the LoadFromEnumerable() method. This same method allows you to populate the model with records from a database:
private IDataView trainingData;
public IEnumerable<RecommendationData> Load(string trainingDataPath)
{
var data = File.ReadAllLines(trainingDataPath)
.Skip(1)
.Select(x => x.Split(';'))
.Select(x => new RecommendationData
{
Label = uint.Parse(x[4]),
TravelerType = x[6],
Hotel = x[13]
})
.OrderBy(x => (x.GetHashCode())) // Cheap Randomization.
.Take(400);
// Populating an IDataView from an IEnumerable.
trainingData = _mlContext.Data.LoadFromEnumerable(data);
// Keep DataView in memory.
trainingData = _mlContext.Data.Cache(trainingData);
// Populating an IEnumerable from an IDataView.
return _mlContext.Data.CreateEnumerable<RecommendationData>(trainingData, reuseRowObject: false);
}
Part of the data set will be used for training, and another for evaluating the model. Since the original data set is sorted on Hotel name, we applied cheap randomization logic to the rows by sorting them on their GetHashCode() value.
The Cache() method keeps the selected columns (in our case: all columns) in memory after they’re accessed for the first time. For iterative algorithms this really is a time saver – at least if the data fits into memory.
Defining and Building the Model
The recommendation model is an ITransformer that is created from an EstimatorChain with a MatrixFactorization at its heart. You have to specify the label (labelColumn) and the two features (matrixRowIndexColumnName and matrixColumnIndexColumnName) and some options to fine tune the algorithm. Before sending the feature values to the transformer, they’re added to a dictionary with MapValueToKey(). The reverse function of that is MapKeyToValue(). It ensures that the original values are returned with the predicted score.
Here’s the whole pipeline:
private ITransformer _model;
public void Build()
{
var pipeline = _mlContext.Transforms.Conversion.MapValueToKey("Hotel")
.Append(_mlContext.Transforms.Conversion.MapValueToKey("TravelerType"))
.Append(_mlContext.Recommendation().Trainers.MatrixFactorization(
labelColumn: DefaultColumnNames.Label,
matrixColumnIndexColumnName: "Hotel",
matrixRowIndexColumnName: "TravelerType",
// Optional fine tuning:
numberOfIterations: 20,
approximationRank: 8,
learningRate: 0.4))
.Append(_mlContext.Transforms.Conversion.MapKeyToValue("Hotel"))
.Append(_mlContext.Transforms.Conversion.MapKeyToValue("TravelerType"));
// Place a breakpoint here to peek the training data.
var preview = pipeline.Preview(trainingData, maxRows: 10);
_model = pipeline.Fit(trainingData);
}
The extremely useful Preview() method was recently added to the API. It allows you to inspect the content and schema of the pipeline while debugging – it feels a bit like the old SSIS Data Viewer:
The prediction model is trained with a Fit() call.
Evaluating the Model
It’s always a good idea to evaluate your freshly trained model. Typically this is done by sending it a set of previously unknown –but labeled- data set rows. The Transform() call generates the predictions, while Evaluate() compares these with the original labels:
public RegressionMetrics Evaluate(string testDataPath)
{
//var testData = _mlContext.Data.LoadFromTextFile<RecommendationData>(testDataPath);
var data = File.ReadAllLines(testDataPath)
.Skip(1)
.Select(x => x.Split(';'))
.Select(x => new RecommendationData
{
Label = uint.Parse(x[4]),
TravelerType = x[6],
Hotel = x[13]
})
.OrderBy(x => (x.GetHashCode())) // Cheap Randomization.
.TakeLast(200);
var testData = _mlContext.Data.LoadFromEnumerable(data);
var scoredData = _model.Transform(testData);
var metrics = _mlContext.Recommendation().Evaluate(scoredData);
// Place a breakpoint here to inspect the quality metrics.
return metrics;
}
The evaluation returns a RegressionMetrics instance with useful information on the quality of the model – such as the coefficient of determination, and the relative squared error:
If you notice that your model lacks accuracy, then you need to fine tune its parameters and/or provide more representative training data and/or select another algorithm.
Persisting the Model
The model can be serialized and persisted with a call to Save():
public void Save(string modelName)
{
var storageFolder = ApplicationData.Current.LocalFolder;
string modelPath = Path.Combine(storageFolder.Path, modelName);
_mlContext.Model.Save(_model, inputSchema: null, filePath: modelPath);
}
Inferencing with the model
There are two ways for creating recommendation scores. The first one generates a prediction for a single feature combination: a score for one specific traveler type/hotel combination. The API for this scenario cannot be more straightforward: you create a prediction engine with CreatePredictionEngine() and then you call Predict() to … predict:
public RecommendationPrediction Predict(RecommendationData recommendationData)
{
// Single prediction
var predictionEngine = _model.CreatePredictionEngine<RecommendationData, RecommendationPrediction>(_mlContext);
return predictionEngine.Predict(recommendationData);
}
This code is triggered when you select a hotel from the lower left combo box on the page:
The second way to generate recommendations takes a list of feature pairs instead of a single one. When you select an entry in the traveler type combo box, we first create a list of RecommendationData records – one for each hotel in the original data set. Then we call the Predict() method in the ViewModel – the sample app uses a lightweight MVVM architecture:
// Group Prediction
var recommendations = new List<RecommendationData>();
foreach (var hotel in ViewModel.Hotels)
{
recommendations.Add(new RecommendationData
{
Hotel = hotel,
TravelerType = TravelerTypesCombo.SelectedValue.ToString()
});
}
var predictions = await ViewModel.Predict(recommendations);
This list is changed into an IDataView with same the LoadFromEnumerable() call that we encountered when loading the training data. The recommendation model transforms it into a IDataView that adheres to the output schema through the Transform() method. Finally, with the CreateEnumerable() method this structure is translated to a list of prediction entities:
public IEnumerable<RecommendationPrediction> Predict(IEnumerable<RecommendationData> recommendationData)
{
// Group prediction
var data = _mlContext.Data.LoadFromEnumerable(recommendationData);
var predictions = _model.Transform(data);
return _mlContext.Data.CreateEnumerable<RecommendationPrediction>(predictions, reuseRowObject: false);
}
There are 21 hotels in the data set, so this method returns 21 ratings. The end user is of course not interested in all of these. With a little LINQ query you can get the 10 most appropriate recommendations:
var recommendationsResult = predictions
.Select(p => p)
.OrderByDescending(p => p.Score)
.ToList()
.Take(10)
.Reverse();
[Note: The reverse() is only there because we build up the bar chart from bottom to top.]
A word of warning
The current NuGet package for Microsoft.ML carries the v1.0.0-preview tag, so we may be close to an official release. This is not the case for the Microsoft.ML.Recommender. This one seems to need some extra stabilization sprints. In its current version, Matrix Factorization yields different types of exceptions when you’re running in x86 mode. With a little luck you only get weird results like these:
Don’t worry, it’s a known issue, the team is working on it…
Let there be XAML
Let’s jump to the visualization of the predictions. For the horizontal bar chart on the sample page, we borrowed the diagram from the MultiClass Classification sample. XAML-wise we declared a PlotView with its PlotModel. The model has a CategoryAxis for the hotel names and a LinearAxis for the predicted score (0-5). The values are represented in a BarSeries:
<oxy:PlotView x:Name="Diagram"
Background="Transparent"
BorderThickness="0"
Margin="0 0 40 60"
Grid.Column="1">
<oxy:PlotView.Model>
<oxyplot:PlotModel Subtitle="Recommended Hotels"
PlotAreaBorderColor="{x:Bind OxyForeground}"
TextColor="{x:Bind OxyForeground}"
TitleColor="{x:Bind OxyForeground}"
SubtitleColor="{x:Bind OxyForeground}">
<oxyplot:PlotModel.Axes>
<axes:CategoryAxis Position="Left"
TextColor="{x:Bind OxyForeground}"
TicklineColor="{x:Bind OxyForeground}"
TitleColor="{x:Bind OxyForeground}" />
<axes:LinearAxis Position="Bottom"
Title="Predicted Score (higher is better)"
TextColor="{x:Bind OxyForeground}"
TicklineColor="{x:Bind OxyForeground}"
TitleColor="{x:Bind OxyForeground}" />
</oxyplot:PlotModel.Axes>
<oxyplot:PlotModel.Series>
<series:BarSeries LabelPlacement="Inside"
LabelFormatString="{}{0:0.00}"
TextColor="{x:Bind OxyText}"
FillColor="{x:Bind OxyFill}" />
</oxyplot:PlotModel.Series>
</oxyplot:PlotModel>
</oxy:PlotView.Model>
</oxy:PlotView>
When the predictions come in, we add category (with the name) and a BarItem (with the score) for each of the maximum 10 hotels. These are added to their respective series, and the plot is refreshed:
// Update diagram
var categories = new List<string>();
var bars = new List<BarItem>();
foreach (var prediction in recommendationsResult)
{
categories.Add(prediction.Hotel);
bars.Add(new BarItem { Value = prediction.Score });
}
var plotModel = Diagram.Model;
(plotModel.Axes[0] as CategoryAxis).ItemsSource = categories;
(plotModel.Series[0] as BarSeries).ItemsSource = bars;
plotModel.InvalidatePlot(true);
That’s it for today. The UWP sample app –which is featured on the ML.NET Community Samples page- lives here on GitHub.
Enjoy!
). In the mean time we’ll stick to 
