Image Classification In ASP.NET Core Using ML.NET

Image classification is a common case in many business scenarios. For these cases, you can either use pre-trained models or train your own model to classify images specific to your custom domain.

There are two data sources: the tsv file and the image files. The tsv file contains two columns: the first one is defined as ImagePath and the second one is the Label corresponding to the image. As you can observe, the file does not have a header row, and looks like this:

• Wikimedia Commons, the free media repository. Retrieved 10:48, October 17, 2018 from:
• https://commons.wikimedia.org/wiki/Pizza
• https://commons.wikimedia.org/wiki/Coffee_pot
• https://commons.wikimedia.org/wiki/Toaster
• https://commons.wikimedia.org/wiki/Category:Canoes
• https://commons.wikimedia.org/wiki/Teddy_bear

There are multiple models which are pre-trained for classifying images. In this case, we will use a model based on an Inception topology, and trained with images from Image.Net. This model can be downloaded from here, but it's also available here.

The class library project ImageClassification.Score can be used to classify sample images based on the pre-trained Inception-5h TensorFlow model.

Code Walkthrough

ML.NET: Model Scoring

The image file used to load images has two columns: the first one is defined as ImagePath and the second one is the Labelcorresponding to the image.

It is important to highlight that the label in the ImageNetData class is not really used when scoring with the TensorFlow model. It is used when testing the predictions so you can compare the actual label of each sample data with the predicted label provided by the TensorFlow model.

Finally, we extract the prediction engine after fitting the estimator pipeline. The prediction engine receives as a parameter an object of type ImageNetData (containing 2 properties: ImagePath and Label) and then returns an object of type ImagePrediction.

When obtaining the prediction, we get an array of floats in the property PredictedLabels. Each position in the array is assigned to a label, so for example, if the model has 5 different labels, the array will be length = 5. Each position in the array represents the label's probability in that position; the sum of all array values (probabilities) is equal to one. Then, you need to select the biggest value (probability), and check which is the assigned label to that position.

Download this from here and include this project into your Visual Studio solution.

Let's start to build ASP.NET Core application which will consume the Tensor Flow pre-trained model using ML.NET API.

• Open Visual Studio>New Projects>ASP.NET Core
• Set Project name as ImageClassification.Web
• Click Create button>MVC Web Application
• A new Project will be added to your solution.
• Open Nuget Manager Console and write Install-Package Microsoft.ML
• Right-click on the project and Add refrence>Select ImageClassification.Score>Press Ok
• Download assets from here and include in project
• Here is how your project will look,
  
  
  
  
  
• Now we need to consume the model. Create a new Controller or Open HomeController and add the following snippet.
  1. using System;
  2. using System.Collections.Generic;
  3. using System.Diagnostics;
  4. using System.Linq;
  5. using System.Threading.Tasks;
  6. using Microsoft.AspNetCore.Mvc;
  7. using ImageClassification.Web.Models;
  8. using System.IO;
  9. using ImageClassification.ModelScorer;
  10. using Microsoft.AspNetCore.Http;
  11. using ImageClassification.ImageDataStructures;
  13. namespace ImageClassification.Web.Controllers
  14. {
  15. public class HomeController : Controller
  16. {
  17. public IActionResult Index()
  18. {
  19. return View();
  20. }
  21. [HttpPost]
  22. public async Task<IActionResult> Index(IFormFile image)
  23. {
  24. string assetsRelativePath = @"../../../assets";
  25. string assetsPath = GetAbsolutePath(assetsRelativePath);
  27. var tagsTsv = Path.Combine(assetsPath, "inputs", "images", "tags.tsv");
  28. var imagesFolder = Path.Combine(assetsPath, "inputs", "images");
  29. //Path of Tensor Flow pre trained model
      
  30. var inceptionPb = Path.Combine(assetsPath, "inputs", "inception", "tensorflow_inception_graph.pb");
  31. var labelsTxt = Path.Combine(assetsPath, "inputs", "inception", "imagenet_comp_graph_label_strings.txt");
  33. try
  34. {
  35. var filePath = Path.GetTempFileName();
  36. //Save image to temp
  37. using(var ms=new FileStream(filePath,FileMode.Create))
  38. {
  39. await image.CopyToAsync(ms);
  40. }
  41. //Create Tensor Flow model for image classification and prediction using ML.NET API
      
  42. var modelScorer = new TFModelScorer(tagsTsv, imagesFolder, inceptionPb, labelsTxt);
  43. //Check image prediction
  44. var prediction=(ImageNetDataProbability)modelScorer.Score(new ImageNetData() { ImagePath=filePath,Label=""});
  45. //Store results to ViewBag to show in View
  46. ViewBag.PredictedLabel = prediction.PredictedLabel;
  47. ViewBag.Probability = prediction.Probability;
  48. }
  49. catch
  50. {
  51. return View();
  52. }
  53. return View();
  54. }
  55. public static string GetAbsolutePath(string relativePath)
  56. {
  57. FileInfo _dataRoot = new FileInfo(typeof(Program).Assembly.Location);
  58. string assemblyFolderPath = _dataRoot.Directory.FullName;
  59. string fullPath = Path.Combine(assemblyFolderPath, relativePath);
  60. return fullPath;
  61. }
  62. }
  63. }
• Now we need to create an index view which will be used to upload image to server and show image classification or prediction result from server. Please add following snippet to your view.
  1. @{
  2. ViewData["Title"] = "Home Page";
  3. }
  4. <h2>Image Classification in ASP.NET Core Using ML.NET with TensorFlow</h2>
  5. <hr />
  6. <div class="row">
  7. <div class="col-md-6">
  8. <form method="post" enctype="multipart/form-data">
  9. <input type="file" name="image" />
  10. <button class="btn btn-sm btn-primary" type="submit">Classify</button>
  11. </form>
  12. </div>
  13. <div class="col-md-6">
  14. @if (ViewBag.Probability != null)
  15. {
  16. <h3>Prediction</h3>
  17. <h6>Predicted Label: @ViewBag.PredictedLabel</h6>
  18. <h6>Probability: @ViewBag.Probability</h6>
  19. }
  21. </div>
  22. </div>
• Now try to build and run the ImageClassification.Web application

Demo

Conclusion