Implementing neural networks in C# - Part 2

In this post, we revisit key aspects of logistic regression and highlight its limitations, thereby motivating the introduction of more powerful algorithms.

What is logistic regression ?

We will not delve into the details of logistic regression in this post, as we have already dedicated an entire series to this topic earlier. We refer the reader to this link. Here, we will simply recap the final results.

Logistic regression is a statistical method used for binary classification problems, where the outcome variable has two possible classes. It models the probability that an instance belongs to a particular class, and its predictions are in the form of probabilities between 0 and 1. The logistic function $\sigma$ (sigmoid) is employed to transform a linear combination of input features into a probability score.

$$\sigma (x)=\dfrac{1}{1+e^{-x}}$$

Observing logistic regression in action

To witness logistic regression in action, we will swiftly implement it with ML.NET in the straightforward example outlined below, where we have two possible outcomes for 2-dimensional inputs.

The corresponding dataset is illustrated below.

 1X;Y;Category
 20.12;0.154;1
 30.135;0.26;1
 40.125;0.142;1
 50.122;0.245;1
 60.115;0.142;1
 70.132;0.2;1
 80.84;0.76;0
 90.834;0.8;0
100.855;0.78;0
110.865;0.84;0
120.835;0.82;0
130.82;0.745;0

Implementing logistic regression with ML.NET

ML.NET is an open-source machine learning framework developed by Microsoft that allows developers to integrate machine learning models into their .NET applications. ML.NET supports a variety of machine learning tasks, including classification, regression, clustering, and recommendation.

The C# code for employing logistic regression is presented below.

 1static void Main(string[] args)
 2{
 3    var ctx = new MLContext();
 4
 5    // Load data from file
 6    var path = AppContext.BaseDirectory + "/dataset.csv";
 7    var data = ctx.Data.LoadFromTextFile<DataRecord>(path, hasHeader: true, separatorChar: ';');
 8                
 9    var dataPipe = ctx.Transforms.Concatenate("Features", new[]
10    {
11        "X", "Y"
12    }).Append(ctx.BinaryClassification.Trainers.LbfgsLogisticRegression(featureColumnName: "Features"));
13               
14    
15    var model = dataPipe.Fit(data);  
16   
17    // Predict an unforeseen input
18    var record = new DataRecord()
19    {
20        X = 0.25f,
21        Y = 0.24f
22    };
23
24    var pe = ctx.Model.CreatePredictionEngine<DataRecord, DataRecordPrediction>(model);
25    var category = pe.Predict(record);
26}

Here are the predicted values for some generated inputs.

This example is a simplified one, as the data is well-separated, making it relatively easy for a method like logistic regression to predict the correct values. But how will this algorithm perform with a significantly more complex dataset ?

What happens if data is not linearly separable ?

We will now apply logistic regression to the dataset depicted below. It is notably more intricate than the previous one (despite being in a two-dimensional space, facilitating data representation), and, most importantly, the data is not linearly separable. The objective is to observe how logistic regression behaves under such circumstances.

We will utilize the identical C# code as above (with different data), and observe some predicted values.

Here, it is evident that many predicted values are inaccurate. The algorithm consistently returns the same predicted probability (0.5), indicating that it cannot adapt to the specific problem at hand.

This is normal as we must recall that logistic regression applies a sigmoid to linear data and can only be effective in separating linearly separable data. As it is clearly not the case in the given sample, this method proves to be inappropriate.

Is it possible to alleviate this phenomenon ?

The short answer is yes: theoretically, there are mathematical methods that can address this issue, and we will observe them in action in the next post. The extended answer is that, in practice, it can be very challenging.

Implementing neural networks in C# - Part 3