Truly understanding logistic regression - Part 1

Dec 28, 2023 · 2 min read ·

Share on:

Starting today, we are launching a series of articles on logistic regression, taking a progressive approach: our purpose is to blend theoretical concepts with practical implementation (in C#), offering clear illustrations of the associated challenges (with a particular focus on the IRLS algorithm).

Machine learning is a highly popular field with a wide range of applications. It is used for tasks such as predicting future values (regression) or identifying the most likely category from a set of predefined classes (classification). In this series, we will focus on the latter and our goal is ultimately to train a model that can learn patterns and relationships in the data, allowing it to make predictions or assign labels to new, unseen instances.
Classification tasks are prevalent in various domains, such as spam detection (classifying emails as spam or not spam), image recognition (recognizing digits in handwritten digits recognition) or sentiment analysis (determining the sentiment (positive, negative, neutral) in textual data). It is therefore a crucial topic in artificial intelligence that every practitioner should be familiar with.

Among the various algorithms employed for classification tasks, commonly used ones include decision trees, support vector machines, k-nearest neighbors, and neural networks. In this series, we will delve into classical logistic regression, highlighting the inherent mathematical intricacies associated with this technique.

The following textbooks on this topic merit consultation. These books extend beyond logistic regression and covers a myriad of expansive and general machine learning topics.

Pattern Recognition and Machine Learning (Bishop)
Machine Learning: An Algorithmic Perspective (Marsland)
Probabilistic Machine Learning: An Introduction (Murphy)
Probabilistic Machine Learning: Advanced Topics (Murphy)

The following textbook focuses specifically on optimization methods, which are extensively used in machine learning. A solid understanding of mathematics is assumed.

Numerical Optimization (Nocedal, Wright)

Without further ado and as usual, let's begin with a few prerequisites to correctly understand the underlying concepts. Continue here.