Predicting Machine Failures - A look at Imbalanced Classification

June 2024, Pavlo Mysak

In this project, we explore the task of predicting machine failures using data from the following Kaggle competition: [https://www.kaggle.com/competitions/playground-series-s3e17/data].

The primary objective of this challenge is to predict instances of machine failures from a given dataset, highlighting the intricacies involved in imbalanced classification problems.

Imbalanced classification is a common scenario in various industry applications where the number of instances of one class significantly outnumbers the instances of the other class. This issue is prevalent in fields such as fraud detection, customer churn prediction, and medical diagnosis, where the minority class often represents the critical cases we aim to identify. In this context, machine failures are the minority class that we seek to predict accurately amidst a large volume of non-failure data.

For this analysis, we will utilize powerful Python libraries, including Pandas for data manipulation, Scikit-Learn for building and evaluating machine learning models, and IMB-Learn for handling imbalanced datasets through techniques like SMOTE (Synthetic Minority Over-sampling Technique).

Imbalanced classification problems present unique challenges. Achieving great model results can be difficult due to the skewed class distribution. Moreover, the performance metrics and model evaluation often depend heavily on the business use-case. In some scenarios, having a high precision (minimizing false positives) is paramount, such as in fraud detection, where incorrectly flagging a legitimate transaction can have adverse effects. Conversely, in cases like disease screening, recall (minimizing false negatives) might be more critical, as missing a true positive can have severe consequences.

Through this project, we aim to explore various models and techniques to address the imbalanced classification challenge in predicting machine failures. We will investigate the performance of different algorithms, both with and without the application of SMOTE, to understand their strengths and limitations in this context. By presenting a comprehensive analysis, we hope to provide valuable insights into effectively handling imbalanced datasets and making informed decisions based on the specific requirements of the use-case.

Background and Motivation

Machine failures can have significant impacts on businesses, including downtime, increased maintenance costs, and potential safety hazards. Predicting these failures before they occur can help in proactive maintenance and prevent costly disruptions. This problem is particularly relevant in industries such as manufacturing, aviation, and data centers, where machine reliability is critical.

Methodology

Data Preprocessing:

• Cleaning and preparing the data for analysis.
• Identifying distributions and relationships present in the data.

Feature Engineering:

• Creating and selecting features that improve model performance.

Handling Imbalanced Data:

• Implimenting Algorithm-Level appraoches such as cost-sensitive learning.
• Implimenting Data-Level approaches such as oversampling (specifically SMOTE).

Model Building:

• Developing several machine learning classifiers, including Decision Trees, AdaBoost, and HGBoost.
• Utilizing Stratified K-Fold Cross Validation to ensure that each fold has the same proportion of class labels as the original dataset.

Model Evaluation:

• Evaluating models using metrics such as F1 Score, ROC AUC, Precision and Recall.
• Comparing SMOTE-based models and Cost-Sensitive-Learning-based models.
• Discussing differences in evaluation with business-case in mind.
• Tuning a classifiers' decision thresholds in regard to the optimization of specific evaluation metrics.

Let's dig in!!

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# sklearn and imblearn
from sklearn.model_selection import train_test_split, StratifiedKFold, RandomizedSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder, FunctionTransformer
from sklearn.metrics import recall_score, precision_score, f1_score, roc_auc_score, confusion_matrix, ConfusionMatrixDisplay
from imblearn.pipeline import Pipeline
from imblearn.over_sampling import SMOTE

# sklearn models
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import AdaBoostClassifier, HistGradientBoostingClassifier

# Filter out warning messages
import warnings

dt = pd.read_csv('/Users/pavlomysak/playground-series-s3e17/train.csv')

dt.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 136429 entries, 0 to 136428
Data columns (total 14 columns):
 #   Column                   Non-Null Count   Dtype  
---  ------                   --------------   -----  
 0   id                       136429 non-null  int64  
 1   Product ID               136429 non-null  object 
 2   Type                     136429 non-null  object 
 3   Air temperature [K]      136429 non-null  float64
 4   Process temperature [K]  136429 non-null  float64
 5   Rotational speed [rpm]   136429 non-null  int64  
 6   Torque [Nm]              136429 non-null  float64
 7   Tool wear [min]          136429 non-null  int64  
 8   Machine failure          136429 non-null  int64  
 9   TWF                      136429 non-null  int64  
 10  HDF                      136429 non-null  int64  
 11  PWF                      136429 non-null  int64  
 12  OSF                      136429 non-null  int64  
 13  RNF                      136429 non-null  int64  
dtypes: float64(3), int64(9), object(2)
memory usage: 14.6+ MB

dt.head()

	id	Product ID	Type	Air temperature [K]	Process temperature [K]	Rotational speed [rpm]	Torque [Nm]	Tool wear [min]	Machine failure	TWF	HDF	PWF	OSF	RNF
0	0	L50096	L	300.6	309.6	1596	36.1	140	0	0	0	0	0	0
1	1	M20343	M	302.6	312.1	1759	29.1	200	0	0	0	0	0	0
2	2	L49454	L	299.3	308.5	1805	26.5	25	0	0	0	0	0	0
3	3	L53355	L	301.0	310.9	1524	44.3	197	0	0	0	0	0	0
4	4	M24050	M	298.0	309.0	1641	35.4	34	0	0	0	0	0	0

# A look at the class imbalance
dt['Machine failure'].value_counts(normalize=True)

0    0.984256
1    0.015744
Name: Machine failure, dtype: float64

Notes on Next Steps:

We see that we have 14 features with 136,429 observations in the original data. Only 1.6% of observations depict a machine failure (the positive case). When creating our training and testing sets, we should split the data in a stratified fashion to ensure that we have a comparable number of positive cases in both of the sets.

# Creating our Train/Test splits with stratification
X_train, X_test, y_train, y_test = train_test_split(dt.drop('Machine failure', axis = 1), 
                                                    dt['Machine failure'],
                                                    test_size=0.3, 
                                                    stratify=(dt['Machine failure']))

Numeric Data Overview

# Creating histograms of our data
X_train.hist(figsize=(14,10), bins = 100)

array([[<Axes: title={'center': 'id'}>,
        <Axes: title={'center': 'Air temperature [K]'}>,
        <Axes: title={'center': 'Process temperature [K]'}>],
       [<Axes: title={'center': 'Rotational speed [rpm]'}>,
        <Axes: title={'center': 'Torque [Nm]'}>,
        <Axes: title={'center': 'Tool wear [min]'}>],
       [<Axes: title={'center': 'TWF'}>, <Axes: title={'center': 'HDF'}>,
        <Axes: title={'center': 'PWF'}>],
       [<Axes: title={'center': 'OSF'}>, <Axes: title={'center': 'RNF'}>,
        <Axes: >]], dtype=object)

Histogram Observations:

• Air Temp:

  • Sort of Bimodal. Normalization should be done.
  • There are some odd gaps in the data in a few spots.

• Process Temp:

  • Bimodal with pronounced and frequent gaps in the distribution.

• Rotational Speed:

  • Minimally Right Skewed

• Torque:

  • Guassan with mu around 40

• Tool Wear:

  • A mess.
  • Kind of uniform.

• TWF, HDF, PWF, ODF, RNF:

  • Binary
  • Mostly 0

# Creating a scatterplot matrix of our numeric data
pd.plotting.scatter_matrix(X_train[X_train.columns[1:6]], figsize=(14,14))

array([[<Axes: xlabel='Air temperature [K]', ylabel='Air temperature [K]'>,
        <Axes: xlabel='Process temperature [K]', ylabel='Air temperature [K]'>,
        <Axes: xlabel='Rotational speed [rpm]', ylabel='Air temperature [K]'>],
       [<Axes: xlabel='Air temperature [K]', ylabel='Process temperature [K]'>,
        <Axes: xlabel='Process temperature [K]', ylabel='Process temperature [K]'>,
        <Axes: xlabel='Rotational speed [rpm]', ylabel='Process temperature [K]'>],
       [<Axes: xlabel='Air temperature [K]', ylabel='Rotational speed [rpm]'>,
        <Axes: xlabel='Process temperature [K]', ylabel='Rotational speed [rpm]'>,
        <Axes: xlabel='Rotational speed [rpm]', ylabel='Rotational speed [rpm]'>]],
      dtype=object)

Scatterplot Observations:

• Positive Linear Relationship between Process Temp and Air Temp
• Absolitely no relationship between either Temps and Rotational Speed

Note for later: We will take a look at the interaction between Air Temp and Process Temp as a potential feature for our models.

Categorical Data Overview

# How many unique values do we have for Type?
X_train['Type'].value_counts()

L    66681
M    22532
H     6287
Name: Type, dtype: int64

# How many unique values do we have for Product ID?
X_train['Product ID'].value_counts()

L53257    100
L53271     97
L49056     93
L48892     89
L54275     87
         ... 
M16761      1
H39399      1
M15516      1
M15190      1
H34966      1
Name: Product ID, Length: 9928, dtype: int64

Notes on Categorical Variables:

Because Type only has 3 unique and meaningful values, it may be wise to leave it in for training. Product ID on the other hand would not be a good candidate as a feature for training because it has 9,928 unique values. Given that we'd have to convert these to Dummy Variables, it would be too computationally expensive to train these models on such a large feature set. It probably wouldn't improve our model either. An argument could be made to feature engineer an 'Other' category of product ID's for ID's that have a frequency lower than some threshold, however, the Type feature seems to already capture a portion of the Product ID column and should be sufficient for this project.

Mining into Binary Variable Relationships

We have 5 binary features in addition to our target variable. Because there was no data dictionary included with this data, there's no way to note exaclty what these binary indicators represent, however, we can estimate their importance to our target variable with the use of confusion matrices.

# Creating a Confusion Matrix for all Binary Columns with "OR" Operator
ConfusionMatrixDisplay(confusion_matrix(y_train, pd.DataFrame(np.where((X_train['TWF']==1) | (X_train['HDF']==1) | (X_train['PWF']==1) | (X_train['OSF']==1) | (X_train['RNF']==1), 
         1, 
         0)), normalize = 'true')).plot()
plt.title('All Binary "OR"')
plt.show()

# Creating a vector of our Binary Columns
bina_cols = ['TWF', 'HDF', 'PWF', 'OSF', 'RNF']

fig, axes = plt.subplots(nrows=1, ncols=len(bina_cols), figsize=(15, 3))

for i, col in enumerate(bina_cols):
    ax = axes[i]
    ConfusionMatrixDisplay(confusion_matrix(y_train, X_train[col], normalize='true'), display_labels=[0, 1]).plot(ax=ax)
    ax.set_title(f'Confusion Matrix for {col}')

plt.tight_layout()
plt.show()

Notes on Binary Variables:

We observed that a new binary feature, indicating whether any of the original binary indicators are positive, could be a valuable predictor for our models.

We will experiment with two approaches: one where we replace all original binary features with this new indicator, and another where we include both the new indicator and the original binary features in the training data. We will include a toggle in the function that produces this column to help us determine the best configuration for our models.

Creating Data Preprocessing Pipelines

# Defining a function to create the Binary column
def bina_col(X_dt):
    X_dt['binary_agg'] = np.where(
        (X_dt['TWF'] == 1) | 
        (X_dt['HDF'] == 1) | 
        (X_dt['PWF'] == 1) | 
        (X_dt['OSF'] == 1) | 
        (X_dt['RNF'] == 1), 
        1, 0
    )
    return X_dt
# Transforming this function to be usable in a pipeline
bina_col_tr = FunctionTransformer(bina_col)


# Defining a function to remove unnecessary columns (including a bypass for binary indicators)
def col_remover(X_dt, bypass=False):
    if not bypass:
        return X_dt.drop(['id', 'Product ID', 'TWF', 'HDF', 'PWF', 'OSF', 'RNF'], axis=1)
    else:
        return X_dt.drop(['id', 'Product ID'], axis=1)

# Transforming this function to be usable in a pipeline
col_rem_tr = FunctionTransformer(col_remover, kw_args={'bypass': False})
    
    
# Defining a function to create the temperature ratio
def temp_ratio(X_dt):
    X_dt['Temp_Ratio'] = X_dt['Air temperature [K]']/X_dt['Process temperature [K]']
    return X_dt

# Transforming this function to be usable in a pipeline
temp_ratio_tr = FunctionTransformer(temp_ratio)

    
# Creating our categorical and numeric preprocessing pipelines
cat_prepr = make_pipeline(OneHotEncoder())
num_prepr = make_pipeline(StandardScaler())

preprocessing_pipeline = ColumnTransformer(
    transformers=[
        ('cat_tr', cat_prepr, col_remover(X_train).select_dtypes('object').columns),  
        ('num_tr', num_prepr, col_remover(X_train).select_dtypes(exclude = 'object').columns) 
    ],
    remainder = 'passthrough'
)

mast_pipeline = make_pipeline(bina_col_tr, 
                         temp_ratio_tr, 
                         col_rem_tr, 
                         preprocessing_pipeline)




prepro_dat = mast_pipeline.fit_transform(X_train)
test_prepro = mast_pipeline.fit_transform(X_test)

pd.DataFrame(prepro_dat).sample(10)

	0	1	2	3	4	5	6	7	8	9
32054	0.0	1.0	0.0	-1.323080	-1.184903	0.380656	-0.699143	0.965051	0.0	0.964645
44938	0.0	1.0	0.0	-2.181489	-2.409741	-0.643294	1.087025	0.167574	0.0	0.964775
56312	1.0	0.0	0.0	-1.430381	-1.112854	-0.217850	-0.076334	0.230122	0.0	0.963684
49132	0.0	1.0	0.0	-1.430381	-1.112854	-0.477443	0.663985	1.043235	0.0	0.963684
77127	0.0	1.0	0.0	-0.303719	-1.040804	0.423922	0.111683	1.668706	0.0	0.970178
414	0.0	1.0	0.0	-0.625622	-0.320312	0.712358	-0.041081	0.871230	0.0	0.965105
4199	0.0	1.0	0.0	1.091195	-0.176213	-0.535130	0.428963	-1.364831	0.0	0.974814
79818	0.0	1.0	0.0	-0.893875	-1.545150	0.164329	-0.828405	-0.567355	0.0	0.968811
75758	0.0	1.0	0.0	-0.947526	-1.329002	0.863787	-0.887161	-0.614265	0.0	0.967543
10160	0.0	1.0	0.0	-0.518321	0.111984	-1.068738	0.816749	-0.833180	0.0	0.963883

Running our Classification Models

We will run 6 total models:

• Decision Tree with Cost Sensitive Learning
• Adaptive Boost Decision Trees with Cost Sensitive Learning

• Hist Gradient Boosted Trees with Cost Sensitive Learning

• Decision Tree with Synthetic Minority Oversampling Technique (SMOTE)

• Adaptive Boost Decision Trees with Synthetic Minority Oversampling Technique (SMOTE)
• Hist Gradient Boosted Trees with Synthetic Minority Oversampling Technique (SMOTE)

Why use Decision Trees?

Basic Structure

Decision Trees are a type of model that splits data into branches to make decisions based on the features. At each node, the algorithm selects the feature that best separates the data into distinct classes.

Suitability for Imbalanced Learning

Decision Trees are intuitive and easy to understand. They can handle imbalanced data by assigning different weights to classes, which helps them focus more on the minority class. This makes them suitable for scenarios where certain outcomes (like machine failures) are rare but critical to identify.

Why use AdaBoost Decision Trees?

Basic Structure

AdaBoost (Adaptive Boosting) is an ensemble learning technique that combines multiple weak classifiers, in our case Decision Trees, to create a strong classifier. It adjusts the weights of misclassified instances so that subsequent classifiers focus more on those cases.

Suitability for Imbalanced Learning

AdaBoost is effective for imbalanced datasets because it iteratively adjusts the weights of the instances, giving more attention to the minority class. By focusing on the harder-to-classify instances, AdaBoost can improve the model's ability to detect rare events like machine failures.

Why use HGBoost?

Basic Structure

Hist Gradient Boosted Trees (HGBoost) is an ensemble technique that builds trees sequentially, each one correcting the errors of the previous ones. HGBoost uses histogram-based methods to bin continuous features, making the algorithm faster and more efficient than a typical Gradient Boosting Trees Algorithm.

Suitability for Imbalanced Learning

HGBoost is robust to imbalanced datasets due to its iterative approach in refining predictions. It can use customized loss functions and weight adjustments to handle the imbalance, making it a powerful tool for predicting rare events.

The Importance of Using Comprehensive Evaluation Metrics in Imbalanced Classification

When dealing with machine learning models in the context of imbalanced datasets, relying on accuracy as an evaluation metric can be misleading and insufficient. Here’s why we cannot just use accuracy, sklearn's default evaluation metric, and why we need to use other metrics such as F1 score, ROC AUC, precision, and recall:

Understanding Accuracy

Accuracy is defined as the ratio of correctly predicted instances to the total instances. While it provides a quick snapshot of overall model performance, it does not account for the distribution of class labels. In an imbalanced classification problem, where the majority class vastly outnumbers the minority class, a high accuracy can be achieved by simply predicting the majority class most of the time.

Limitations of Accuracy in Imbalanced Datasets

Consider a scenario where 95% of the instances belong to the majority class (non-failure) and only 5% belong to the minority class (failure). A model that predicts all instances as the majority class will have an accuracy of 95%, but it completely fails to identify any of the minority class instances. This high accuracy gives a false sense of model effectiveness.

Importance of Other Metrics

Precision and Recall:

Precision measures the proportion of true positive predictions (correctly predicted failures) out of all positive predictions (all predicted failures). High precision means that the model has a low false positive rate.

Recall (or Sensitivity) measures the proportion of true positive predictions out of all actual positives (all actual failures). High recall means that the model captures most of the actual failures. In machine failure prediction, missing an actual failure (low recall) can be more costly than falsely predicting a failure (low precision). Therefore, recall might be prioritized in critical applications where preventing failures is crucial.

The F1 score is the harmonic mean of precision and recall, providing a single metric that balances both concerns. It is particularly useful when we need to balance the trade-off between precision and recall. A balanced F1 score ensures that the model performs well in identifying failures without generating too many false alarms, making it suitable for operational decision-making.

The ROC AUC (Receiver Operating Characteristic - Area Under Curve) metric evaluates the model's ability to distinguish between the positive and negative classes across various threshold settings. It provides an aggregate measure of performance across all classification thresholds. A high ROC AUC indicates that the model is good at distinguishing between machine failures and non-failures, which is crucial for reliable prediction in real-world applications.

Cost Sensitive Learning

Why use Cost-Sensitive Learning?

Basic Structure

Cost-Sensitive Learning involves modifying the learning process to take into account the cost associated with misclassifying different classes. This is usually done by assigning higher weights to the minority class during training.

Suitability for Imbalanced Learning

Cost-Sensitive Learning is particularly useful for imbalanced data as it directly addresses the imbalance by making the model more sensitive to the minority class. By increasing the penalty for misclassifying minority class instances, the model is incentivized to improve its performance on these critical cases.

Decision Trees w Cost Sensitive Learning

DT = DecisionTreeClassifier(criterion='log_loss')

# Configuring Parameter Grid for Hyperparameter Tuning
DT_prm = [{'min_samples_split':[2, 3, 4], 
                  'class_weight':[{0:1, 1:9}, {0:1, 1:10}, {0:1, 1:11}, {0:1, 1:12}, {0:1, 1:13}], 
                  'min_weight_fraction_leaf':[0.09, 0.1, 0.12],
                  'max_depth': [3, 5, 7, 10],
                  'min_samples_leaf': [1, 5, 10, 20]}]

# Initializing Stratified K-Fold Cross Validation
skf = StratifiedKFold(n_splits=10)

DT_cl = RandomizedSearchCV(DT, 
                        DT_prm, 
                        cv=skf, 
                        scoring='f1')

# fitting the model
DT_cl.fit(prepro_dat, y_train)
print('done!')

done!

Adaptive Boost Decision Trees w Cost Sensitive Learning

ada_prm = [{'n_estimators': [250, 300, 350, 400]}]

ada = AdaBoostClassifier(
    DecisionTreeClassifier(max_depth = 1, criterion = 'log_loss'), 
    algorithm="SAMME.R",
    learning_rate = 0.2
)

bdt_cl = RandomizedSearchCV(ada, 
                   ada_prm, 
                   cv=skf,
                   scoring='f1')


bdt_cl.fit(prepro_dat, y_train)
print('done!')

/Users/pavlomysak/opt/anaconda3/lib/python3.9/site-packages/sklearn/model_selection/_search.py:307: UserWarning: The total space of parameters 4 is smaller than n_iter=10. Running 4 iterations. For exhaustive searches, use GridSearchCV.
  warnings.warn(

done!

Hist Gradient Boost Trees w Cost Sensitive Learning

HGB_prm = [{'learning_rate': [0.1, 0.01],
       'class_weight':[{0:1, 1:9}, {0:1, 1:10}, {0:1, 1:11}, {0:1, 1:12}, {0:1, 1:8}],
       'max_depth':[3, 5, 7, 9],
       'min_samples_leaf':[40, 50]}]

HGB = HistGradientBoostingClassifier()

HGB_cl = RandomizedSearchCV(HGB, 
                   HGB_prm, 
                   cv=skf,
                   scoring='f1')


HGB_cl.fit(prepro_dat, y_train)
print('done!')

done!

Evaluating on Witheld Data (Test Data)

fig, axes = plt.subplots(1, 3, figsize=(15, 5))

# Decision Tree
ConfusionMatrixDisplay(confusion_matrix(y_test, DT_cl.predict(test_prepro), normalize='true')).plot(ax=axes[0])
axes[0].set_title('Decision Tree')

# Adaptive Boost DT
ConfusionMatrixDisplay(confusion_matrix(y_test, bdt_cl.predict(test_prepro), normalize='true')).plot(ax=axes[1])
axes[1].set_title('Adaptive Boost DT')

# HGBoost
ConfusionMatrixDisplay(confusion_matrix(y_test, HGB_cl.predict(test_prepro), normalize='true')).plot(ax=axes[2])
axes[2].set_title('HGBoost')

# Adjust layout
plt.tight_layout()
plt.show()

pd.DataFrame(
    {'f1': [f1_score(y_test, DT_cl.predict(test_prepro)),
            f1_score(y_test, bdt_cl.predict(test_prepro)),
            f1_score(y_test, HGB_cl.predict(test_prepro))
           ],
     'roc': [roc_auc_score(y_test, DT_cl.predict(test_prepro)),
            roc_auc_score(y_test, bdt_cl.predict(test_prepro)),
            roc_auc_score(y_test, HGB_cl.predict(test_prepro))
            ],
     'precision': [precision_score(y_test, DT_cl.predict(test_prepro)),
                  precision_score(y_test, bdt_cl.predict(test_prepro)),
                  precision_score(y_test, HGB_cl.predict(test_prepro))
                  ],
     'recall': [recall_score(y_test, DT_cl.predict(test_prepro)),
               recall_score(y_test, bdt_cl.predict(test_prepro)),
               recall_score(y_test, HGB_cl.predict(test_prepro))]
    }, index = ['Decision Tree', 'Ada Boost', 'HGBoost']
)

	f1	roc	precision	recall
Decision Tree	0.807443	0.886268	0.842905	0.774845
Ada Boost	0.814751	0.868205	0.909962	0.737578
HGBoost	0.812242	0.880293	0.869027	0.762422

Synthetic Minority Oversampling Technique

Why use SMOTE?

Basic Structure

The Synthetic Minority Oversampling Technique (SMOTE) generates synthetic samples for the minority class by interpolating between existing minority instances. This balances the class distribution by increasing the number of minority class instances. This, however, should be used with caution for two reasons: First, it's feeding synthetic data into your model, which may introduce unnecessary noise. Second, it can be computationally expensive depending on the size of your dataset.

Suitability for Imbalanced Learning

SMOTE is effective for imbalanced datasets because it directly addresses the class imbalance by creating more examples of the minority class. This helps models learn the characteristics of the minority class better, leading to improved detection of rare events.

Decision Tree w SMOTE

smt = SMOTE(random_state=42)

DT_SMT_PIPL = Pipeline([('smt', smt),
                         ('dt', DT)])

SMDT_param_grid = [{'dt__min_samples_split':[2, 3, 4],  
                  'dt__min_weight_fraction_leaf':[0.09, 0.1, 0.12],
                  'dt__max_depth': [3, 5, 7, 10],
                  'dt__min_samples_leaf': [1, 5, 10, 20]}]

SM_DT_srch = RandomizedSearchCV(DT_SMT_PIPL, 
                        SMDT_param_grid, 
                        cv=skf, 
                        scoring='f1')

# fitting the model
SM_DT_srch.fit(prepro_dat, y_train)
print('done!')

done!

Adaptive Boost Decision Trees w SMOTE

ada_SMT_PIPL = Pipeline([('smt', smt),
                         ('ada', ada)])

SMT_ada_prm = [{'ada__n_estimators': [250, 300, 350, 400]}]

SMT_bdt = RandomizedSearchCV(ada_SMT_PIPL, 
                   SMT_ada_prm, 
                   cv=skf,
                   scoring='f1')


# fitting the model
SMT_bdt.fit(prepro_dat, y_train)
print('done!')

/Users/pavlomysak/opt/anaconda3/lib/python3.9/site-packages/sklearn/model_selection/_search.py:307: UserWarning: The total space of parameters 4 is smaller than n_iter=10. Running 4 iterations. For exhaustive searches, use GridSearchCV.
  warnings.warn(

done!

Hist Gradient Boosted Trees w SMOTE

HGB_SMT_PIPL = Pipeline([('smt', smt),
                         ('HGB', HGB)])

SMT_HGB_prm = [{'HGB__learning_rate': [0.1, 0.01],
       'HGB__max_depth':[3, 5, 7, 9],
       'HGB__min_samples_leaf':[40, 50]}]


SMT_HGB_cl = RandomizedSearchCV(HGB_SMT_PIPL, 
                   SMT_HGB_prm, 
                   cv=skf,
                   scoring='f1')


SMT_HGB_cl.fit(prepro_dat, y_train)
print('done!')

done!

Evaluating on Witheld Data (Test Data)

fig, axes = plt.subplots(1, 3, figsize=(15, 5))

ConfusionMatrixDisplay(confusion_matrix(y_test, SM_DT_srch.predict(test_prepro),normalize = 'true')).plot(ax=axes[0])
axes[0].set_title('Decision Tree')

ConfusionMatrixDisplay(confusion_matrix(y_test, SMT_bdt.predict(test_prepro),normalize = 'true')).plot(ax=axes[1])
axes[1].set_title('Adaptive Boost DT')

ConfusionMatrixDisplay(confusion_matrix(y_test, SMT_HGB_cl.predict(test_prepro),normalize = 'true')).plot(ax=axes[2])
axes[2].set_title('HGBoost')

plt.tight_layout()
plt.show()

pd.DataFrame(
    {'f1': [f1_score(y_test, SM_DT_srch.predict(test_prepro)),
            f1_score(y_test, SMT_bdt.predict(test_prepro)),
            f1_score(y_test, SMT_HGB_cl.predict(test_prepro))
           ],
     'roc': [roc_auc_score(y_test, SM_DT_srch.predict(test_prepro)),
            roc_auc_score(y_test, SMT_bdt.predict(test_prepro)),
            roc_auc_score(y_test, SMT_HGB_cl.predict(test_prepro))
            ],
     'precision': [precision_score(y_test, SM_DT_srch.predict(test_prepro)),
                  precision_score(y_test, SMT_bdt.predict(test_prepro)),
                  precision_score(y_test, SMT_HGB_cl.predict(test_prepro))
                  ],
     'recall': [recall_score(y_test, SM_DT_srch.predict(test_prepro)),
               recall_score(y_test, SMT_bdt.predict(test_prepro)),
               recall_score(y_test, SMT_HGB_cl.predict(test_prepro))]
    }, index = ['Decision Tree', 'Ada Boost', 'HGBoost']
)

	f1	roc	precision	recall
Decision Tree	0.807443	0.886268	0.842905	0.774845
Ada Boost	0.457215	0.908305	0.313218	0.846273
HGBoost	0.229669	0.890562	0.132235	0.872671

Comparing All Models (Cost-Sensitive & Oversampled)

pd.DataFrame(
    {'f1': [f1_score(y_test, DT_cl.predict(test_prepro)),
            f1_score(y_test, SM_DT_srch.predict(test_prepro)),
            f1_score(y_test, bdt_cl.predict(test_prepro)),
            f1_score(y_test, SMT_bdt.predict(test_prepro)),
            f1_score(y_test, HGB_cl.predict(test_prepro)),
            f1_score(y_test, SMT_HGB_cl.predict(test_prepro))
           ],
     'roc': [roc_auc_score(y_test, DT_cl.predict(test_prepro)),
             roc_auc_score(y_test, SM_DT_srch.predict(test_prepro)),
             roc_auc_score(y_test, bdt_cl.predict(test_prepro)),
            roc_auc_score(y_test, SMT_bdt.predict(test_prepro)),
             roc_auc_score(y_test, HGB_cl.predict(test_prepro)),
            roc_auc_score(y_test, SMT_HGB_cl.predict(test_prepro))
            ],
     'precision': [precision_score(y_test, DT_cl.predict(test_prepro)),
                   precision_score(y_test, SM_DT_srch.predict(test_prepro)),
                   precision_score(y_test, bdt_cl.predict(test_prepro)),
                   precision_score(y_test, SMT_bdt.predict(test_prepro)),
                   precision_score(y_test, HGB_cl.predict(test_prepro)),
                  precision_score(y_test, SMT_HGB_cl.predict(test_prepro))
                  ],
     'recall': [recall_score(y_test, DT_cl.predict(test_prepro)),
                recall_score(y_test, SM_DT_srch.predict(test_prepro)),
                recall_score(y_test, bdt_cl.predict(test_prepro)),
                recall_score(y_test, SMT_bdt.predict(test_prepro)),
                recall_score(y_test, HGB_cl.predict(test_prepro)),
                recall_score(y_test, SMT_HGB_cl.predict(test_prepro))]
    }, index = ['Decision Tree', 'SMOTE Decision Tree', 'AdaBoost', 'SMOTE AdaBoost','HGBoost', 'SMOTE HGBoost']
)

	f1	roc	precision	recall
Decision Tree	0.807443	0.886268	0.842905	0.774845
SMOTE Decision Tree	0.807443	0.886268	0.842905	0.774845
AdaBoost	0.814751	0.868205	0.909962	0.737578
SMOTE AdaBoost	0.457215	0.908305	0.313218	0.846273
HGBoost	0.812242	0.880293	0.869027	0.762422
SMOTE HGBoost	0.229669	0.890562	0.132235	0.872671

# Models without SMOTE
models_no_smote = [DT_cl, bdt_cl, HGB_cl]
titles_no_smote = ['Decision Tree', 'AdaBoost', 'HGBoost']

# Models with SMOTE
models_smote = [SM_DT_srch, SMT_bdt, SMT_HGB_cl]
titles_smote = ['Decision Tree - SMOTE', 'AdaBoost - SMOTE', 'HGBoost - SMOTE']

# Set up a 2x3 grid of plots
fig, axes = plt.subplots(2, 3, figsize=(15, 10))

# Plot confusion matrices for models without SMOTE
for i, (model, title) in enumerate(zip(models_no_smote, titles_no_smote)):
    cm = confusion_matrix(y_test, model.predict(test_prepro), normalize='true')
    disp = ConfusionMatrixDisplay(confusion_matrix=cm)
    disp.plot(ax=axes[0, i], colorbar=False)
    axes[0, i].set_title(title)

# Plot confusion matrices for models with SMOTE
for i, (model, title) in enumerate(zip(models_smote, titles_smote)):
    cm = confusion_matrix(y_test, model.predict(test_prepro), normalize='true')
    disp = ConfusionMatrixDisplay(confusion_matrix=cm)
    disp.plot(ax=axes[1, i], colorbar=False)
    axes[1, i].set_title(title)

# Adjust layout to avoid overlap
plt.tight_layout()
plt.show()

Performance Evaluation

From the confusion matrices, we can see that each model had varying success in correctly predicting the machine failures (minority class):

• Decision Tree without SMOTE: Achieved an F1 score of 0.8074, with a precision of 0.8429 and recall of 0.7748. This model showed a good balance between precision and recall.
• Decision Tree with SMOTE: Showed identical performance metrics to the Decision Tree without SMOTE, indicating that SMOTE did not improve this particular model.
• AdaBoost without SMOTE: Achieved the highest F1 score among the models at 0.8148, with a high precision of 0.9099 but lower recall at 0.7376. This suggests AdaBoost is better at avoiding false positives but may miss some true positives.
• AdaBoost with SMOTE: Demonstrated a significant drop in F1 score to 0.4572, with a much lower precision (0.3132) but high recall (0.8463). This indicates that while AdaBoost with SMOTE is good at identifying true positives, it also has a high rate of false positives.
• HGBoost without SMOTE: Had a strong performance with an F1 score of 0.8122, precision of 0.8690, and recall of 0.7624. This model also maintained a good balance between precision and recall.
• HGBoost with SMOTE: Had the lowest F1 score of 0.2297, indicating that applying SMOTE did not benefit this model and resulted in a very high rate of false positives (precision of 0.1322) despite having high recall (0.8727).

Business Use-Case Considerations

Different business cases might prioritize different evaluation metrics based on the specific needs and consequences of misclassification:

When Precision is More Important: In cases where false positives are costly or highly disruptive (e.g., fraud detection, where flagging a legitimate transaction as fraud can cause inconvenience and loss of trust), models like AdaBoost without SMOTE (high precision of 0.9099) would be preferred. High precision ensures that when the model predicts a failure, it is likely to be correct.

When Recall is More Important: In scenarios where missing a positive instance is highly undesirable (e.g., medical diagnoses, where failing to identify a disease could have severe consequences), models with higher recall should be chosen. For instance, AdaBoost with SMOTE has a high recall of 0.8463, meaning it is effective at capturing most of the actual machine failures.

Decision Threshold Adjustments

To further tailor to various business use-cases, we can manually tune the decision thresholds. Decision thresholds in classification problems determine how predicted probabilities are translated into class labels. The default threshold is usually 0.5, where predictions above 0.5 are classified as positive (class 1) and below as negative (class 0). Setting the decision threshold manually and optimizing for different metrics like precision, recall or F1-score can be looked at as a final 'tuning' of our predictions. Below is a look at this process using a custom function that finds the optinal threshold for optimizing different metric scores.

def optimal_threshold(model, X_test, y_test, metric):
    """
    Finds the optimal threshold for a given model to maximize the F1 score.

    Parameters:
    model: Trained classifier
    X_test: Features of the test set
    y_test: True labels of the test set

    Returns:
    best_threshold: Threshold that maximizes the F1 score
    metrics: Dictionary containing the F1 score, accuracy, precision, recall, and ROC AUC score
    """
    y_pred_proba = model.predict_proba(X_test)[:, 1]
    
    # Initialize variables to store the best F1 score and corresponding threshold
    best_threshold = 0.0
    best_score = 0.0
    
    # Generate possible thresholds
    thresholds = np.arange(0.0, 1.0, 0.01)
    
    if metric == 'f1':
        # Iterate through thresholds to find the best one
        for threshold in thresholds:
            y_pred = (y_pred_proba >= threshold).astype(int)
            current_score = f1_score(y_test, y_pred)
            if current_score > best_score:
                best_score = current_score
                best_threshold = threshold
            
    if metric == 'precision':
        # Iterate through thresholds to find the best one
        for threshold in thresholds:
            y_pred = (y_pred_proba >= threshold).astype(int)
            current_score = precision_score(y_test, y_pred)
            if current_score > best_score:
                best_score = current_score
                best_threshold = threshold
        
    if metric == 'recall':
        # Iterate through thresholds to find the best one
        for threshold in thresholds:
            y_pred = (y_pred_proba >= threshold).astype(int)
            current_score = recall_score(y_test, y_pred)
            if current_score > best_score:
                best_score = current_score
                best_threshold = threshold
        
        
    # Predict with the optimal threshold
    y_pred_optimal = (y_pred_proba >= best_threshold).astype(int)
    
    # Calculate other metrics with the optimal threshold
    metrics = {
        'Threshold': best_threshold,
        'F1 Score': f1_score(y_test, y_pred_optimal),
        'Precision': precision_score(y_test, y_pred_optimal),
        'Recall': recall_score(y_test, y_pred_optimal)
    }
    
    return metrics


all_models = models_no_smote + models_smote
all_titles = titles_no_smote + titles_smote

# filter out warnings
warnings.filterwarnings('ignore')


# Create a DataFrame to store the results
results_precision = pd.DataFrame(columns=['Model', 'Threshold', 'F1 Score', 'Precision', 'Recall'])

# Iterate through all models and find the optimal thresholds to maximize precision
for model, title in zip(all_models, all_titles):
    metrics = optimal_threshold(model, test_prepro, y_test, 'precision')
    metrics['Model'] = title
    results_precision = results_precision.append(metrics, ignore_index=True)

# Set the Model column as the index
results_precision.set_index('Model', inplace=True)

results_precision

	Threshold	F1 Score	Precision	Recall
Model
Decision Tree	0.16	0.807443	0.842905	0.774845
AdaBoost	0.53	0.060241	1.000000	0.031056
HGBoost	0.68	0.659898	0.953079	0.504658
Decision Tree - SMOTE	0.40	0.807443	0.842905	0.774845
AdaBoost - SMOTE	0.55	0.042553	1.000000	0.021739
HGBoost - SMOTE	0.99	0.814750	0.937374	0.720497

# Create a DataFrame to store the results
results_f1 = pd.DataFrame(columns=['Model', 'Threshold', 'F1 Score', 'Precision', 'Recall'])

# Iterate through all models and find the optimal thresholds to maximize the f1 score
for model, title in zip(all_models, all_titles):
    metrics = optimal_threshold(model, test_prepro, y_test, 'f1')
    metrics['Model'] = title
    results_f1 = results_f1.append(metrics, ignore_index=True)

# Set the Model column as the index
results_f1.set_index('Model', inplace=True)

results_f1

	Threshold	F1 Score	Precision	Recall
Model
Decision Tree	0.16	0.807443	0.842905	0.774845
AdaBoost	0.50	0.814751	0.909962	0.737578
HGBoost	0.53	0.824129	0.909944	0.753106
Decision Tree - SMOTE	0.40	0.807443	0.842905	0.774845
AdaBoost - SMOTE	0.52	0.812925	0.898496	0.742236
HGBoost - SMOTE	0.98	0.823932	0.916350	0.748447

Conclusion

In this project aimed at predicting machine failures, we evaluated six different models using three different algorithms (Decision Tree, AdaBoost, and Hist Gradient Boosting) with and without the application of the Synthetic Minority Oversampling Technique (SMOTE). Additionally, we tuned decision thresholds with different business use-cases in mind. The goal was to handle the inherent class imbalance in our dataset and determine which model configurations provided the best performance.

Ultimately, the choice of model and technique depends on the specific business needs and the acceptable trade-offs between precision and recall. Our analysis indicates that while certain models perform well overall, the application of SMOTE had mixed results, improving recall at the expense of precision in some cases. Decision Trees and HGBoost models without SMOTE provided balanced performance, whereas AdaBoost models showed clear differences in performance metrics depending on whether SMOTE was applied. Threshold adjustments to optimize the F1-Score yielded similar results to the default decision threshold value, while optimizing the threshold to maximize precision either had a similar score to the default or tanked the F1-Score. This highlights the importance of carefully considering the business context and the specific requirements of the task when selecting a model for imbalanced classification problems.