Explainable AI with Logistic Regression

Logistic Function Formula

Logistic regression equation: p(x) = 1 / (1 + e^z)

Where:

• f(x) is the output value (probability) between 0 and 1.
• z is the linear combination of the features and coefficients, i.e., z = β₀ + β₁x₁ + β₂x₂ + ... + β_nx_n.
• β₀ is the intercept.
• β₁, β₂, ..., β_n are the coefficients of the independent variables (features).
• x₁, x₂, ..., x_n are the independent variables (features).
• e is Euler's number, the base of the natural logarithm, approximately equal to 2.71828.

Benefits of Logistic Models:

• Starting with logistic models offers simplicity, aiding in understanding and establishing a baseline for comparison with more complex models.

Causality Caution:

• Avoid making causal conclusions, as relationships may involve proxy variables. Beware of confirmation bias and aim for objective analysis.

Data Exploration:

• Investigate correlation (multicollinearity) and normality of features (distribution of features).
• Detect outliers using Z-score and Interquartile Range (IQR).
• Conduct Exploratory Data Analysis (EDA) including correlation and scatter plots.

Feature Engineering:

• Encode categorical features.
• Consider inverses of features and logical ratios for intuitive feature construction.
• Discretize continuous features to transform them into categorical features.

Finding Non-Linear Relationships:

• Utilize domain knowledge, data exploration (e.g., correlations), and non-linear models (e.g., Random Forest, XGBoost, DNN) to identify patterns.

Feature Selection:

• Use correlation-based selection and Variance Inflation Factor (VIF) to identify and remove redundant features.
• Eliminate irrelevant features lacking predictive power.
• Evaluate predictive power through feature groups and variable clustering methods.

Considerations in Feature Selection:

• Group features based on similarity.
• Use variable clustering methods like k-means or hierarchical clustering.
• Ensure data authenticity and availability in production.
• Validate feature predictiveness across time.
• Prioritize interpretable features and ensure legal and ethical compliance.

Hierarchical Clustering for Feature Selection:

• Employ hierarchical clustering to group features based on correlations while retaining interpretability.

Choosing Model:

• Standard Logistic Regression
• Regularized Logistic Regression (L1 or L2 regularization)
• Polynomial Logistic Regression
• Multiclass Logistic Regression

Choosing Cost Function:

• Standard Logistic Regression
• Regularized Logistic Regression (L1 or L2 regularization)
• Polynomial Logistic Regression
• Multiclass Logistic Regression

Train Model:

Cost Function:

• Binary logistic problems → binary cross + entropy loss (log loss)
• Multiple logistic problems → Categorical cross-entropy loss (softmax loss)

Regularization:

• L1 regularization (Lasso): L1 regularization tends to produce sparse solutions by pushing less important features' coefficients to zero. This can be beneficial in imbalanced datasets because it helps in feature selection, reducing the impact of potentially noisy or irrelevant features that might be more prevalent in the majority class.
• Combination of L1 and L2 regularization (ElasticNet): ElasticNet regularization combines both L1 and L2 penalties. It offers a balance between feature selection (from L1 regularization) and stability (from L2 regularization), which can be advantageous in handling imbalanced datasets. It provides the flexibility to adjust the trade-off between L1 and L2 penalties using the l1_ratio parameter.
• L2 regularization:By focusing on reducing the overall cost, might prioritize minimizing errors for the majority class, neglecting the minority class further.

Other:

• Adjust learning rate and number of epochs.
• Specify polynomial degree for polynomial models (e.g., 2 or 3).

Evaluate Model:

• Accuracy = ratio correctly prediction true positives and true negatives. Calculation: Accuracy = (True Positives + True Negatives) / (Total Samples). With imbalanced data accuracy can be misleading due to the class imbalance. The model can achieve high accuracy by simply predicting the majority class most of the time.
• Precision = true positives / false positives + true positives → use when costs of false positives are high
• Recall = true positives / true positives + false negatives → correctly predicting true positives to actual positives use when missing positives has high consequences.
• F1 score = harmonic mean precision and recall. Calculation: F1 = 2 * (Precision * Recall) / (Precision + Recall)
• AUROC score = Area under ROC. Interpertation: 0.5 indicates a random guess, the closer the AUC is to 1, the better the model's ability to distinguish between positive and negative cases.
• Cross-validation = Always use StratifiedKFold in case of imbalanced data. This balances your target variable into equally balanced groups for validation

Tune Model

Threshold Adjustment

• Lowering the threshold increases positive classifications, potentially boosting recall.
• However, it might also raise false positives.

Cost-Sensitive Learning

• Penalize misclassification of the minority class more heavily.
• Communicates the importance of correctly predicting the minority class to the model.

SMOTE

• Over and undersampling data reduces model explainability.

Visualizing Linear Regression Performance:

• ROC curve – Y axis (Sensitivity/recall/true positive rate) X axis (false positive rate / 1-specificity)
• Confusion matrix = Each row represents the instances in the actual class, and each column represents the instances in a predicted class.
• Scatter Plot: Model line overlaid on actual data points.
• Correlation Heatmap: Depict inter-feature relationships.
• Weight Plot: Illustrate feature importance through coefficients.
• Mean Effect Plot: Display mean feature effects systematically.
• Individual Effect Plot: Explore effects on a single instance.
• Trend Effect Plot: Explain trends between features and the target.
• Partial Dependence Plots: Show the relationship between a feature and the predicted outcome.
• SHAP Values: Provide a summary of feature contributions to model predictions.