Disadvantaged Communities Classification project

CSE 5120 – DACs Classification

Author: Omar Ribadu
Course: Introduction to Artificial Intelligence (CSE 5120)
Instructor: California State University, San Bernardino

Overview

This project implements two machine learning classifiers —

• Support Vector Machine (SVM)
• Random Forest (RF)

to predict the CES 4.0 Percentile Range (a measure of environmental and socio-economic disadvantage) for California’s Disadvantaged Communities (DACs) dataset.

The workflow is divided into two main files:

• DACs_classification.py: builds and trains both models.
• evaluation.py: loads and evaluates the saved models on a test dataset.

Part 1 – DACs_classification.py

import pandas as pd
import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier
import pickle

Explanation:
This imports all the required libraries:

• pandas, numpy → data manipulation and numerical operations.
• StandardScaler → feature standardization.
• SimpleImputer → handles missing values by replacing them with means.
• train_test_split → splits the data into training and testing sets.
• SVC and RandomForestClassifier → the two main ML models.
• pickle → saves and loads trained models for reuse.

# Step 1: Load the dataset
df = pd.read_csv('disadvantaged_communities.csv')  # Replace with actual dataset filename

# Debugging: Print column names and first few rows to verify the structure
print("Column names in the dataset:", df.columns)
print("First few rows of the dataset:")
print(df.head())

Explanation:
Loads the DACs dataset into a pandas DataFrame.
Prints column names and the first few rows to confirm data integrity and structure.

# Step 2: Set the target column
target_column = 'CES 4.0 Percentile Range'  # Update if needed

# Drop non-numeric columns or encode them if necessary
non_numeric_columns = df.select_dtypes(include='object').columns.tolist()
print(f"Non-numeric columns: {non_numeric_columns}")

# Drop target and non-numeric columns
X = df.drop(columns=[target_column] + non_numeric_columns, errors='ignore')
y = df[target_column]

Explanation:
Defines the target variable (CES 4.0 Percentile Range) which represents the pollutant percentile range grouped by 5%.
Identifies all non-numeric columns and drops them to ensure numerical-only features.

# Step 3: Handle Missing Values
# Option 1: Drop rows with NaN values
# df = df.dropna()

# Option 2: Impute missing values (default)
imputer = SimpleImputer(strategy='mean')  # Use mean to replace NaN
X = imputer.fit_transform(X)

# Debugging: Validate no missing values remain
if np.any(np.isnan(X)):
    print("There are still missing values in the dataset!")
else:
    print("No missing values detected in the dataset!")

Explanation:
Handles missing data using mean imputation, replacing NaN values with the column mean.
Prints a confirmation message once missing values are filled.

# Step 4: Split the dataset
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1, random_state=42)

Explanation:
Splits the dataset into 90% training and 10% testing subsets.
random_state=42 ensures reproducibility.

# Step 5: Scale the features
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

Explanation:
Standardizes the features so that each has a mean of 0 and standard deviation of 1 — essential for SVMs since they’re sensitive to feature scale.

# Step 6: Train SVM Classifier
print("--- Training SVM Classifier ---")
svm_model = SVC(kernel='rbf', C=10, gamma=0.3, probability=True)
svm_model.fit(X_train, y_train)

# Save the SVM model
pickle.dump(svm_model, open('SvmClassifier.sav', 'wb'))
print("SVM model saved as 'SvmClassifier.sav'.")

Explanation:
Trains a Support Vector Classifier using a Radial Basis Function (RBF) kernel, which handles non-linear separations effectively.
C=10 and gamma=0.3 control the margin and influence of individual data points.
The trained model is serialized and saved as SvmClassifier.sav using pickle.

# Step 7: Train Random Forest Classifier
print("--- Training Random Forest Classifier ---")
rf_model = RandomForestClassifier(n_estimators=100, random_state=42)
rf_model.fit(X_train, y_train)

# Save the Random Forest model
pickle.dump(rf_model, open('RfClassifier.sav', 'wb'))
print("Random Forest model saved as 'RfClassifier.sav'.")

Explanation:
Trains a Random Forest with 100 decision trees.
The ensemble approach improves accuracy and reduces overfitting.
Saves the trained model as RfClassifier.sav.

Summary of Part 1:

This file:

• Loads and cleans the DACs dataset.
• Handles missing values and scales features.
• Trains SVM and Random Forest classifiers.
• Saves both trained models for later evaluation.

Part 2 – evaluation.py

import pandas as pd
import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import confusion_matrix, accuracy_score, precision_score, recall_score
import pickle

Explanation:
Imports required libraries for evaluating model performance.

# Step 1: Load the test dataset
df_test = pd.read_csv('pulsar_star_test.csv')  # Replace with the correct test dataset filename

Explanation:
Loads the test dataset.
(Note: Replace 'pulsar_star_test.csv' with your DACs test file when evaluating the DACs dataset.)

# Step 2: Separate features (X_test) and target (Y_test)
X_test = df_test.iloc[:, :-1]
Y_test = df_test.iloc[:, -1]

Explanation:
Splits the dataset into input features (X_test) and labels (Y_test).

# Step 3: Scale the features using StandardScaler
scaler = StandardScaler()
X_test = scaler.fit_transform(X_test)

Explanation:
Standardizes the test data features so they are on the same scale as those used during training.

# Step 4: Load the saved SVM model
with open('SvmClassifier.sav', 'rb') as f:
    svm_classifier = pickle.load(f)

Explanation:
Loads the previously trained SVM classifier using pickle for evaluation.

# Step 5: Evaluate the SVM classifier
print("--- SVM Classifier Evaluation ---")
Y_pred_svm = svm_classifier.predict(X_test)

# Accuracy
accuracy_svm = accuracy_score(Y_test, Y_pred_svm)
print(f"SVM Accuracy: {accuracy_svm:.3f}")

# Confusion Matrix
cm_svm = confusion_matrix(Y_test, Y_pred_svm)
tn, fp, fn, tp = cm_svm.ravel()
print("SVM Confusion Matrix:")
print(cm_svm)

# Metrics
precision_svm = precision_score(Y_test, Y_pred_svm, average='macro')
recall_svm = recall_score(Y_test, Y_pred_svm, average='macro')
specificity_svm = tn / (tn + fp)

print(f"SVM Precision: {precision_svm:.3f}")
print(f"SVM Recall: {recall_svm:.3f}")
print(f"SVM Specificity: {specificity_svm:.3f}")

Explanation:
Evaluates the SVM model on test data.
Prints:

• Accuracy → overall correctness.
• Confusion Matrix → shows predicted vs actual values.
• Precision, Recall, and Specificity → detailed performance metrics.

# Step 6: Load the saved Random Forest model
with open('RfClassifier.sav', 'rb') as f:
    rf_classifier = pickle.load(f)

Explanation:
Loads the Random Forest classifier saved from training.

# Step 7: Evaluate the Random Forest classifier
print("\n--- Random Forest Classifier Evaluation ---")
Y_pred_rf = rf_classifier.predict(X_test)

# Accuracy
accuracy_rf = accuracy_score(Y_test, Y_pred_rf)
print(f"Random Forest Accuracy: {accuracy_rf:.3f}")

# Confusion Matrix
cm_rf = confusion_matrix(Y_test, Y_pred_rf)
tn, fp, fn, tp = cm_rf.ravel()
print("Random Forest Confusion Matrix:")
print(cm_rf)

# Metrics
precision_rf = precision_score(Y_test, Y_pred_rf, average='macro')
recall_rf = recall_score(Y_test, Y_pred_rf, average='macro')
specificity_rf = tn / (tn + fp)

print(f"Random Forest Precision: {precision_rf:.3f}")
print(f"Random Forest Recall: {recall_rf:.3f}")
print(f"Random Forest Specificity: {specificity_rf:.3f}")

Explanation:
Evaluates the Random Forest classifier on the same test data.
Computes and displays the same set of metrics as the SVM section.

Summary of Part 2:

The evaluation.py script:

• Loads both trained models (SvmClassifier.sav, RfClassifier.sav).
• Evaluates each on the test dataset.
• Reports key performance metrics (Accuracy, Precision, Recall, Specificity, Confusion Matrix).

Final Notes

Observations:
The SVM (RBF kernel) is powerful for capturing non-linear relationships.
The Random Forest tends to perform better on larger, noisy datasets with mixed features.
Comparing both provides valuable insights into trade-offs between interpretability and performance.

In conclusion:
This project successfully implements and evaluates two ML classifiers for the California Disadvantaged Communities dataset, demonstrating data preprocessing, model training, persistence using Pickle, and performance evaluation — all aligned with the CSE 5120 Assignment 1 requirements.