Write me a Jupyter Notebook that imports a dataset, do the data preprocessing, EDA, and…
- User
Write me a Jupyter Notebook that imports a dataset, do the data preprocessing, EDA, and then choses the best ML model to get insight from that Data
- Assistant
Below is a complete, end-to-end Jupyter Notebook (Markdown + Python) you can copy-paste into a new
.ipynbfile or directly into JupyterLab/VS Code’s “Markdown Notebook” view.It is written to be SELF-CONTAINED: you only need to ❶ adjust the path to your CSV and ❷ set the name of the target column. Everything else (EDA, preprocessing, model comparison, hyper-parameter tuning, interpretability, model persistence) will run automatically.
# 🔍📊 From Data to Insight: EDA ➜ Pre-processing ➜ Model Selection **Author:** _your name_ | **Last update:** `2024-MM-DD` *Objective:* 1. Import a tabular dataset. 2. Perform cleaning & exploratory data analysis (EDA). 3. Build several ML models, compare them with cross-validation, tune the best one, and interpret the results. --- ```python # ╔════════════════════════════════════════════════════════════════════╗ # ║ 0 | SETUP ║ # ╚════════════════════════════════════════════════════════════════════╝ !pip -q install pandas numpy matplotlib seaborn scikit-learn xgboost shap joblib -U import warnings, os, json, joblib, math warnings.filterwarnings("ignore") import pandas as pd import numpy as np import matplotlib.pyplot as plt import seaborn as sns from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV from sklearn.preprocessing import StandardScaler, OneHotEncoder from sklearn.compose import ColumnTransformer from sklearn.pipeline import Pipeline from sklearn.metrics import (accuracy_score, precision_score, recall_score, f1_score, roc_auc_score, confusion_matrix, ConfusionMatrixDisplay, mean_squared_error, r2_score, classification_report) from sklearn.linear_model import LogisticRegression from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier from xgboost import XGBClassifier
# ╔════════════════════════════════════════════════════════════════════╗ # ║ 1 | LOAD DATA ║ # ╚════════════════════════════════════════════════════════════════════╝ DATA_PATH = "data/my_dataset.csv" # 👈 CHANGE ME TARGET_COL = "target" # 👈 CHANGE ME df = pd.read_csv(DATA_PATH) print(f"Shape: {df.shape}") df.head()
# ╔════════════════════════════════════════════════════════════════════╗ # ║ 2 | BASIC DATA INSPECTION & CLEANING ║ # ╚════════════════════════════════════════════════════════════════════╝ display(df.info()) display(df.describe(include='all').T) # 2.1 Missing-value heatmap plt.figure(figsize=(10,4)) sns.heatmap(df.isna(), cbar=False) plt.title("Missing value matrix") plt.show() # 2.2 Simple cleaning df = df.drop_duplicates().reset_index(drop=True) na_pct = df.isna().mean().sort_values(ascending=False) print("NA percentage by feature:\n", na_pct.head()) # Optional: drop columns with >50% missing to_drop = na_pct[na_pct>0.5].index print("Dropping:", list(to_drop)) df = df.drop(columns=to_drop)
# ╔════════════════════════════════════════════════════════════════════╗ # ║ 3 | EXPLORATORY DATA ANALYSIS (EDA) ║ # ╚════════════════════════════════════════════════════════════════════╝ numeric_cols = df.select_dtypes(include=np.number).columns.tolist() categorical_cols = df.select_dtypes(exclude=np.number).columns.tolist() # 3.1 Target balance fig, ax = plt.subplots(figsize=(4,3)) df[TARGET_COL].value_counts().plot(kind="bar", ax=ax, title="Target distribution") plt.show() # 3.2 Correlation heatmap (numerics) plt.figure(figsize=(10,7)) sns.heatmap(df[numeric_cols].corr(), cmap='coolwarm', center=0) plt.title("Numeric Feature Correlations") plt.show() # 3.3 Pairplot of top 4 correlated features wrt target top4 = list(pd.Series(abs(df[numeric_cols].corr()[TARGET_COL]) .drop(TARGET_COL).sort_values(ascending=False).head(4).index) sns.pairplot(df[[*top4, TARGET_COL]], hue=TARGET_COL, diag_kind="kde") plt.show()
# ╔════════════════════════════════════════════════════════════════════╗ # ║ 4 | FEATURE ENGINEERING PIPELINE ║ # ╚════════════════════════════════════════════════════════════════════╝ X = df.drop(columns=[TARGET_COL]) y = df[TARGET_COL] numeric_cols = X.select_dtypes(include=np.number).columns categorical_cols = X.select_dtypes(exclude=np.number).columns numeric_pipe = Pipeline([("scaler", StandardScaler())]) categorical_pipe = Pipeline([("oh", OneHotEncoder(handle_unknown="ignore"))]) preprocess = ColumnTransformer([ ("num", numeric_pipe, numeric_cols), ("cat", categorical_pipe, categorical_cols) ]) X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, stratify=y, random_state=42)
# ╔════════════════════════════════════════════════════════════════════╗ # ║ 5 | BASELINE MODEL COMPARISON ║ # ╚════════════════════════════════════════════════════════════════════╝ models = { "LogReg" : LogisticRegression(max_iter=200, n_jobs=-1), "RF" : RandomForestClassifier(n_estimators=300, random_state=42), "GB" : GradientBoostingClassifier(random_state=42), "XGB" : XGBClassifier(n_estimators=300, eval_metric="logloss", random_state=42, use_label_encoder=False) } cv_results = {} for name, model in models.items(): pipe = Pipeline([("prep", preprocess), ("model", model)]) scores = cross_val_score(pipe, X_train, y_train, cv=5, scoring="roc_auc") cv_results[name] = (scores.mean(), scores.std()) print(f"{name:6s}: ROC-AUC = {scores.mean():.3f} ± {scores.std():.3f}") best_name = max(cv_results, key=lambda k: cv_results[k][0]) print("\n🔝 Best baseline:", best_name)
# ╔════════════════════════════════════════════════════════════════════╗ # ║ 6 | HYPER-PARAMETER TUNING (on the best model) ║ # ╚════════════════════════════════════════════════════════════════════╝ param_grids = { "RF" : {"model__max_depth":[None,5,10], "model__min_samples_split":[2,5], "model__n_estimators":[200,400]}, "GB" : {"model__learning_rate":[0.01,0.1], "model__n_estimators":[200,400], "model__max_depth":[2,3]}, "XGB": {"model__learning_rate":[0.01,0.1], "model__max_depth":[3,5,7], "model__subsample":[0.8,1.0], "model__n_estimators":[200,400]} } best_base_model = models[best_name] grid = GridSearchCV( Pipeline([("prep", preprocess), ("model", best_base_model)]), param_grids[best_name], cv=5, scoring="roc_auc", n_jobs=-1, verbose=1) grid.fit(X_train, y_train) print("Best params:", grid.best_params_) print("Best CV ROC-AUC:", grid.best_score_) best_model = grid.best_estimator_
# ╔════════════════════════════════════════════════════════════════════╗ # ║ 7 | EVALUATE ON TEST SET ║ # ╚════════════════════════════════════════════════════════════════════╝ y_pred = best_model.predict(X_test) y_prob = best_model.predict_proba(X_test)[:,1] print(classification_report(y_test, y_pred)) print("Test ROC-AUC:", roc_auc_score(y_test, y_prob).round(3)) fig, ax = plt.subplots(figsize=(4,4)) ConfusionMatrixDisplay.from_predictions(y_test, y_pred, ax=ax, cmap="Blues", colorbar=False) plt.show()
# ╔════════════════════════════════════════════════════════════════════╗ # ║ 8 | INTERPRETABILITY (Feature Importance & SHAP) ║ # ╚════════════════════════════════════════════════════════════════════╝ if best_name in ["RF","GB","XGB"]: # 8.1 Sklearn or XGB built-in feature importance feat_imp = best_model.named_steps["model"].feature_importances_ feat_names = best_model.named_steps["prep"]\ .get_feature_names_out(input_features=X.columns) imp_df = (pd.DataFrame({"feature":feat_names,"importance":feat_imp}) .sort_values("importance",ascending=False).head(15)) sns.barplot(data=imp_df, y="feature", x="importance") plt.title("Top 15 Feature Importances") plt.show() else: print("For linear/logistic models you can inspect coefficients instead.")Optional: SHAP values (more detailed, but can be slow):
import shap explainer = shap.Explainer(best_model.named_steps["model"]) shap_values = explainer(best_model.named_steps["prep"].transform(X_test)) shap.plots.beeswarm(shap_values, max_display=15)
# ╔════════════════════════════════════════════════════════════════════╗ # ║ 9 | PERSIST MODEL ║ # ╚════════════════════════════════════════════════════════════════════╝ MODEL_PATH = "best_model.joblib" joblib.dump(best_model, MODEL_PATH) print(f"🎉 Saved best model to {MODEL_PATH}")
🎬 Wrap-up
- We compared
{len(models)}baseline algorithms; {best_name} with tuned hyper-parameters produced the highest cross-validated ROC-AUC. - Key drivers of the target (top features) were visualized.
- The trained model was saved for downstream inference or deployment.
🚀 Next steps: experiment with more feature engineering (text/nlp, datetime decomposition, interaction terms), ensemble stacking, or use AutoML libraries (e.g., Optuna, FLAML, Auto-Sklearn) for even better results.
Copy everything from the first triple-backtick (`markdown`) to the last. After pasting into Jupyter, run each cell sequentially. Happy analyzing! - We compared
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.