Skip to content
The Chat Archive
o3-2025-04-16· OpenAI LMArena Human Preference 140K

Write me a Jupyter Notebook that imports a dataset, do the data preprocessing, EDA, and…

  1. User

    Write me a Jupyter Notebook that imports a dataset, do the data preprocessing, EDA, and then choses the best ML model to get insight from that Data

  2. Assistant

    Below is a complete, end-to-end Jupyter Notebook (Markdown + Python) you can copy-paste into a new .ipynb file or directly into JupyterLab/VS Code’s “Markdown Notebook” view.

    It is written to be SELF-CONTAINED: you only need to ❶ adjust the path to your CSV and ❷ set the name of the target column. Everything else (EDA, preprocessing, model comparison, hyper-parameter tuning, interpretability, model persistence) will run automatically.

    # 🔍📊 From Data to Insight: EDA ➜ Pre-processing ➜ Model Selection
    
    **Author:** _your name_   |   **Last update:** `2024-MM-DD`
    
    *Objective:*  
    1. Import a tabular dataset.  
    2. Perform cleaning & exploratory data analysis (EDA).  
    3. Build several ML models, compare them with cross-validation, tune the best one, and interpret the results.  
    
    ---
    
    ```python
    # ╔════════════════════════════════════════════════════════════════════╗
    # ║ 0 | SETUP                                                         ║
    # ╚════════════════════════════════════════════════════════════════════╝
    !pip -q install pandas numpy matplotlib seaborn scikit-learn xgboost shap joblib -U
    
    import warnings, os, json, joblib, math
    warnings.filterwarnings("ignore")
    
    import pandas as pd
    import numpy as np
    import matplotlib.pyplot as plt
    import seaborn as sns
    
    from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV
    from sklearn.preprocessing import StandardScaler, OneHotEncoder
    from sklearn.compose import ColumnTransformer
    from sklearn.pipeline import Pipeline
    from sklearn.metrics import (accuracy_score, precision_score, recall_score, f1_score,
                                 roc_auc_score, confusion_matrix, ConfusionMatrixDisplay,
                                 mean_squared_error, r2_score, classification_report)
    from sklearn.linear_model import LogisticRegression
    from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
    from xgboost import XGBClassifier
    

    # ╔════════════════════════════════════════════════════════════════════╗
    # ║ 1 | LOAD DATA                                                     ║
    # ╚════════════════════════════════════════════════════════════════════╝
    DATA_PATH  = "data/my_dataset.csv"   # 👈 CHANGE ME
    TARGET_COL = "target"                # 👈 CHANGE ME
    
    df = pd.read_csv(DATA_PATH)
    print(f"Shape: {df.shape}")
    df.head()
    

    # ╔════════════════════════════════════════════════════════════════════╗
    # ║ 2 | BASIC DATA INSPECTION & CLEANING                              ║
    # ╚════════════════════════════════════════════════════════════════════╝
    display(df.info())
    display(df.describe(include='all').T)
    
    # 2.1 Missing-value heatmap
    plt.figure(figsize=(10,4))
    sns.heatmap(df.isna(), cbar=False)
    plt.title("Missing value matrix")
    plt.show()
    
    # 2.2 Simple cleaning
    df = df.drop_duplicates().reset_index(drop=True)
    na_pct = df.isna().mean().sort_values(ascending=False)
    print("NA percentage by feature:\n", na_pct.head())
    
    # Optional: drop columns with >50% missing
    to_drop = na_pct[na_pct>0.5].index
    print("Dropping:", list(to_drop))
    df = df.drop(columns=to_drop)
    

    # ╔════════════════════════════════════════════════════════════════════╗
    # ║ 3 | EXPLORATORY DATA ANALYSIS (EDA)                              ║
    # ╚════════════════════════════════════════════════════════════════════╝
    numeric_cols = df.select_dtypes(include=np.number).columns.tolist()
    categorical_cols = df.select_dtypes(exclude=np.number).columns.tolist()
    
    # 3.1 Target balance
    fig, ax = plt.subplots(figsize=(4,3))
    df[TARGET_COL].value_counts().plot(kind="bar", ax=ax, title="Target distribution")
    plt.show()
    
    # 3.2 Correlation heatmap (numerics)
    plt.figure(figsize=(10,7))
    sns.heatmap(df[numeric_cols].corr(), cmap='coolwarm', center=0)
    plt.title("Numeric Feature Correlations")
    plt.show()
    
    # 3.3 Pairplot of top 4 correlated features wrt target
    top4 = list(pd.Series(abs(df[numeric_cols].corr()[TARGET_COL])
                          .drop(TARGET_COL).sort_values(ascending=False).head(4).index)
    sns.pairplot(df[[*top4, TARGET_COL]], hue=TARGET_COL, diag_kind="kde")
    plt.show()
    

    # ╔════════════════════════════════════════════════════════════════════╗
    # ║ 4 | FEATURE ENGINEERING PIPELINE                                   ║
    # ╚════════════════════════════════════════════════════════════════════╝
    X = df.drop(columns=[TARGET_COL])
    y = df[TARGET_COL]
    
    numeric_cols = X.select_dtypes(include=np.number).columns
    categorical_cols = X.select_dtypes(exclude=np.number).columns
    
    numeric_pipe = Pipeline([("scaler", StandardScaler())])
    categorical_pipe = Pipeline([("oh", OneHotEncoder(handle_unknown="ignore"))])
    
    preprocess = ColumnTransformer([
        ("num", numeric_pipe, numeric_cols),
        ("cat", categorical_pipe, categorical_cols)
    ])
    
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.2, stratify=y, random_state=42)
    

    # ╔════════════════════════════════════════════════════════════════════╗
    # ║ 5 | BASELINE MODEL COMPARISON                                     ║
    # ╚════════════════════════════════════════════════════════════════════╝
    models = {
        "LogReg" : LogisticRegression(max_iter=200, n_jobs=-1),
        "RF"     : RandomForestClassifier(n_estimators=300, random_state=42),
        "GB"     : GradientBoostingClassifier(random_state=42),
        "XGB"    : XGBClassifier(n_estimators=300, eval_metric="logloss", random_state=42, use_label_encoder=False)
    }
    
    cv_results = {}
    for name, model in models.items():
        pipe = Pipeline([("prep", preprocess), ("model", model)])
        scores = cross_val_score(pipe, X_train, y_train, cv=5, scoring="roc_auc")
        cv_results[name] = (scores.mean(), scores.std())
        print(f"{name:6s}: ROC-AUC = {scores.mean():.3f} ± {scores.std():.3f}")
    
    best_name = max(cv_results, key=lambda k: cv_results[k][0])
    print("\n🔝 Best baseline:", best_name)
    

    # ╔════════════════════════════════════════════════════════════════════╗
    # ║ 6 | HYPER-PARAMETER TUNING (on the best model)                    ║
    # ╚════════════════════════════════════════════════════════════════════╝
    param_grids = {
        "RF" : {"model__max_depth":[None,5,10],
                "model__min_samples_split":[2,5],
                "model__n_estimators":[200,400]},
        "GB" : {"model__learning_rate":[0.01,0.1],
                "model__n_estimators":[200,400],
                "model__max_depth":[2,3]},
        "XGB": {"model__learning_rate":[0.01,0.1],
                "model__max_depth":[3,5,7],
                "model__subsample":[0.8,1.0],
                "model__n_estimators":[200,400]}
    }
    
    best_base_model = models[best_name]
    grid = GridSearchCV(
        Pipeline([("prep", preprocess), ("model", best_base_model)]),
        param_grids[best_name],
        cv=5, scoring="roc_auc", n_jobs=-1, verbose=1)
    grid.fit(X_train, y_train)
    
    print("Best params:", grid.best_params_)
    print("Best CV ROC-AUC:", grid.best_score_)
    best_model = grid.best_estimator_
    

    # ╔════════════════════════════════════════════════════════════════════╗
    # ║ 7 | EVALUATE ON TEST SET                                         ║
    # ╚════════════════════════════════════════════════════════════════════╝
    y_pred = best_model.predict(X_test)
    y_prob = best_model.predict_proba(X_test)[:,1]
    
    print(classification_report(y_test, y_pred))
    print("Test ROC-AUC:", roc_auc_score(y_test, y_prob).round(3))
    
    fig, ax = plt.subplots(figsize=(4,4))
    ConfusionMatrixDisplay.from_predictions(y_test, y_pred, ax=ax, cmap="Blues", colorbar=False)
    plt.show()
    

    # ╔════════════════════════════════════════════════════════════════════╗
    # ║ 8 | INTERPRETABILITY (Feature Importance & SHAP)                  ║
    # ╚════════════════════════════════════════════════════════════════════╝
    if best_name in ["RF","GB","XGB"]:
        # 8.1 Sklearn or XGB built-in feature importance
        feat_imp = best_model.named_steps["model"].feature_importances_
        feat_names = best_model.named_steps["prep"]\
            .get_feature_names_out(input_features=X.columns)
        imp_df = (pd.DataFrame({"feature":feat_names,"importance":feat_imp})
                    .sort_values("importance",ascending=False).head(15))
        sns.barplot(data=imp_df, y="feature", x="importance")
        plt.title("Top 15 Feature Importances")
        plt.show()
    else:
        print("For linear/logistic models you can inspect coefficients instead.")
    

    Optional: SHAP values (more detailed, but can be slow):

    import shap
    explainer = shap.Explainer(best_model.named_steps["model"])
    shap_values = explainer(best_model.named_steps["prep"].transform(X_test))
    shap.plots.beeswarm(shap_values, max_display=15)
    

    # ╔════════════════════════════════════════════════════════════════════╗
    # ║ 9 | PERSIST MODEL                                                ║
    # ╚════════════════════════════════════════════════════════════════════╝
    MODEL_PATH = "best_model.joblib"
    joblib.dump(best_model, MODEL_PATH)
    print(f"🎉 Saved best model to {MODEL_PATH}")
    

    🎬 Wrap-up

    • We compared {len(models)} baseline algorithms; {best_name} with tuned hyper-parameters produced the highest cross-validated ROC-AUC.
    • Key drivers of the target (top features) were visualized.
    • The trained model was saved for downstream inference or deployment.

    🚀 Next steps: experiment with more feature engineering (text/nlp, datetime decomposition, interaction terms), ensemble stacking, or use AutoML libraries (e.g., Optuna, FLAML, Auto-Sklearn) for even better results.

    
    Copy everything from the first triple-backtick (`markdown`) to the last.  
    After pasting into Jupyter, run each cell sequentially.  
    
    Happy analyzing!
    

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.