Write me a Jupyter Notebook that imports a dataset, do the data preprocessing, EDA, and…
- User
Write me a Jupyter Notebook that imports a dataset, do the data preprocessing, EDA, and then choses the best ML model to get insight from that Data
Assistant
Dataset: For this example, we'll use the Iris dataset from sklearn, which is a multiclass classification problem.
Task: We'll perform data preprocessing, exploratory data analysis (EDA), and then choose the best machine learning model to classify the iris flowers into their respective species.
# Import necessary libraries import pandas as pd import numpy as np import matplotlib.pyplot as plt import seaborn as sns from sklearn.datasets import load_iris from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler from sklearn.metrics import accuracy_score, classification_report, confusion_matrix from sklearn.linear_model import LogisticRegression from sklearn.tree import DecisionTreeClassifier from sklearn.ensemble import RandomForestClassifier from sklearn.svm import SVC from sklearn.model_selection import GridSearchCV # Load the iris dataset iris = load_iris() df = pd.DataFrame(data=iris.data, columns=iris.feature_names) df['target'] = iris.target # Data Preprocessing print("Data Preprocessing:") print(df.head()) print(df.info()) print(df.describe()) # Check for missing values print("Missing values count:") print(df.isnull().sum()) # EDA print("\nExploratory Data Analysis:") plt.figure(figsize=(10,8)) sns.heatmap(df.corr(), annot=True, cmap='coolwarm') plt.title('Correlation Matrix') plt.show() sns.pairplot(df, hue='target', height=2.5) plt.show() # Split the data into features (X) and target (y) X = df.drop('target', axis=1) y = df['target'] # Split the data into training and testing sets X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) # Standardize the features scaler = StandardScaler() X_train = scaler.fit_transform(X_train) X_test = scaler.transform(X_test) # Define the models models = { 'Logistic Regression': LogisticRegression(max_iter=1000), 'Decision Tree Classifier': DecisionTreeClassifier(), 'Random Forest Classifier': RandomForestClassifier(), 'Support Vector Machine': SVC() } # Define the hyperparameters for tuning param_grids = { 'Logistic Regression': {'C': [0.1, 1, 10]}, 'Decision Tree Classifier': {'max_depth': [3, 5, 10]}, 'Random Forest Classifier': {'n_estimators': [10, 50, 100], 'max_depth': [3, 5, 10]}, 'Support Vector Machine': {'C': [0.1, 1, 10], 'kernel': ['linear', 'rbf']} } # Perform grid search to find the best model best_model = None best_score = 0 for name, model in models.items(): grid_search = GridSearchCV(model, param_grids[name], cv=5) grid_search.fit(X_train, y_train) y_pred = grid_search.predict(X_test) score = accuracy_score(y_test, y_pred) print(f"Model: {name}, Best Parameters: {grid_search.best_params_}, Accuracy: {score:.3f}") if score > best_score: best_score = score best_model = name print(f"\nThe best model is {best_model} with an accuracy of {best_score:.3f}") # Evaluate the best model if best_model == 'Logistic Regression': best_model_instance = LogisticRegression(max_iter=1000, **param_grids[best_model]) elif best_model == 'Decision Tree Classifier': best_model_instance = DecisionTreeClassifier(**param_grids[best_model]) elif best_model == 'Random Forest Classifier': best_model_instance = RandomForestClassifier(**param_grids[best_model]) elif best_model == 'Support Vector Machine': best_model_instance = SVC(**param_grids[best_model]) best_model_instance.fit(X_train, y_train) y_pred = best_model_instance.predict(X_test) print("\nClassification Report:") print(classification_report(y_test, y_pred)) print("\nConfusion Matrix:") print(confusion_matrix(y_test, y_pred))How to use this code:
- Make sure you have the necessary libraries installed, including pandas, numpy, matplotlib, seaborn, and scikit-learn.
- Run the code in a Jupyter Notebook or a Python environment.
- The code will load the Iris dataset, perform data preprocessing, EDA, and then choose the best machine learning model to classify the iris flowers into their respective species.
- The code will output the accuracy of each model, the best model, and its corresponding accuracy.
- The code will also output the classification report and confusion matrix for the best model.
Tips and Variations:
- You can replace the Iris dataset with your own dataset.
- You can modify the preprocessing steps based on your dataset's needs.
- You can add or remove models from the
modelsdictionary. - You can tune the hyperparameters for each model by modifying the
param_gridsdictionary. - You can use different evaluation metrics, such as precision, recall, or F1-score, instead of accuracy.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.