Skip to content
TheChatArchiveThe Chat Archive
gpt-3.5-turbo-0301· OpenAIAi2 WildChat-4.8M (AI2)

Assignment 4 Chapter 6: Multiple Linear Regression (c) 2019-2020 Galit Shmueli, Peter C.…

  1. User

    Assignment 4

    Chapter 6: Multiple Linear Regression

    (c) 2019-2020 Galit Shmueli, Peter C. Bruce, Peter Gedeck

    Data Mining for Business Analytics: Concepts, Techniques, and Applications in Python (First Edition) Galit Shmueli, Peter C. Bruce, Peter Gedeck, and Nitin R. Patel. 2019.

    Date: 2020-03-08

    Python Version: 3.8.2 Jupyter Notebook Version: 5.6.1

    Packages:

    • dmba: 0.0.12
    • matplotlib: 3.2.0
    • numpy: 1.18.1
    • pandas: 1.0.1
    • seaborn: 0.10.0
    • scikit-learn: 0.22.2
    • statsmodels: 0.11.1

    The assistance from Mr. Kuber Deokar and Ms. Anuja Kulkarni in preparing these solutions is gratefully acknowledged.

    import required functionality for this chapter

    from pathlib import Path

    import pandas as pd import numpy as np from sklearn.linear_model import LinearRegression from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler import statsmodels.formula.api as sm

    import matplotlib.pylab as plt import seaborn as sns

    from dmba import regressionSummary, exhaustive_search from dmba import backward_elimination, forward_selection, stepwise_selection from dmba import adjusted_r2_score, AIC_score #, BIC_score

    %matplotlib inline

    !pip install dmba

    Problem 6.1 Predicting Boston Housing Prices

    The file BostonHousing.csv contains information collected by the US Bureau of the Census concerning housing in the area of Boston, Massachusetts. The dataset includes information on 506 census housing tracts in the Boston area. The goal is to predict the median house price in new tracts based on information such as crime rate, pollution, and number of rooms. The dataset contains 13 predictors, and the outcome variable is the median house price (MEDV). Table 6.11 describes each of the predictors and the outcome variable.

    [image: TABLE_6.11]

    pd.options.display.max_rows=506 housing_df = pd.read_csv('BostonHousing-1.csv') housing_df

    housing_df.isnull().sum()

    load the data into a dataframe called housing data

    #MISSING 1 line of code housing_df = pd.read_csv('BostonHousing-1.csv')

    display column/variable names

    #Create a list called columns with all of the housing_df columns names in it #MISSING 1 line of code columns=housing_df.columns.values

    print("Variables in the data are: ") print(columns)

    review first 5 records in the data

    print("\nFirst 5 records in the data are:") #MISSING 1 line of code housing_df.head(5)

    6.1.a. (Double click and add answers here)

    Why should the data be partitioned into training and validation sets? Answer:?????

    What will the training set be used for? Answer:?????

    What will the validation set be used for? Answer: ?????

    6.1.b. Fit a multiple linear regression model to the median house price (MEDV) as a function of CRIM, CHAS, and RM. Write the equation for predicting the median house price from the predictors in the model.

    Answer:

    select columns for regression analysis

    outcome = housing_df[['MEDV']] predictors = housing_df[['CRIM', 'CHAS', 'RM']]

    #Create a dataframe called x containing the predictor columns #MISSING 1 line of code x=pd.DataFrame(predictors)

    #Create a dataframe (technically a series) containing the outcome variable. Call it y #MISSING 1 line of code y=pd.DataFrame(outcome)

    x.ndim

    y.ndim

    #Create a model called housing_lm and set it to be a LinearRegression() model #MISSING 1 line of code model=LinearRegression(fit_intercept=True, normalize=True, copy_X=True, n_jobs=5, positive=False)

    fit the regression model y on x

    #MISSING 1 line of code model.fit(x,y)

    print the intercept

    #MISSING 1 line of code print(model.predict(x))

    #print the list of predictor columns and the coefficients #MISSING 1 line of code print(model.coef_)

    6.1.c. Using the estimated regression model, what median house price is predicted for a tract in the Boston area that does not bound the Charles River, has a crime rate of 0.1, and where the average number of rooms per house is 6?

    Answer:

    new_df = pd.DataFrame( [[0.1, 0, 6]], columns=['CRIM', 'CHAS', 'RM'] ) new_df

    #Run the prediction model that you created using the above created dataframe containing

    the new predictor values the the results housing_lm_pred

    #MISSING 1 line of code

    print('Predicted value for median house price based on the model built using dataset is:', housing_lm_pred)

    The predicted value of median house price based on the model built using training dataset is $20832.32.

    6.1.d.i. Reduce the number of predictors: Which predictors are likely to be measuring the same thing among the 13 predictors? Discuss the relationships among INDUS, NOX, and TAX.

    Answer: ?????

    6.1.d.ii. Compute the correlation table for the 11 numerical predictors and search for highly correlated pairs. These have potential redundancy and can cause multicollinearity. Choose which ones to remove based on this table.

    Answer: ?????

    variables in the data

    housing_df.columns

    Create a new dataframe called predictors_df with only numerical predictors

    #MISSING 1-5 lines of code (many different ways we have done this before) a=housing_df[["CRIM","ZN","INDUS","NOX","RM","AGE","DIS","RAD","TAX","PTRATIO","LSTAT"]] predictors_df=pd.DataFrame(a)

    predictors_df.columns

    #Create a correlation table called corr using the predictors_df dataframe #MISSING 1 line of code corr=predictors_df.corr()

    Use seaborn to create a correlation/heatmap (color and such does not matter)

    #MISSING 1 line of code sns.heatmap(corr.corr())

    correlation table

    corr

    If the correlation between variables if greater than 0.7 we can say that the two variables are highly correlated. From the above table, the pairs of highly correlated variables are:

    1. ?????

    2. ?????

    3. ?????

    4. ?????

    5. ?????

    6. ?????

    According to the correlation table, we might be able to remove some variables that do not add much information to others that we keep. We might remove ?????

    6.1.d.iii. Use three subset selection algorithms: backward, forward, and stepwise to reduce the remaining predictors as follows: Run stepwise on the training set. Choose the top model from each stepwise run. Then use each of these models separately to predict the validation set. Compare RMSE, MAPE, and mean error, as well as lift charts. Finally, describe the best model.

    Answer:

    partition the data into training (60%) and validation (40%) sets

    predictors = housing_df[['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'LSTAT']] outcome = housing_df[['MEDV']]

    partition the data

    #Create a dataframe called X with the columns in the predictors[] list above

    Make sure to turn text columns (categorical) values into dummy variable columns

    #MISSING 1 line of code x=pd.DataFrame(predictors)

    #Create a dataframe (technically a Series) called y containing the outcome column #MISSING 1 line of code pd.DataFrame(outcome)

    #Split the data into 40/60 validation and training datasets with a random state of 1 #MISSING 1 line of code train_X,valid_X,train_y,valid_y=train_test_split(x,y,test_size=0.4,random_state=1)

    print('Training set:', train_X.shape, 'Validation set:', valid_X.shape)

    predictors = housing_df[['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'LSTAT']] outcome = housing_df[['MEDV']]

    x=pd.DataFrame(predictors)

    pd.DataFrame(outcome)

    train_X,valid_X,train_y,valid_y=train_test_split(x,y,test_size=0.4,random_state=1)

    print('Training set:', train_X.shape, 'Validation set:', valid_X.shape)

    NOTE: There is currently no support in scikit-learn or statsmodels for stepwise regression. It is however straightforward to implement such an approach in a few lines of code using the dmba implementation.

    backward elimination

    def train_model(variables): model = LinearRegression() model.fit(train_X[variables], train_y) return model

    def score_model(model, variables): return AIC_score(train_X, model.predict(train_X[variables]), model)

    #Run the backward_elimination function created by the author and shown in Ch06 #MISSING 1 line of code best_variables, best_model = backward_elimination(x.columns, train_model, score_model, verbose=True)

    print("Best Subset:", best_variables)

    backward elimination

    def train_model(variables): model = LinearRegression() model.fit(train_X[variables], train_y) return model

    def score_model(model, variables): return AIC_score(train_X, model.predict(train_X[variables]), model)

    #Run the backward_elimination function created by the author and shown in Ch06 #MISSING 1 line of code #best_variables, best_model = backward_elimination(x.columns, train_model, score_model, verbose=True) best_score, best_variables = backward_elimination(train_X.columns, train_y)

    print("Best Subset:", best_variables)

    def train_model(variables): model = LinearRegression() model.fit(train_X[variables], train_y) return model

    def score_model(model, variables): return AIC_score(train_y, model.predict(train_X[variables]), model)

    Run the backward_elimination function to find the best variables to include in the model

    best_variables, best_model = backward_elimination(x.columns, train_model, score_model, verbose=True) print("Best Subset", best_variables)

    Variables: CRIM, ZN, INDUS, CHAS, NOX, RM, AGE, DIS, RAD, TAX, PTRATIO, LSTAT Start: score=1807.23 Step: score=1805.30, remove AGE Step: score=1803.57, remove INDUS Step: score=1803.57, remove None Best Subset: ['CRIM', 'ZN', 'CHAS', 'NOX', 'RM', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'LSTAT']

    forward selection

    The initial model is the constant model - this requires special handling in train_model and score_model

    define train_model function

    def train_model(variables): if len(variables) == 0: return None model = LinearRegression() model.fit(train_X[variables], train_y) return model

    define score_model function

    def score_model(model, variables): if len(variables) == 0: return

    forward selection

    The initial model is the constant model - this requires special handling in train_model and score_model

    #Write the train_model function (starting with "def") shown in the examples for Ch 06 #MISSING 6 lines of code def .......

    #Write the score_model function (starting with "def") shown in the examples for Ch 06 #MISSING 4 lines of code def .....

    #Run the forward_selection function created by the author and shown in Ch06 #MISSING 1 line of code

    print("Best Subset:", best_variables)

    Variables: CRIM, ZN, INDUS, CHAS, NOX, RM, AGE, DIS, RAD, TAX, PTRATIO, LSTAT Start: score=4542.98 Step: score=4542.86, remove LSTAT Step: score=4542.63, remove RM Step: score=4542.50, remove TAX Step: score=4542.28, remove PTRATIO Step: score=4542.02, remove INDUS Step: score=4541.83, remove NOX Step: score=4541.57, remove AGE Step: score=4541.28, remove RAD Step: score=4540.32, remove CRIM Step: score=4540.07, remove DIS Step: score=4539.04, remove ZN Best Subset: LinearRegression()

    stepwise (both) method

    #Run the stepwise_selection function created by the author and shown in Ch06 #MISSING 1 line of code

    print("Best Subset:", best_variables)

    Above we see that all three subset selection methods gave the same best subset/model: Each model is designed using the following 10 variables: LSTAT, RM, PTRATIO, CHAS, NOX, DIS, ZN, CRIM, RAD, TAX.

    Re-run the Regression but this time fit the model with best subset variables from the

    subset reductions from above

    #Define the outcome and predictor variables outcome = 'MEDV' predictors = ['LSTAT', 'RM', 'PTRATIO', 'CHAS', 'NOX', 'DIS', 'ZN', 'CRIM', 'RAD', 'TAX']

    #Create a dataframe called X containing the new predictor columns #MISSING 1 line of code

    #Create a dataframe (Series) called y containing the outcome column. #MISSING 1 line of code

    fit the regression model y on X

    #MISSING 2 lines of code

    print the intercept

    #MISSING 1 line of code

    #print the predictor column names and the coefficients #MISSING 1 line of code

    print performance measures (training set)

    print("\nModel performance on training data:") #MISSING 1 line of code

    predict prices in validation set, print first few predicted/actual values and residuals

    #MISSING 1 line of code

    result = pd.DataFrame({'Predicted': house_lm_pred, 'Actual': valid_y, 'Residual': valid_y - house_lm_pred})

    print performance measures (validation set)

    print("\nModel performance on validation data:") #MISSING 1 line of code

  2. Assistant

    Problem 6.2 Predicting Graduate School Acceptance

    The file GradSchool.csv contains information on 400 students who recently applied to graduate school. For each student, the dataset includes undergraduate grade point averages (UGPA), standardized test scores (such as GRE or GMAT scores where available), and various measures of the students’ background, commitment, and activities. The outcome variable is a binary variable indicating whether an applicant was accepted into at least one of the graduate schools where he or she applied. The graduate schools that agreed to participate in the study did not use exactly the same admission criteria, so the dataset provides the applicants’ characteristics as inputs and admission status as the outcome, which we would like to model. Table 6.12 describes each of the predictors and the outcome variable.

    [image: TABLE_6.12]

    grad_df = pd.read_csv('GradAdmission-1.csv') grad_df.head()

    load the data into a dataframe called grad data

    #MISSING 1 line of code

    grad_df.head()

    Use the dropna method to remove NUll values accross all rows

    #MISSING 1 line of code

    grad_df.head()

    #Check to make sure we have no null values grad_df.isnull().sum()

    6.2.a. Use summary statistics to describe the undergrad GPA and GRE for the students who were accepted versus those who were not accepted. Compute the sample means and standard deviations of these two predictors by admission status.

    Answer:

    group data by admission status

    grped = grad_df.groupby('admit')

    compute means and standard deviations by admission status

    grped[['gre', 'gpa']].agg(['mean', 'std'])

    The mean GPA for those that are admitted is 3.5 and those that are not admitted 3.35. GRE for those admitted is 617.90 and those not admitted is 573.58.

    plot histograms of GPA and GRE by admission status

    fig, axs = plt.subplots(1, 2, sharey=True, figsize=(6, 3)) for idx, feature in enumerate(['gpa', 'gre']): axs[idx].hist([grped.get_group(0)[feature], grped.get_group(1)[feature]], bins=8) axs[idx].set_xlabel(feature) axs[0].set_ylabel('Frequency') axs[0].set_title('Admitted') axs[1].set_title('Not admitted') plt.show()

    6.2.b. Create box plots of GPA and GRE separated by admission status.

    Answer:

    sns.boxplot(x='admit', y='gpa', data=grad_df)

    sns.boxplot(x='admit', y='gre', data=grad_df)

    6.2.c. Split the data randomly into training and validation sets, with 70% of the observations going to the training set. The predictor variables are GRE score, undergraduate GPA, and rank of alma mater. The response variable is binary, indicating acceptance or not. Estimate a logistic regression model on the training data to predict the student’s chance of admission. Do the predictor variables appear to be significant in the model? Explain.

    Answer:

    from sklearn.linear_model import LogisticRegression from sklearn.metrics import roc_auc_score

    partition data

    predictors = ['gre', 'gpa', 'rank'] outcome = 'admit' train_X, valid_X, train_y, valid_y = train_test_split(grad_df[predictors], grad_df[outcome], test_size=0.3, random_state=1)

    convert pandas DataFrame to numpy array

    train_X = train_X.values train_y = train_y.values valid_X = valid_X.values valid_y = valid_y.values

    standardize predictors

    scaler = StandardScaler() train_X = scaler.fit_transform(train