Assignment 4 Chapter 6: Multiple Linear Regression (c) 2019-2020 Galit Shmueli, Peter C.…
- User
Assignment 4
Chapter 6: Multiple Linear Regression
(c) 2019-2020 Galit Shmueli, Peter C. Bruce, Peter Gedeck
Data Mining for Business Analytics: Concepts, Techniques, and Applications in Python (First Edition) Galit Shmueli, Peter C. Bruce, Peter Gedeck, and Nitin R. Patel. 2019.
Date: 2020-03-08
Python Version: 3.8.2 Jupyter Notebook Version: 5.6.1
Packages:
- dmba: 0.0.12
- matplotlib: 3.2.0
- numpy: 1.18.1
- pandas: 1.0.1
- seaborn: 0.10.0
- scikit-learn: 0.22.2
- statsmodels: 0.11.1
The assistance from Mr. Kuber Deokar and Ms. Anuja Kulkarni in preparing these solutions is gratefully acknowledged.
import required functionality for this chapter
from pathlib import Path
import pandas as pd import numpy as np from sklearn.linear_model import LinearRegression from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler import statsmodels.formula.api as sm
import matplotlib.pylab as plt import seaborn as sns
from dmba import regressionSummary, exhaustive_search from dmba import backward_elimination, forward_selection, stepwise_selection from dmba import adjusted_r2_score, AIC_score #, BIC_score
%matplotlib inline
!pip install dmba
Problem 6.1 Predicting Boston Housing Prices
The file BostonHousing.csv contains information collected by the US Bureau of the Census concerning housing in the area of Boston, Massachusetts. The dataset includes information on 506 census housing tracts in the Boston area. The goal is to predict the median house price in new tracts based on information such as crime rate, pollution, and number of rooms. The dataset contains 13 predictors, and the outcome variable is the median house price (MEDV). Table 6.11 describes each of the predictors and the outcome variable.
[image: TABLE_6.11]
pd.options.display.max_rows=506 housing_df = pd.read_csv('BostonHousing-1.csv') housing_df
housing_df.isnull().sum()
load the data into a dataframe called housing data
#MISSING 1 line of code housing_df = pd.read_csv('BostonHousing-1.csv')
display column/variable names
#Create a list called columns with all of the housing_df columns names in it #MISSING 1 line of code columns=housing_df.columns.values
print("Variables in the data are: ") print(columns)
review first 5 records in the data
print("\nFirst 5 records in the data are:") #MISSING 1 line of code housing_df.head(5)
6.1.a. (Double click and add answers here)
Why should the data be partitioned into training and validation sets? Answer:?????
What will the training set be used for? Answer:?????
What will the validation set be used for? Answer: ?????
6.1.b. Fit a multiple linear regression model to the median house price (MEDV) as a function of CRIM, CHAS, and RM. Write the equation for predicting the median house price from the predictors in the model.
Answer:
select columns for regression analysis
outcome = housing_df[['MEDV']] predictors = housing_df[['CRIM', 'CHAS', 'RM']]
#Create a dataframe called x containing the predictor columns #MISSING 1 line of code x=pd.DataFrame(predictors)
#Create a dataframe (technically a series) containing the outcome variable. Call it y #MISSING 1 line of code y=pd.DataFrame(outcome)
x.ndim
y.ndim
#Create a model called housing_lm and set it to be a LinearRegression() model #MISSING 1 line of code model=LinearRegression(fit_intercept=True, normalize=True, copy_X=True, n_jobs=5, positive=False)
fit the regression model y on x
#MISSING 1 line of code model.fit(x,y)
print the intercept
#MISSING 1 line of code print(model.predict(x))
#print the list of predictor columns and the coefficients #MISSING 1 line of code print(model.coef_)
6.1.c. Using the estimated regression model, what median house price is predicted for a tract in the Boston area that does not bound the Charles River, has a crime rate of 0.1, and where the average number of rooms per house is 6?
Answer:
new_df = pd.DataFrame( [[0.1, 0, 6]], columns=['CRIM', 'CHAS', 'RM'] ) new_df
#Run the prediction model that you created using the above created dataframe containing
the new predictor values the the results housing_lm_pred
#MISSING 1 line of code
print('Predicted value for median house price based on the model built using dataset is:', housing_lm_pred)
The predicted value of median house price based on the model built using training dataset is $20832.32.
6.1.d.i. Reduce the number of predictors: Which predictors are likely to be measuring the same thing among the 13 predictors? Discuss the relationships among INDUS, NOX, and TAX.
Answer: ?????
6.1.d.ii. Compute the correlation table for the 11 numerical predictors and search for highly correlated pairs. These have potential redundancy and can cause multicollinearity. Choose which ones to remove based on this table.
Answer: ?????
variables in the data
housing_df.columns
Create a new dataframe called predictors_df with only numerical predictors
#MISSING 1-5 lines of code (many different ways we have done this before) a=housing_df[["CRIM","ZN","INDUS","NOX","RM","AGE","DIS","RAD","TAX","PTRATIO","LSTAT"]] predictors_df=pd.DataFrame(a)
predictors_df.columns
#Create a correlation table called corr using the predictors_df dataframe #MISSING 1 line of code corr=predictors_df.corr()
Use seaborn to create a correlation/heatmap (color and such does not matter)
#MISSING 1 line of code sns.heatmap(corr.corr())
correlation table
corr
If the correlation between variables if greater than 0.7 we can say that the two variables are highly correlated. From the above table, the pairs of highly correlated variables are:
-
?????
-
?????
-
?????
-
?????
-
?????
-
?????
According to the correlation table, we might be able to remove some variables that do not add much information to others that we keep. We might remove ?????
6.1.d.iii. Use three subset selection algorithms: backward, forward, and stepwise to reduce the remaining predictors as follows: Run stepwise on the training set. Choose the top model from each stepwise run. Then use each of these models separately to predict the validation set. Compare RMSE, MAPE, and mean error, as well as lift charts. Finally, describe the best model.
Answer:
partition the data into training (60%) and validation (40%) sets
predictors = housing_df[['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'LSTAT']] outcome = housing_df[['MEDV']]
partition the data
#Create a dataframe called X with the columns in the predictors[] list above
Make sure to turn text columns (categorical) values into dummy variable columns
#MISSING 1 line of code x=pd.DataFrame(predictors)
#Create a dataframe (technically a Series) called y containing the outcome column #MISSING 1 line of code pd.DataFrame(outcome)
#Split the data into 40/60 validation and training datasets with a random state of 1 #MISSING 1 line of code train_X,valid_X,train_y,valid_y=train_test_split(x,y,test_size=0.4,random_state=1)
print('Training set:', train_X.shape, 'Validation set:', valid_X.shape)
predictors = housing_df[['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'LSTAT']] outcome = housing_df[['MEDV']]
x=pd.DataFrame(predictors)
pd.DataFrame(outcome)
train_X,valid_X,train_y,valid_y=train_test_split(x,y,test_size=0.4,random_state=1)
print('Training set:', train_X.shape, 'Validation set:', valid_X.shape)
NOTE: There is currently no support in scikit-learn or statsmodels for stepwise regression. It is however straightforward to implement such an approach in a few lines of code using the
dmbaimplementation.backward elimination
def train_model(variables): model = LinearRegression() model.fit(train_X[variables], train_y) return model
def score_model(model, variables): return AIC_score(train_X, model.predict(train_X[variables]), model)
#Run the backward_elimination function created by the author and shown in Ch06 #MISSING 1 line of code best_variables, best_model = backward_elimination(x.columns, train_model, score_model, verbose=True)
print("Best Subset:", best_variables)
backward elimination
def train_model(variables): model = LinearRegression() model.fit(train_X[variables], train_y) return model
def score_model(model, variables): return AIC_score(train_X, model.predict(train_X[variables]), model)
#Run the backward_elimination function created by the author and shown in Ch06 #MISSING 1 line of code #best_variables, best_model = backward_elimination(x.columns, train_model, score_model, verbose=True) best_score, best_variables = backward_elimination(train_X.columns, train_y)
print("Best Subset:", best_variables)
def train_model(variables): model = LinearRegression() model.fit(train_X[variables], train_y) return model
def score_model(model, variables): return AIC_score(train_y, model.predict(train_X[variables]), model)
Run the backward_elimination function to find the best variables to include in the model
best_variables, best_model = backward_elimination(x.columns, train_model, score_model, verbose=True) print("Best Subset", best_variables)
Variables: CRIM, ZN, INDUS, CHAS, NOX, RM, AGE, DIS, RAD, TAX, PTRATIO, LSTAT Start: score=1807.23 Step: score=1805.30, remove AGE Step: score=1803.57, remove INDUS Step: score=1803.57, remove None Best Subset: ['CRIM', 'ZN', 'CHAS', 'NOX', 'RM', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'LSTAT']
forward selection
The initial model is the constant model - this requires special handling in train_model and score_model
define train_model function
def train_model(variables): if len(variables) == 0: return None model = LinearRegression() model.fit(train_X[variables], train_y) return model
define score_model function
def score_model(model, variables): if len(variables) == 0: return
forward selection
The initial model is the constant model - this requires special handling in train_model and score_model
#Write the train_model function (starting with "def") shown in the examples for Ch 06 #MISSING 6 lines of code def .......
#Write the score_model function (starting with "def") shown in the examples for Ch 06 #MISSING 4 lines of code def .....
#Run the forward_selection function created by the author and shown in Ch06 #MISSING 1 line of code
print("Best Subset:", best_variables)
Variables: CRIM, ZN, INDUS, CHAS, NOX, RM, AGE, DIS, RAD, TAX, PTRATIO, LSTAT Start: score=4542.98 Step: score=4542.86, remove LSTAT Step: score=4542.63, remove RM Step: score=4542.50, remove TAX Step: score=4542.28, remove PTRATIO Step: score=4542.02, remove INDUS Step: score=4541.83, remove NOX Step: score=4541.57, remove AGE Step: score=4541.28, remove RAD Step: score=4540.32, remove CRIM Step: score=4540.07, remove DIS Step: score=4539.04, remove ZN Best Subset: LinearRegression()
stepwise (both) method
#Run the stepwise_selection function created by the author and shown in Ch06 #MISSING 1 line of code
print("Best Subset:", best_variables)
Above we see that all three subset selection methods gave the same best subset/model: Each model is designed using the following 10 variables: LSTAT, RM, PTRATIO, CHAS, NOX, DIS, ZN, CRIM, RAD, TAX.
Re-run the Regression but this time fit the model with best subset variables from the
subset reductions from above
#Define the outcome and predictor variables outcome = 'MEDV' predictors = ['LSTAT', 'RM', 'PTRATIO', 'CHAS', 'NOX', 'DIS', 'ZN', 'CRIM', 'RAD', 'TAX']
#Create a dataframe called X containing the new predictor columns #MISSING 1 line of code
#Create a dataframe (Series) called y containing the outcome column. #MISSING 1 line of code
fit the regression model y on X
#MISSING 2 lines of code
print the intercept
#MISSING 1 line of code
#print the predictor column names and the coefficients #MISSING 1 line of code
print performance measures (training set)
print("\nModel performance on training data:") #MISSING 1 line of code
predict prices in validation set, print first few predicted/actual values and residuals
#MISSING 1 line of code
result = pd.DataFrame({'Predicted': house_lm_pred, 'Actual': valid_y, 'Residual': valid_y - house_lm_pred})
print performance measures (validation set)
print("\nModel performance on validation data:") #MISSING 1 line of code
- Assistant
Problem 6.2 Predicting Graduate School Acceptance
The file GradSchool.csv contains information on 400 students who recently applied to graduate school. For each student, the dataset includes undergraduate grade point averages (UGPA), standardized test scores (such as GRE or GMAT scores where available), and various measures of the students’ background, commitment, and activities. The outcome variable is a binary variable indicating whether an applicant was accepted into at least one of the graduate schools where he or she applied. The graduate schools that agreed to participate in the study did not use exactly the same admission criteria, so the dataset provides the applicants’ characteristics as inputs and admission status as the outcome, which we would like to model. Table 6.12 describes each of the predictors and the outcome variable.
[image: TABLE_6.12]
grad_df = pd.read_csv('GradAdmission-1.csv') grad_df.head()
load the data into a dataframe called grad data
#MISSING 1 line of code
grad_df.head()
Use the dropna method to remove NUll values accross all rows
#MISSING 1 line of code
grad_df.head()
#Check to make sure we have no null values grad_df.isnull().sum()
6.2.a. Use summary statistics to describe the undergrad GPA and GRE for the students who were accepted versus those who were not accepted. Compute the sample means and standard deviations of these two predictors by admission status.
Answer:
group data by admission status
grped = grad_df.groupby('admit')
compute means and standard deviations by admission status
grped[['gre', 'gpa']].agg(['mean', 'std'])
The mean GPA for those that are admitted is 3.5 and those that are not admitted 3.35. GRE for those admitted is 617.90 and those not admitted is 573.58.
plot histograms of GPA and GRE by admission status
fig, axs = plt.subplots(1, 2, sharey=True, figsize=(6, 3)) for idx, feature in enumerate(['gpa', 'gre']): axs[idx].hist([grped.get_group(0)[feature], grped.get_group(1)[feature]], bins=8) axs[idx].set_xlabel(feature) axs[0].set_ylabel('Frequency') axs[0].set_title('Admitted') axs[1].set_title('Not admitted') plt.show()
6.2.b. Create box plots of GPA and GRE separated by admission status.
Answer:
sns.boxplot(x='admit', y='gpa', data=grad_df)
sns.boxplot(x='admit', y='gre', data=grad_df)
6.2.c. Split the data randomly into training and validation sets, with 70% of the observations going to the training set. The predictor variables are GRE score, undergraduate GPA, and rank of alma mater. The response variable is binary, indicating acceptance or not. Estimate a logistic regression model on the training data to predict the student’s chance of admission. Do the predictor variables appear to be significant in the model? Explain.
Answer:
from sklearn.linear_model import LogisticRegression from sklearn.metrics import roc_auc_score
partition data
predictors = ['gre', 'gpa', 'rank'] outcome = 'admit' train_X, valid_X, train_y, valid_y = train_test_split(grad_df[predictors], grad_df[outcome], test_size=0.3, random_state=1)
convert pandas DataFrame to numpy array
train_X = train_X.values train_y = train_y.values valid_X = valid_X.values valid_y = valid_y.values
standardize predictors
scaler = StandardScaler() train_X = scaler.fit_transform(train