Write a python code that will analyse the datapoints i attach in a csv file, using…
- User
Write a python code that will analyse the datapoints i attach in a csv file, using various statistical approach to check what might be the relationship and also predict/forecast the potential trend of the primary series. Forecasting might be done using Models like ARIMA and LSTM
- Assistant
Sure! Below is a Python code that will help you analyze your data points from a CSV file using various statistical approaches, check the relationships, and forecast the potential trend of the primary series using ARIMA and LSTM models.
Please make sure you have the necessary libraries installed before running the code. You can install missing libraries using
pip install library_name.# Import necessary libraries import pandas as pd import numpy as np import matplotlib.pyplot as plt import seaborn as sns # For statistical modeling from statsmodels.tsa.stattools import adfuller from statsmodels.graphics.tsaplots import plot_acf, plot_pacf from statsmodels.tsa.arima.model import ARIMA # For LSTM model from sklearn.preprocessing import MinMaxScaler from tensorflow.keras.models import Sequential from tensorflow.keras.layers import LSTM, Dense # For evaluating models from sklearn.metrics import mean_squared_error import math1. Read the Data
# Replace 'your_data.csv' with the path to your CSV file df = pd.read_csv('your_data.csv', parse_dates=True, index_col=0) # Display the first few rows print(df.head())- Note: Ensure that your CSV file has a date column set as the index for time-series analysis. If not, adjust the
read_csvparameters accordingly.
2. Exploratory Data Analysis (EDA)
2.1. Basic Statistics
# Descriptive statistics print(df.describe())2.2. Visualize the Time Series
# Plot the primary series plt.figure(figsize=(12,6)) plt.plot(df['Primary_Series_Column_Name']) plt.title('Primary Series over Time') plt.ylabel('Values') plt.xlabel('Time') plt.show()- Replace
'Primary_Series_Column_Name'with the actual column name of your primary series.
2.3. Checking for Missing Values
# Check for missing values print(df.isnull().sum())- Handle missing values if any:
# For simplicity, we can forward-fill missing values df = df.fillna(method='ffill')3. Relationship Analysis
3.1. Correlation Matrix
# Compute the correlation matrix corr_matrix = df.corr() # Plot the correlation heatmap plt.figure(figsize=(12,8)) sns.heatmap(corr_matrix, annot=True, cmap='coolwarm') plt.title('Correlation Matrix') plt.show()- This heatmap will show the correlation coefficients between the variables, helping identify potential relationships.
3.2. Pair Plot
# Pair plot of all variables sns.pairplot(df) plt.show()4. ARIMA Model for Forecasting
4.1. Stationarity Check
Time-series data needs to be stationary for ARIMA modeling.
# Performing the Augmented Dickey-Fuller test from statsmodels.tsa.stattools import adfuller def adf_test(series): result = adfuller(series.dropna()) print('ADF Statistic:', result[0]) print('p-value:', result[1]) for key, value in result[4].items(): print('Critical Value (%s): %.3f' % (key, value)) adf_test(df['Primary_Series_Column_Name'])- If the series is not stationary (p-value > 0.05), differencing is needed.
4.2. Differencing
# First-order differencing df_diff = df['Primary_Series_Column_Name'].diff().dropna() # Re-test stationarity adf_test(df_diff)4.3. Determine ARIMA Order (p, d, q)
# Plot ACF and PACF fig, axes = plt.subplots(1, 2, figsize=(16,4)) # ACF plot plot_acf(df_diff, ax=axes[0], lags=20) axes[0].set_title('Autocorrelation Function (ACF)') # PACF plot plot_pacf(df_diff, ax=axes[1], lags=20) axes[1].set_title('Partial Autocorrelation Function (PACF)') plt.show()- Use the plots to determine the values of
pandq.
4.4. Fit ARIMA Model
# Define the ARIMA model model = ARIMA(df['Primary_Series_Column_Name'], order=(p,d,q)) # Fit the model model_fit = model.fit() # Print the model summary print(model_fit.summary())- Replace
(p,d,q)with the appropriate integers based on the ACF and PACF plots.
4.5. Forecasting with ARIMA
# Forecasting the next N periods N = 10 # Number of periods to forecast forecast = model_fit.forecast(steps=N) print(forecast) # Plot the forecast plt.figure(figsize=(12,6)) plt.plot(df['Primary_Series_Column_Name'], label='Historical') plt.plot(forecast, label='Forecast', color='red') plt.title('ARIMA Forecast') plt.xlabel('Time') plt.ylabel('Values') plt.legend() plt.show()5. LSTM Model for Forecasting
5.1. Data Preparation
5.1.1. Scaling
# Scaling the data scaler = MinMaxScaler(feature_range=(0,1)) scaled_data = scaler.fit_transform(df[['Primary_Series_Column_Name']])5.1.2. Creating Sequences
# Create sequences and corresponding outputs def create_sequences(data, seq_length): X = [] y = [] for i in range(len(data)-seq_length): X.append(data[i:(i+seq_length), 0]) y.append(data[i+seq_length, 0]) return np.array(X), np.array(y) seq_length = 10 # Number of time steps to look back X, y = create_sequences(scaled_data, seq_length) # Reshape X for LSTM input X = X.reshape((X.shape[0], X.shape[1], 1))5.2. Split Data into Training and Testing Sets
# Split data into training and testing sets train_size = int(len(X) * 0.80) X_train, X_test = X[:train_size], X[train_size:] y_train, y_test = y[:train_size], y[train_size:]5.3. Building the LSTM Model
model = Sequential() model.add(LSTM(50, activation='relu', input_shape=(seq_length, 1))) model.add(Dense(1)) model.compile(optimizer='adam', loss='mse') # Model summary model.summary()5.4. Training the LSTM Model
# Train the model history = model.fit(X_train, y_train, epochs=50, batch_size=32, validation_data=(X_test, y_test), verbose=1)5.5. Evaluate the Model
# Plot training & validation loss values plt.figure(figsize=(12,6)) plt.plot(history.history['loss'], label='Train') plt.plot(history.history['val_loss'], label='Test') plt.title('Model Loss') plt.ylabel('Loss') plt.xlabel('Epoch') plt.legend() plt.show()5.6. Make Predictions
# Make predictions y_pred = model.predict(X_test) # Inverse transform predictions and actual values y_pred_inv = scaler.inverse_transform(y_pred) y_test_inv = scaler.inverse_transform(y_test.reshape(-1,1))5.7. Calculate RMSE
# Calculate RMSE rmse = math.sqrt(mean_squared_error(y_test_inv, y_pred_inv)) print('Test RMSE: %.3f' % rmse)5.8. Plot the Predictions
# Plot actual vs predicted plt.figure(figsize=(12,6)) plt.plot(y_test_inv, label='Actual') plt.plot(y_pred_inv, label='Predicted') plt.title('LSTM Model Predictions') plt.xlabel('Time') plt.ylabel('Values') plt.legend() plt.show()5.9. Forecasting Future Values
# Use the last sequence from the test set to predict future values last_sequence = scaled_data[-seq_length:] forecast = [] # Number of future periods to forecast N_future = 10 current_seq = last_sequence.reshape((1, seq_length, 1)) for _ in range(N_future): pred = model.predict(current_seq) forecast.append(pred[0,0]) current_seq = np.append(current_seq[:,1:,:], [[pred]], axis=1) # Inverse transform the forecasted values forecast_inv = scaler.inverse_transform(np.array(forecast).reshape(-1,1)) # Create a date range for the forecasted values last_date = df.index[-1] forecast_dates = pd.date_range(last_date, periods=N_future+1, freq='D')[1:] # Plot the forecasted values plt.figure(figsize=(12,6)) plt.plot(df['Primary_Series_Column_Name'], label='Historical') plt.plot(forecast_dates, forecast_inv, label='Forecast', color='red') plt.title('LSTM Forecast') plt.xlabel('Time') plt.ylabel('Values') plt.legend() plt.show()6. Conclusion
The above code performs the following:
- Reads and preprocesses the data.
- Conducts exploratory data analysis and visualizes relationships.
- Checks for stationarity and fits an ARIMA model for forecasting.
- Builds and trains an LSTM neural network model for time-series forecasting.
- Evaluates and visualizes the predictions from both models.
Note:
- Replace placeholder names like
'Primary_Series_Column_Name'with the actual column names from your dataset. - Adjust parameters like
p,d,q,seq_length, and model hyperparameters based on your specific data and model performance. - Ensure your data is properly indexed by date for time-series analysis.
- Depending on the size and nature of your data, you might need to tweak the models for optimal performance.
Optional Enhancements:
- Perform hyperparameter tuning using grid search for both ARIMA and LSTM models.
- Incorporate additional features or exogenous variables if available.
- Validate the models using cross-validation techniques.
Feel free to run the code step by step and adjust parameters as needed to suit your data and analysis requirements.
- Note: Ensure that your CSV file has a date column set as the index for time-series analysis. If not, adjust the
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.