Skip to content
The Chat Archive
gpt-4-1106-preview· OpenAI WildChat-4.8M (AI2)

import pandas as pd import numpy as np from itertools import combinations from…

  1. User

    import pandas as pd import numpy as np from itertools import combinations from backtesting import Backtest, Strategy

    class MyStrategy(Strategy): def init(self): pass def next(self): pass

    def run_cpcv_with_purging_and_embargoing(data, strategy_class, n_splits, embargo_pct): results = [] all_indices = range(len(data)) n_test_samples = int(len(data) * embargo_pct)

    for split in combinations(all_indices, n_splits):
        for start in split:
            end = start + n_test_samples
            bt = Backtest(data, strategy_class, cash=10_000, commission=.002)
            stats = bt.run()
            results.append(stats)
    return results
    

    reduce time complexity of this python code

  2. Assistant

    Your current code is using itertools.combinations over the range of indices of your data, which could be computationally explosive for larger datasets as it tries to generate combinations of indices for splitting the dataset. This will not actually perform cross-validation but will instead create unique index combinations without properly segmenting the data.

    Instead of using combinations, you should use time-series cross-validation with a rolling window approach. This takes into account the temporal nature of financial data and avoids lookahead bias. The GapWalkForward class from the sklearn.model_selection package (which we'll emulate here) or similar functionality will perform this time-series cross-validation more efficiently.

    Here is a function to use the rolling window approach with purging and embargo to calculate backtest results. This approach doesn't use combinations and iterates through the time series data once:

    from backtesting import Backtest, Strategy
    
    def run_cpcv_with_purging_and_embargoing(data, strategy_class, n_splits, embargo_pct):
        results = []
        
        # Calculate the sizes of the training and test sets
        test_size = int(len(data) * (1 / n_splits))
        embargo_size = int(test_size * embargo_pct)
        
        # Walk forward over the data: for n_splits folds, increment the starting point
        for i in range(n_splits):
            # Split data into train and test set with embargo
            train_start = 0
            test_start = i * test_size
            test_end = test_start + test_size
            train_end = test_start - embargo_size
            
            if test_end + embargo_size >= len(data):
                break  # Not enough data left for this split, so exit the loop
            
            # Create training and test sets
            train_data = data.iloc[train_start:train_end]
            test_data = data.iloc[test_start:test_end]
            
            # Merge train and test data to run backtesting (skipping the embargo period)
            bt_data = pd.concat([train_data, test_data])
    
            # Run backtesting on the combined training and test set
            bt = Backtest(bt_data, strategy_class, cash=10_000, commission=.002)
            stats = bt.run()
            
            results.append(stats)
            
        return results
    
    # ... MyStrategy class definition ...
    
    # Usage example:
    # result = run_cpcv_with_purging_and_embargoing(data, MyStrategy, 5, 0.01)
    

    This code is significantly faster because it only passes through the dataset n_splits number of times, creating a rolling window for training and testing without purging entirely overlapping test sets. It correctly implements embargoing by skipping a period of data after the training set and before the testing set.

    Remember, this function assumes your data is a Pandas DataFrame and has been preprocessed accordingly (e.g., indexed by date if it's a time series).

    Adjust the run_cpcv_with_purging_and_embargoing function parameters according to your dataset size, and remember that backtesting itself can be quite time-consuming depending on the complexity of your MyStrategy class. Thus, the total time will also be impacted by the strategy block, which is not optimized in this example.

  3. User

    import polars as pl import numpy as np import numba from joblib import parallel, delayed import asyncio import time

    class backtest:

    def __init__(self, OhlcvDf, IndicatorsDict, ShortConditionsToEnter, ShortConditionsToExit, LongConditionsToEnter, LongConditionsToExit, indicatorFunctions):
        self.OhlcvDf                    =   OhlcvDf
        self.IndicatorsDict             =   IndicatorsDict
        self.ShortConditionsToEnter     =   ShortConditionsToEnter
        self.ShortConditionsToExit      =   ShortConditionsToExit
        self.LongConditionsToEnter      =   LongConditionsToEnter
        self.LongConditionsToExit       =   LongConditionsToExit
    def ComputeIndicators(self):
        
        for indicator, params in self.IndicatorsDict.items():
            func            =   self.indicatorFunctions[indicator]
            close_column    =   self.OhlcvDf[params[0]]
            self.OhlcvDf    =   func(self.OhlcvDf, close_column, *params[1:])
            self.OhlcvDf    =   self.OhlcvDf.with_column(pl.lit(0).alias("signals"))
    
    def CalculateLongEntrySignal(self):
        
        
        
        
        
            
    

    i want to calculate entry signal using the LongConditionsToEnter which will have something like ohlcvdf['sma']>ohlcv['close']

  4. Assistant

    In order to calculate entry signals for long positions of your backtest, you can evaluate the condition specified in LongConditionsToEnter. Given that LongConditionsToEnter would include a condition like ohlcvdf['sma'] > ohlcvdf['close'], you would need to implement a method to check this condition against your OHLCV DataFrame (OhlcvDf) and accordingly set the signal.

    Here's how it might look as part of your backtest class, assuming that OhlcvDf is a polars DataFrame:

    class backtest:
        # ... your existing code ...
    
        def ComputeIndicators(self):
            # ... your existing code ...
    
        def CalculateLongEntrySignal(self):
            # Assuming LongConditionsToEnter is a string like "ohlcv['sma'] > ohlcv['close']"
            # Evaluate the condition after replacing "ohlcv" by "self.OhlcvDf"
            long_entry_condition = self.LongConditionsToEnter.replace("ohlcv", "self.OhlcvDf")
    
            # Evaluate the condition using eval
            entry_signal = eval(long_entry_condition)
    
            # Add the signal to the DataFrame
            self.OhlcvDf = self.OhlcvDf.with_column(entry_signal.alias("long_entry_signal"))
    
    # Example usage:
    # Assuming you have a function like this to calculate SMA
    def calculate_sma(df, close_column, period):
        return df.with_column(close_column.rolling_mean(window_size=period).alias('sma'))
    
    ohlcvdf = pl.DataFrame({
        'datetime': ['2021-01-01', '2021-01-02', '2021-01-03'],
        'open': [100, 105, 103],
        'high': [110, 108, 106],
        'low': [95, 102, 101],
        'close': [105, 107, 104],
    })
    
    indicators = {'sma': ('close', 2)}
    long_conditions_to_enter = "ohlcv['sma'] > ohlcv['close']"
    
    # Assuming you have functions mapping
    indicator_functions = {
        'sma': calculate_sma
    }
    
    bt = backtest(ohlcvdf, indicators, None, None, long_conditions_to_enter, None, indicator_functions)
    bt.ComputeIndicators()
    bt.CalculateLongEntrySignal()
    print(bt.OhlcvDf)
    

    Please note that eval usage can be dangerous if the expressions you are evaluating are not carefully controlled or if they come from untrusted sources, as it can lead to security issues.

    In the example code, we've assumed that you have a corresponding function (calculate_sma in this case) to calculate each of the needed indicators. This function can be any other indicator calculation function, and you would pass all such functions in the indicator_functions dict.

    The .with_column() method from polars is used to add the 'long_entry_signal' column to your OhlcvDf DataFrame. This signal would be True where the condition is met (mnemonic for an entry signal) and False otherwise. If you need numerical signals (e.g., 1 for entry and 0 for no-entry), you can modify the code accordingly, using pl.col() and pl.when().then().otherwise() from polars.