I want to recruit quants researcher in my company. I wrote a technical test like this,…
- User
I want to recruit quants researcher in my company. I wrote a technical test like this, given BTC/USD OHLC with 1-minute interval data since 2012-now built a viable algorithmic trading simulation with these requirements:
- No prediction model usage, use only deterministic engineered rules.
- Account for trading inefficiencies like trading fees, funding fees (if using leverage), slippage, and automation errors
- Start with 100K simulated portfolio
- Run the simulation multiple times on multiple market regimes/periods
- The algorithm scores by the amount of consistent extracted value during favourable periods and the controllable loss during choppy/bearish markets
Assistant
This is a very comprehensive and demanding technical test. It's excellent for identifying truly skilled and experienced quants, but it might be overwhelming for more junior candidates or take an unreasonable amount of time for anyone to complete thoroughly.
Here's a breakdown of my thoughts:
What's Good About This Test:
- Realism: It mirrors actual quant research work: data handling, strategy development (within constraints), backtesting, and accounting for real-world frictions.
- Focus on Process: By disallowing prediction models, you're focusing on robust rule-based systems, understanding of market mechanics, and risk management rather than just curve-fitting a machine learning model.
- Emphasis on Inefficiencies: This is crucial. Many simple strategies look great pre-costs and fall apart once realistic frictions are applied. Testing this understanding is key.
- Market Regimes: Requiring simulations across different regimes tests the robustness and adaptability of the candidate's thinking and their strategy.
- Scoring Criteria: The scoring criteria are well-aligned with what a good quant strategy should achieve – consistent alpha generation in favorable conditions and capital preservation in unfavorable ones.
- Data Scale: Working with 1-minute data since 2012 is a significant data engineering challenge in itself, which tests practical skills.
Potential Challenges and Areas for Clarification/Refinement:
-
Time Commitment:
- Massive Undertaking: Sourcing, cleaning (if not provided clean), processing 1-minute data from 2012 for BTC/USD is a huge task. Then building a backtester, implementing rules, and running multiple simulations is even bigger.
- Candidate Filter: This will filter out many candidates who simply don't have 40-100+ hours to dedicate to a take-home test, even if they are highly skilled.
-
Ambiguity/Scope:
- "Viable Algorithmic Trading Simulation": "Viable" can be subjective. Does it mean profitable after costs? Or just a well-structured simulation environment even if the specific rules aren't world-beating?
- "Deterministic Engineered Rules": While "no prediction model" is clear, "deterministic engineered rules" could still be very broad. Are you looking for technical indicator-based rules (MA crossovers, RSI, Bollinger Bands), order flow imbalances, arbitrage-like logic (though harder without multiple exchanges)? Providing a few examples of types of rules might be helpful.
- Slippage Model: How sophisticated should this be? Fixed percentage? Variable based on order size/volatility? The latter is much harder.
- Automation Errors: How should these be modeled? Random chance of failed order? Latency spikes? This is very open-ended and hard to implement generically.
- Market Regimes: Should the candidate define these regimes themselves (e.g., using volatility clustering, trend detection), or will you provide pre-defined periods? The former adds another layer of complexity.
- Backtesting Framework: Are they expected to build one from scratch (very time-consuming) or can they use existing libraries (e.g.,
backtrader,zipline,vectorbtin Python)? Specify this. Building from scratch tests deeper understanding but takes much longer.
-
Data Provision:
- Are you providing the data? If not, sourcing reliable 1-minute BTC/USD data since 2012 can be a challenge in itself (cost, API limits, data gaps, data cleaning).
- If providing, ensure it's clean or specify that cleaning is part of the test.
-
Evaluation Focus:
- What's more important: the sophistication of the backtesting engine, the ingenuity of the rules, the thoroughness of the cost modeling, or the analysis of the results? The current setup tests all of these heavily.
Suggestions for Improvement/Refinement:
-
Clarify Scope & Expectations:
- Time Limit: Provide a realistic estimated time for completion (e.g., "We expect this to take X-Y hours"). This helps candidates gauge the depth required.
- Deliverables: What exactly do you want?
- Just the code?
- A written report detailing the methodology, rules, assumptions, and results?
- A presentation?
- "Viable": Define it. E.g., "A simulation that demonstrates positive expectancy over at least one favorable regime after costs," or "A well-structured backtest with clear logic, even if the P&L isn't stellar."
- Rule Examples: Give 1-2 examples of the type of deterministic rules you're imagining (e.g., "a momentum rule based on N-period returns," "a mean-reversion rule using Bollinger Bands").
- Backtesting Framework: Explicitly state if they can use libraries or must build from scratch. Using libraries is more practical for a test.
-
Reduce the Workload (without sacrificing core testing goals):
- Provide Clean Data: Offer a pre-cleaned dataset for specific, shorter periods representing different regimes (e.g., a 1-year bull run, a 1-year bear market, a 1-year choppy market). This drastically cuts down data prep time.
- Simplify Inefficiencies (Slightly):
- Give specific fee percentages (e.g., 0.1% taker, 0.05% maker).
- For slippage, suggest a simple model (e.g., "assume 0.05% slippage on all aggressive orders" or "slippage proportional to 1-min ATR").
- For automation errors, perhaps make it optional or suggest a very simple model (e.g., "assume 1% of orders fail").
- Specify if leverage is expected/allowed. If so, give a sample funding rate.
- Define Market Regimes: Provide the date ranges for the different regimes you want them to test on. "Run your simulation on Period A (YYYY-MM-DD to YYYY-MM-DD - Bullish), Period B (... - Bearish), Period C (... - Choppy)."
-
Focus the Evaluation:
- Decide what you most want to see. If it's strategy creativity and robustness, then provide a basic backtesting scaffold and the data. If it's engineering, then building the backtester is more key.
- Your current scoring is good, but you'll need a clear way to measure "consistent extracted value" and "controllable loss" (e.g., Sharpe ratio, Sortino ratio, max drawdown, profit factor per regime).
-
Tiered Approach (Optional):
- If you want to keep it broad, consider making some parts "core" and others "bonus."
- E.g., Core: Implement one simple deterministic strategy, account for fees and basic slippage on one bull and one bear period.
- Bonus: More sophisticated slippage, funding fees, automation errors, more regimes, multiple strategies.
Revised Test Example Snippet (Illustrative):
"Objective: Build an algorithmic trading simulation for BTC/USD using the provided 1-minute OHLC data.
Data:
- We provide 3 CSV files containing 1-minute OHLCV BTC/USD data for the following periods:
period_bull_2017.csv(Represents a strong bull market)period_bear_2018.csv(Represents a bear market)period_choppy_2019.csv(Represents a sideways/choppy market)
- Assume data is clean.
Requirements:
- Strategy:
- Develop 1-2 deterministic, rule-based trading strategies (e.g., based on moving averages, RSI, price channel breakouts, volatility).
- No predictive ML/AI models.
- Clearly document the logic and parameters of your chosen rule(s).
- Simulation Engine:
- You may use common Python libraries like
pandas,numpy. For backtesting, you can use libraries likebacktraderorvectorbt, or build a simple event-driven backtester. - Start with a simulated portfolio of $100,000 USD.
- No leverage unless you specifically model funding fees.
- You may use common Python libraries like
- Trading Inefficiencies:
- Trading Fees: Assume 0.1% per trade (covers both entry and exit, or 0.05% each way).
- Slippage: Model slippage as 0.05% of the transaction value for every market order.
- (Optional Bonus): If using leverage, model funding fees at an illustrative 0.01% per 8-hour period on the notional value of the leveraged position.
- (Optional Bonus): Briefly discuss how you might model automation errors.
- Execution & Analysis:
- Run your strategy simulation on each of the 3 provided market periods.
- Report standard performance metrics for each period: Total P&L, Sharpe Ratio (assume RF=0), Max Drawdown, Win Rate, Profit Factor.
- Analyze why your strategy performed as it did in each regime.
Scoring: Your submission will be evaluated on:
- Clarity and soundness of the strategy logic (30%)
- Correctness and robustness of the backtesting implementation (30%)
- Thoroughness in accounting for specified inefficiencies (20%)
- Quality of results analysis and insights into strategy behavior across regimes (20%)
Deliverables:
- Your Python code (well-commented).
- A brief report (e.g., Jupyter Notebook or PDF) detailing your strategy, implementation choices, results, and analysis.
Expected Time Commitment: We estimate this task should take approximately 8-12 hours of focused work."
This revised version is still challenging but more scoped and manageable, allowing candidates to showcase their core quant skills effectively within a reasonable timeframe. You can adjust the complexity based on the seniority of the role. For a very senior role, your original test (with a few clarifications on data and allowable tools) might be appropriate if you give them a week or more.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.