I have an XGBoost model that is trained on predicting the likelyhood to get a lease.…
- User
I have an XGBoost model that is trained on predicting the likelyhood to get a lease. I've trained it on multiple time windows (i.e. a lease will occur in the next 28/56/84 days in the future). When I try to evaluate results the longer the window results in better performance but it's not strongly ranking what I believe to be key features like price. What's the best way to objectively evaluate which approach is best?
- Assistant
This is an excellent and classic problem in predictive modeling. Your observation is astute: raw statistical performance (like AUC) doesn't always align with business intuition or perceived model sensibility.
The core of the issue is that your three models (28, 56, 84-day windows) are solving three slightly different problems, and "best" depends entirely on the business question you're trying to answer.
Let's break down why you're seeing this behavior and then lay out a framework for objective evaluation.
Why This is Happening
-
The Problem Gets "Easier": A longer time window increases the base rate of the positive class (a lease occurs). There's simply more opportunity for a lease to happen in 84 days than in 28. This higher base rate naturally inflates metrics like Accuracy and can make ranking easier for the model, leading to a higher AUC. The model can rely on more general, long-term signals.
-
The Nature of Features Changes with Time:
- 28-Day Window (Urgency): For a short-term prediction, highly dynamic and immediate factors are critical. Price is paramount. Is it a good deal right now? What's the current demand? Are there competing listings that just opened up? The model correctly learns that price is a key driver for an imminent decision.
- 84-Day Window (Desirability): For a long-term prediction, the model is predicting general desirability over a season. Factors like location, property type, number of bedrooms, and seasonality become much more dominant. A great apartment in a prime location is very likely to be leased at some point in the next 3 months, almost regardless of a minor price variation. The model correctly learns that these stable, long-term features are better predictors of an eventual lease, thus down-weighting the importance of immediate price.
The 84-day model isn't "wrong" for de-prioritizing price; it's simply reflecting that over a long horizon, a property's fundamental characteristics outweigh its immediate price tag.
The Best Way to Objectively Evaluate
To choose the best approach, you need a multi-faceted evaluation framework that goes beyond a single metric and connects directly to business utility.
Here is a 3-pillar framework for your evaluation:
Pillar 1: Rigorous Statistical Performance (Beyond AUC)
You need to use metrics that are robust to class imbalance (leases are likely a rarer event than non-leases) and measure what you actually care about.
- Precision-Recall AUC (AUC-PR): This is often much more informative than ROC AUC for imbalanced datasets. It answers the question: "Of the properties the model flagged as likely to lease, what fraction actually did?" High precision is critical if taking action is costly (e.g., offering a discount). Compare the AUC-PR for all three models.
- Calibration Plots & Brier Score: How trustworthy are the probabilities? A model that predicts 80% should be right about 80% of the time. A well-calibrated model is crucial if you're using the probability score itself (e.g., to tier your marketing spend). Plot calibration curves for all three models and compare their Brier scores (lower is better). The 84-day model might be overconfident.
- Top Decile Analysis: Instead of a global metric, look at the performance on the slice of data you'd actually use. For the top 10% of properties ranked by each model, what is the actual lease rate (the "lift")? The model that finds more "needles in the haystack" in its top predictions is often the most useful.
Pillar 2: Business Utility Simulation
This is the most critical pillar. Frame the evaluation around the specific business action the model will drive.
-
Define the Business Goal: What will you do with this prediction?
- A) Proactive Marketing: Identify at-risk properties to get more marketing attention.
- B) Dynamic Pricing: Identify properties that are unlikely to lease and recommend a price drop.
- C) Inventory Management: Forecast future occupancy for financial planning.
-
Cost-Benefit Analysis: This is the ultimate objective measure. Assign dollar values and simulate the outcome.
- Scenario (using goal B - Dynamic Pricing):
- Action: If a property's predicted lease probability is below a certain threshold (e.g., 50%), you recommend a 5% price reduction.
- Cost of Action: The 5% revenue loss on properties that would have leased anyway (False Positives in this framing).
- Benefit of Action: The revenue gained from properties that leased because of the price drop, which otherwise wouldn't have (True Positives).
- Run the Simulation: On a hold-out test set, apply this logic for each of your three models (28, 56, 84 days). Which model generates the highest net profit? The 28-day model might win here because it's better at identifying imminent risk where a price change has the most immediate impact.
- Scenario (using goal B - Dynamic Pricing):
-
Lift and Gain Charts: These charts visually show the business value.
- Gain Chart: "If we contact the top X% of leads identified by the model, what percentage of the total actual leases will we capture?"
- Compare the charts for all three models. The 28-day model might show a much steeper curve, indicating it's better at identifying the "hottest" leads who will act soon.
Pillar 3: Model Interpretability and Stability
This pillar addresses your concern about feature importance directly and builds trust in the chosen model.
- SHAP (SHapley Additive exPlanations): Go beyond global feature importance.
- Global Summary: Use a SHAP summary plot (beeswarm) for each model. This will confirm your finding about price but will also show the distribution and impact of other features.
- Local Explanations: Pick a few specific properties from your test set. For a property that did lease quickly, what reasons did the 28-day model give? What about the 84-day model? This will help you understand how they think differently and which model's logic feels more sound for your business purpose.
- Partial Dependence Plots (PDP): For each model, plot the predicted probability of a lease as a function of
price, holding all other features constant. You will likely see that the slope of the price curve is much steeper for the 28-day model, visually confirming its higher importance in that context.
Recommended Action Plan
- Clearly Define the Business Action: Write down the single most important decision this model will inform. This will be your North Star.
- Use a Held-Out Test Set: Ensure you have a clean, out-of-time test set to perform all evaluations.
- Calculate the Full Suite of Metrics: For each model (28, 56, 84), compute:
- ROC AUC and Precision-Recall AUC.
- Brier Score.
- Lift in the top decile.
- Run a Financial Simulation: Perform the cost-benefit analysis described in Pillar 2 based on your defined business action. This will likely give you the clearest "winner."
- Analyze with SHAP: Use SHAP summary plots to understand the overall logic and individual explanations to validate that the chosen model behaves sensibly on key examples.
Conclusion:
The "best" model isn't the one with the highest AUC. The "best" model is the one that maximizes utility for your specific business objective and decision-making timeline.
It's highly probable that if your goal is to take immediate action (like a price drop or marketing push this week), the 28-day model will be the most valuable, even with a lower overall AUC, because it's optimized for predicting near-term events where factors like price are most critical. The 84-day model is better suited for long-range strategic forecasting. By aligning your evaluation with your business action, you'll find the objective answer you're looking for.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.