Auditing Evaluation Leakage in Formula 1 Race-Strategy Machine Learning: A Track-Identity Confound and a Corrected Protocol

1. Introduction

Formula 1 teams make pit-stop and tyre-management decisions under significant time pressure, and a growing body of applied machine learning work uses lap-level telemetry (much of it sourced from the open FastF1 API) to model lap time degradation and pit-stop timing. A recurring methodological pattern in this literature, including in an earlier version of the pipeline this paper corrects, is to evaluate these models with a row-wise random train/test split over individual laps. Because laps from the same race and the same driver's stint are highly autocorrelated, this split lets information leak between train and test: a model can effectively interpolate within a race it has already partially seen, rather than generalize to conditions it has not.

This paper is a leakage audit, not primarily a new modeling technique. Its central contribution is a decomposition that isolates exactly how much of a plausible-looking result is attributable to (a) the row-wise split itself and (b) one-hot encoding driver and circuit identity as model features. We show that these two choices, independently or combined, are sufficient to reproduce a near-perfect lap-time R² that has essentially nothing to do with modeling tyre degradation. We then propose and validate an honest evaluation protocol: grouped cross-validation, with an explicit distinction between generalizing to an unseen circuit (track-agnostic) versus an unseen driver on a known circuit (track-aware).

We do not claim state-of-the-art predictive performance. On the pit-stop classification task specifically, our corrected classifier trails several recently published results (Section 2), and we report this plainly in Section 5 rather than around it. The contribution we do make is a methodology and an empirical demonstration, on a real dataset, of how large an evaluation-leakage confound can be in this application domain.

2. Related Work

Machine learning for lap-time prediction. Zhao (2024) trains a deep neural network to forecast qualifying lap times, categorizing historical data by driver and circuit; this specialization is functionally close to encoding driver and circuit identity as features, the same mechanism this paper identifies as the dominant confound. Brusik (2024) compares univariate and multivariate LSTM and LSTM-FCN architectures for multi-step lap-time forecasting on FastF1 data spanning 2018–2023 (136,122 laps), using a strict chronological split; error analysis identifies track-status changes as the primary source of large forecasting errors.

Deep learning for pit-stop prediction. Sasikumar, Leema, and Balakrishnan (2025) benchmark five deep sequence and convolutional architectures on FastF1-sourced pit-stop prediction, evaluated on a temporal holdout. Their best model, Bi-LSTM, reaches precision=0.77, recall=0.86, F1=0.81, and ROC-AUC=0.988. Our corrected pit-stop classifier (F1=0.303, ROC-AUC=0.888) does not match it. Fatima and Johrendt's Deep-Racing (2023) reports F1=0.67, and García Tejada's SVM-based classifier (2023) reports F1=0.621.

Explainable F1 strategy models. Todd et al. (2025) apply explainable deep learning and XGBoost to tyre energy degradation prediction. Thomas et al. (2025) target the sequential strategy decision with a reinforcement learning policy (RSRL), achieving an average finishing position of P5.33 on the 2023 Bahrain Grand Prix. Our use of SHAP (TreeExplainer) is in the same interpretability spirit, applied to a simpler supervised tabular formulation.

Positioning. We position this paper's contribution as methodological: a race- and driver-grouped cross-validation protocol, an explicit decomposition of how leaky evaluation inflates results in this domain, honest non-trivial baselines, and a cross-season generalization test, rather than a claim of predictive superiority over any cited comparator.

3. Methodology

3.1 Data

We use the FastF1 Python API to fetch lap-by-lap timing data for all 22 races of the 2023 Formula 1 season (24,420 raw laps). Weather is merged onto lap data with a nearest-match merge on timestamp (5-minute tolerance) before races are concatenated. For the cross-season generalization test, we additionally fetch six 2024 races (Bahrain, Japan, Monaco, Great Britain, Singapore, Brazil) chosen to reuse 2023 circuit names. Laps are cleaned by requiring an accurate-lap flag and dropping rows missing key fields, leaving 20,447 laps used throughout the paper.

3.2 Features and targets

Numeric features: tyre life, stint, lap number, lap-time delta from the previous lap, an approximate gap to the car ahead, track/air temperature, rainfall, position, humidity, and wind speed. Categorical features: tyre compound and track status. The regression target is lap time in seconds. The classification target, PitStopNextLap, is defined as (NextTyreLife < TyreLife) AND (TyreLife > 2), shifted within (Grand Prix, Driver) groups. Under this definition, 707 of 20,447 laps (3.5%) are positive.

3.3 Leakage decomposition

We evaluate four scenarios for the lap-time regression task: A — random row-wise split, driver and GP included (reproduces the original leaky setup). B — grouped by race, driver and GP included (GP unseen at test). C — grouped by race, no identity features ("track-agnostic"). D — grouped by driver, GP kept as a feature ("track-aware"). Scenario C is our primary track-agnostic result; Scenario D is our primary track-aware result and the version used for SHAP analysis, the deployed model, and the 2024 generalization test.

3.4 Baselines, model, explainability, and generalization test

For lap-time regression we compare XGBoost against a persistence baseline and linear regression. For pit-stop classification we compare XGBoost against a majority-class baseline and logistic regression with balanced class weights. We compute SHAP values (TreeExplainer) for both final (track-aware) models. Both track-aware models, trained once on the full 2023 season, are evaluated without retraining on the 6-race 2024 holdout (6,133 laps after cleaning).

4. Results

4.1 Leakage decomposition

Table 1 confirms that the original evaluation's near-perfect R² is reproducible almost exactly once both the row-wise split and identity features are restored (Scenario A), and that identity features alone provide no benefit once the split is honest and the circuit is genuinely unseen (Scenario B). Scenario D shows that keeping circuit identity is not illegitimate in general, and yields a strong, honestly-validated result.

4.2 Lap time regression

The track-agnostic model performs worse than predicting the training-set mean on unseen circuits (R² < 0), because absolute lap time is dominated by track length and layout. The track-aware model is a real improvement over a strong persistence baseline (MAE 0.54s vs. 0.61s).

4.3 Pit-stop classification

The track-agnostic classifier barely beats a majority-class baseline on F1 and is actually worse than logistic regression on ROC-AUC (0.645 vs. 0.669), while the track-aware classifier is a clear improvement over both non-trivial baselines.

4.4 Explainability

Figure 1 shows that the track-aware lap-time model's global feature importance is dominated by circuit identity (GP) and weather. Figure 2 shows the opposite, more domain-sensible pattern for the pit-stop model: tyre life and stint dominate, with circuit identity contributing comparatively little.

Figure 1. SHAP analysis, track-aware lap-time model.

Figure 2. SHAP analysis, track-aware pit-stop model.

4.5 Cross-season generalization

Both track-aware models, trained only on 2023 data, degrade substantially when evaluated without retraining on the 2024 holdout.

5. Limitations

The pit-stop classifier trails the current published field, and we do not soften this. Our track-aware classifier reaches F1=0.303 and ROC-AUC=0.888, behind Sasikumar et al.'s Bi-LSTM (2025: F1=0.81), Fatima and Johrendt's Deep-Racing (2023: F1=0.67), and García Tejada's SVM (2023: F1=0.621). This paper's contribution is the leakage audit and corrected evaluation protocol, not a claim of competing with the best published pit-stop predictors.

The track-aware framing is narrower than "lap time prediction" as originally framed. It generalizes to an unseen driver on a known circuit, not to an unseen circuit.

Cross-season generalization is weak without retraining. Neither model should be deployed across a season boundary without recalibration.

Safety cars, red flags, and team orders are not modeled. Track status is a coarse proxy at best.

Driver and constructor identity are deliberately excluded from the pit-stop model and the lap-time model's driver dimension, to avoid the identity-memorization problem this paper documents.

Training data is single-season. The 2024 data is used only for evaluation, not training.

The lap-time delta feature is quasi-derived from the regression target. An ablation removing it changed the track-agnostic result by less than 0.03 R².

6. Conclusion

We audited an existing Formula 1 lap-time and pit-stop prediction pipeline and found that its previously reported near-perfect performance was substantially an artifact of a row-wise random train/test split combined with one-hot-encoded driver and circuit identity, not genuine modeling of race conditions. A track-aware reformulation restores strong, honestly-validated performance for lap time, and a more modest but still non-trivial improvement over baselines for pit-stop classification, which nonetheless trails the current published state of the art. We release the corrected pipeline, the leakage decomposition, and all metrics reported here to support more reproducible evaluation practice in this application area.

References

1. Zhao, Z. (2024). Deep Neural Network-based Lap Time Forecasting of Formula 1 Racing. Applied and Computational Engineering, 47, 61–66. doi:10.54254/2755-2721/47/20241191
2. Brusik, F. (2024). Predicting Lap Times in a Formula 1 Race Using Deep Learning Algorithms [Master's thesis, Tilburg University].
3. Sasikumar, A., Leema, A. A., & Balakrishnan, P. (2025). Data-driven pit stop decision support for Formula 1 using deep learning models. Frontiers in Artificial Intelligence, 8, 1673148. doi:10.3389/frai.2025.1673148
4. Fatima, S. S. W., & Johrendt, J. (2023). Deep-Racing: An Embedded Deep Neural Network (EDNN) Model to Predict the Winning Strategy in Formula One Racing. International Journal of Machine Learning, 13(3), 97–103. doi:10.18178/ijml.2023.13.3.1135
5. García Tejada, L. (2023). Applying Machine Learning to Forecast Formula 1 Race Outcomes [Master's thesis, Aalto University].
6. Todd, J., Jiang, J., Russo, A., Winkler, S., Sale, S., McMillan, J., & Rago, A. (2025). Explainable Time Series Prediction of Tyre Energy in Formula One Race Strategy. arXiv:2501.04067
7. Thomas, D., Jiang, J., Kori, A., Russo, A., Winkler, S., Sale, S., McMillan, J., Belardinelli, F., & Rago, A. (2025). Explainable Reinforcement Learning for Formula One Race Strategy. arXiv:2501.04068
8. Sander, T. (2023). FastF1: Python API for Accessing Formula 1 Telemetry and Timing Data (v3.3) [GitHub repository]. github.com/theOehrly/Fast-F1