How the benchmarks are run

What the results can and cannot tell you

The benchmarks cover a small number of datasets, from a few countries, with a single setup. A model that does well here may do worse on your data, and the other way round. Performance depends heavily on the disease, the country, the level of detail and quality of the data, and how far ahead you forecast.

To choose a model for your own use, evaluate the candidates on your own data: in the DHIS2 Modeling App, which can evaluate and compare models, or with the chap eval command. Use the benchmarks to narrow down which models to try, not to make the final decision.

Who runs them

The Chap team runs every benchmark on its own benchmarking server, in the same way for every model. No results are self-reported by model authors. Benchmarking is opt-in: only models whose authors have opted in are benchmarked, and their new versions are benchmarked automatically once they are listed.

Datasets

Monthly data from Vietnam, Laos and Thailand, from the open climate-health-data repository.

Method

Each model is tested by backtesting. It is trained on the data up to a point in time and forecasts the next three months, and the forecasts are compared with what actually happened. This is repeated six times, moving the point in time forward one month each time, and the results are combined. See the evaluation pipeline for the details.

Metrics

Every run records the metrics the Chap platform computes, such as the continuous ranked probability score (CRPS), mean absolute error (MAE) and interval coverage. What each one measures is described in chap-core's metric definitions; the Metrics menu above each table shows the same descriptions.

Ranking

By default, each table shows the metrics the Chap platform shows by default, in its default order, and models are sorted by the first of them: currently CRPS on log-transformed case counts. This is a default, not a verdict on which model is best, so take it with a grain of salt. You can sort by any other metric, or pick the metrics shown.

We have not yet decided how to rank models across datasets, and more benchmarks are needed before the results say much. Look at several metrics, and above all at the ones that matter for how you will use the forecasts.