ChapDHIS2.orgCommunityGitHub
Chap Marketplace

Benchmarks

How the listed models perform on a few public datasets: a quick, rough picture of model performance, not a guide to which model to choose. How the benchmarks are run →

These results cover a few datasets with a single setup, and a model may do better or worse on your data. Evaluate the candidates on your own data before choosing one, and use these results to narrow down which to try.

Scores aggregated across all splits and organisation units; the best in each column is in bold. Compare with a baseline to see how much better or worse each model does than it.

Laos monthly · horizon 3

Dataset
Laos monthly
Horizon
3 months
Backtest splits
6
Harness
chap 2.6.0
4 of 4 configured models evaluated
#Model · pinVersionCRPS (log1p)Winkler 25-75 (log1p)MAERMSEMAPECoverage 10-90Status
1monthly · auto_arima_chapkit@49ee939stableStable · 1.0.40.77373.261145.1291.8141.6%0.4167Successful
2monthly_climate · chapkit_simple_multistep_model@409ec80stableStable · 0.1.50.90463.766132263.5235%0.3333Successful
3monthly_selfhistory · chapkit_simple_multistep_model@409ec80stableStable · 0.1.51.1274.663166.1340.1237.8%0.4271Successful
4monthly · chapkit_rwanda_malaria_bym_model@597636dstableStable · 0.1.62.29710.05194.1387.293.65%0.2951Successful

Auto-ARIMA

Successful run
Laos monthly · horizon 3 · rank 1
monthly · auto_arima_chapkit@49ee939 · 1.0.4
CRPS (log1p)
0.7737
Winkler 25-75 (log1p)
3.261
MAE
145.1
RMSE
291.8
MAPE
141.6%
Coverage 10-90
0.4167

Other recorded metrics

Coverage 25-75
0.3021
Proportion of observations within 25th-75th percentile (higher is better, up to the 0.5 target)
CRPS
125.8
Continuous Ranked Probability Score - measures calibration and sharpness
CRPS Normalized
0.04762
Normalized CRPS - CRPS divided by the range of observed values
Ratio Above Truth
0.3075
Proportion of forecast samples exceeding the observed value (0.5 = unbiased)
Winkler Score 10-90
1,096
Winkler score for 10th-90th percentile prediction interval
Winkler 10-90 (log1p)
6.908
Winkler score on log(1+x)-transformed values for 10-90 interval
Winkler Score 25-75
532.4
Winkler score for 25th-75th percentile prediction interval
1 of 4
Model page

Thailand monthly · horizon 3

Dataset
Thailand monthly
Horizon
3 months
Backtest splits
6
Harness
chap 2.6.0
4 of 4 configured models evaluated
#Model · pinVersionCRPS (log1p)Winkler 25-75 (log1p)MAERMSEMAPECoverage 10-90Status
1monthly · auto_arima_chapkit@49ee939stableStable · 1.0.40.36881.633.49110.4131.3%0.9221Successful
2monthly_climate · chapkit_simple_multistep_model@409ec80stableStable · 0.1.50.47192.03733.7116.8150.6%0.798Successful
3monthly_selfhistory · chapkit_simple_multistep_model@409ec80stableStable · 0.1.50.6322.6939.37122.7183.7%0.7958Successful
4monthly · chapkit_rwanda_malaria_bym_model@597636dstableStable · 0.1.60.95274.2547.53151.173.52%0.6162Successful

Auto-ARIMA

Successful run
Thailand monthly · horizon 3 · rank 1
monthly · auto_arima_chapkit@49ee939 · 1.0.4
CRPS (log1p)
0.3688
Winkler 25-75 (log1p)
1.6
MAE
33.49
RMSE
110.4
MAPE
131.3%
Coverage 10-90
0.9221

Other recorded metrics

Coverage 25-75
0.767
Proportion of observations within 25th-75th percentile (higher is better, up to the 0.5 target)
CRPS
28.94
Continuous Ranked Probability Score - measures calibration and sharpness
CRPS Normalized
0.006774
Normalized CRPS - CRPS divided by the range of observed values
Ratio Above Truth
0.4848
Proportion of forecast samples exceeding the observed value (0.5 = unbiased)
Winkler Score 10-90
232.8
Winkler score for 10th-90th percentile prediction interval
Winkler 10-90 (log1p)
2.769
Winkler score on log(1+x)-transformed values for 10-90 interval
Winkler Score 25-75
131.5
Winkler score for 25th-75th percentile prediction interval
1 of 4
Model page

Vietnam monthly · horizon 3

Dataset
Vietnam monthly
Horizon
3 months
Backtest splits
6
Harness
chap 2.6.0
4 of 4 configured models evaluated
#Model · pinVersionCRPS (log1p)Winkler 25-75 (log1p)MAERMSEMAPECoverage 10-90Status
1monthly · auto_arima_chapkit@49ee939stableStable · 1.0.40.27261.197153302.6111.3%0.7266Successful
2monthly_climate · chapkit_simple_multistep_model@409ec80stableStable · 0.1.50.51112.162132.9271167.1%0.5688Successful
3monthly_selfhistory · chapkit_simple_multistep_model@409ec80stableStable · 0.1.50.63642.671155.3305.1155.4%0.5988Successful
4monthly · chapkit_rwanda_malaria_bym_model@597636dstableStable · 0.1.60.98914.428210402.995.29%0.694Successful

Auto-ARIMA

Successful run
Vietnam monthly · horizon 3 · rank 1
monthly · auto_arima_chapkit@49ee939 · 1.0.4
CRPS (log1p)
0.2726
Winkler 25-75 (log1p)
1.197
MAE
153
RMSE
302.6
MAPE
111.3%
Coverage 10-90
0.7266

Other recorded metrics

Coverage 25-75
0.5802
Proportion of observations within 25th-75th percentile (higher is better, up to the 0.5 target)
CRPS
125.9
Continuous Ranked Probability Score - measures calibration and sharpness
CRPS Normalized
0.01734
Normalized CRPS - CRPS divided by the range of observed values
Ratio Above Truth
0.3986
Proportion of forecast samples exceeding the observed value (0.5 = unbiased)
Winkler Score 10-90
1,008
Winkler score for 10th-90th percentile prediction interval
Winkler 10-90 (log1p)
2.218
Winkler score on log(1+x)-transformed values for 10-90 interval
Winkler Score 25-75
551.6
Winkler score for 25th-75th percentile prediction interval
1 of 4
Model page
Each model's detail page compares its versions on every dataset.Browse models