Evaluations

We computed for every projection evaluations metrics against observed before and after downscaling against observations in the evaluation period. Evaluation metrics include Pearson’s correlation coefficient (CC), root mean square error (RMSE), standard deviation of errors (SDE), and mean bias (bias).

Code

t <- read_tsv("outputs/chelsa_eval.tsv") %>%
  group_by(area, variable) %>%
  summarise(
    CC = paste0(round(mean(CC), 2), " (", round(sd(CC), 2), ")"),
    RMSE = paste0(round(mean(RMSE), 2), " (", round(sd(RMSE), 2), ")")
  ) %>%
  mutate(area = ifelse(area == "Côte-d'Ivoire", "Ivory-Coast", area)) %>%
  mutate(area = gsub("-", " ", area)) %>%
  mutate(variable = recode(variable,
    "pr" = "CHIRPS precipitation",
    "tas" = "GSHTD temperature"
  )) %>%
  rename(Country = area, Evaluation = variable)
write_tsv(t, "figures/ts01.tsv")
read_tsv("figures/ts01.tsv") %>%
  kable(caption = "Table S1. Evaluation of baseline CHELSA 2 data between 2006 and 2019  against CHIRPS precipitation and GSHTD temperature data in Ivory Coast, French Guiana and New Caledonia. Numbers indicate root mean square error (RMSE) or correlation coefficient (CC) with mean followed by standard deviation in parentheses. Assessments are averaged across months.") # nolint

Table S1. Evaluation of baseline CHELSA 2 data between 2006 and 2019 against CHIRPS precipitation and GSHTD temperature data in Ivory Coast, French Guiana and New Caledonia. Numbers indicate root mean square error (RMSE) or correlation coefficient (CC) with mean followed by standard deviation in parentheses. Assessments are averaged across months.

Country	Evaluation	CC	RMSE
Ivory Coast	CHIRPS precipitation	0.62 (0.22)	51.78 (35.07)
Ivory Coast	GSHTD temperature	0.77 (0.22)	0.62 (0.25)
French Guiana	CHIRPS precipitation	0.21 (0.36)	94.37 (53.18)
French Guiana	GSHTD temperature	0.52 (0.12)	0.91 (0.2)
New Caledonia	CHIRPS precipitation	0.08 (0.08)	136.49 (59.1)
New Caledonia	GSHTD temperature	0.73 (0.09)	1.51 (0.47)

Code

g <- read_tsv("results/evaluation/evaluations.tsv") %>%
  mutate(area = ifelse(area == "Côte-d'Ivoire", "Ivory-Coast", area)) %>%
  mutate(area = gsub("-", " ", area)) %>%
  filter(metric %in% c("CC", "RMSE"), variable %in% c("tas", "pr")) %>%
  filter(type == "downscaled", base_eval != "chelsa2") %>%
  mutate(base_eval = recode(base_eval,
    "chirps" = "precipitation (mm)",
    "gshtd" = "temperature (°C)"
  )) %>%
  mutate(area = paste(area, base_eval)) %>%
  select(-base_eval) %>%
  mutate(experiment = recode(experiment,
    "rcp26" = "2.6~W~m^{-~2}",
    "rcp85" = "8.5~W~m^{-~2}",
    "ssp126" = "2.6~W~m^{-~2}",
    "ssp585" = "8.5~W~m^{-~2}"
  )) %>%
  mutate(origin = paste0(origin, "~", experiment)) %>%
  select(-experiment) %>%
  pivot_wider(names_from = metric, values_from = value) %>%
  group_by(model, variable, origin, area) %>%
  summarise(
    sd_m = sd(RMSE) / 2, value_m = mean(RMSE),
    sd_c = sd(CC) / 2, value_c = mean(CC)
  ) %>%
  mutate(area = factor(area,
    levels = c(
      "French Guiana precipitation (mm)",
      "Ivory Coast precipitation (mm)",
      "New Caledonia precipitation (mm)",
      "French Guiana temperature (°C)",
      "Ivory Coast temperature (°C)",
      "New Caledonia temperature (°C)"
    )
  )) %>%
  ggplot(aes(value_m, value_c, color = origin)) +
  geom_linerange(aes(xmin = value_m - sd_m, xmax = value_m + sd_m),
    alpha = .2
  ) +
  geom_linerange(aes(ymin = value_c - sd_c, ymax = value_c + sd_c),
    alpha = .2
  ) +
  geom_line(aes(group = model), alpha = .3, col = "black") +
  geom_point() +
  facet_wrap(~area, scales = "free") +
  theme_bw() +
  theme(legend.position = "bottom") +
  scale_color_manual("",
    labels = scales::parse_format(),
    values = c("#95B4CC", "#0018A9", "#FEC9C9", "#ED1C23")
  ) +
  xlab("Root Mean Square Error (RMSE)") +
  ylab("Correlation Coefficient (CC)") +
  scale_y_reverse()
ggsave("figures/f03.png", g, dpi = 800, bg = "white", width = 8)
g

Fig 3. Evaluation of downscaled projections from global (CMIP6) and regional (CORDEX) climate models. Monthly precipitation and mean temperature from climate models were downscaled against CHELSA2 data for the period 1980-2005 for French Guiana, Ivory Coast and New Caledonia. We evaluated downscaled projected monthly precipitation against CHIRPS data and downscaled projected monthly mean temperature against GSHTD data for the period 2006-2019. The x-axis represents the root mean square error (RMSE) and the y-axis the correlation coefficient (CC), such that the best evaluation is located in the lower left corner with a RMSEP of 0 and a CC of 1. Each point represents the mean evaluation across months of a single climate model for scenario 2.6 or 8.5, with intervals around representing half the standard deviation across months. Blue dots represent CMIP6 global climate models with scenario 2.6 in light blue and 8.5 in dark blue, while red dots represent CORDEX regional climate models with scenario 2.6 in light red and 8.5 in dark red. Grey lines connecting two dots represent matching scenarios for a single model. Zoom on model mean evaluations can be found in Supplementary Figure S3, while details on model evaluations across months can be found in Supplementary Figure S4. The evaluations show that we do not have a single best climate model, but that the best climate models are context dependent, which underlines the importance of the ensemble forecasting approach, as shown in Fig. 4.

Code

g <- read_tsv("results/evaluation/evaluations.tsv") %>%
  mutate(area = ifelse(area == "Côte-d'Ivoire", "Ivory-Coast", area)) %>%
  mutate(area = gsub("-", " ", area)) %>%
  filter(metric %in% c("CC", "RMSE"), variable %in% c("tas", "pr")) %>%
  filter(type == "downscaled", base_eval != "chelsa2") %>%
  mutate(base_eval = recode(base_eval,
    "chirps" = "precipitation (mm)",
    "gshtd" = "temperature (°C)"
  )) %>%
  mutate(area = paste(area, base_eval)) %>%
  select(-base_eval) %>%
  mutate(experiment = recode(experiment,
    "rcp26" = "2.6~W~m^{-~2}",
    "rcp85" = "8.5~W~m^{-~2}",
    "ssp126" = "2.6~W~m^{-~2}",
    "ssp585" = "8.5~W~m^{-~2}"
  )) %>%
  mutate(origin = paste0(origin, "~", experiment)) %>%
  select(-experiment) %>%
  pivot_wider(names_from = metric, values_from = value) %>%
  group_by(model, variable, origin, area) %>%
  summarise(
    sd_m = sd(RMSE), value_m = mean(RMSE),
    sd_c = sd(CC), value_c = mean(CC)
  ) %>%
  mutate(area = factor(area,
    levels = c(
      "French Guiana precipitation (mm)",
      "Ivory Coast precipitation (mm)",
      "New Caledonia precipitation (mm)",
      "French Guiana temperature (°C)",
      "Ivory Coast temperature (°C)",
      "New Caledonia temperature (°C)"
    )
  )) %>%
  ggplot(aes(value_m, value_c, color = origin)) +
  geom_line(aes(group = model), alpha = .3, col = "black") +
  geom_point() +
  facet_wrap(~area, scales = "free") +
  theme_bw() +
  theme(legend.position = "bottom") +
  scale_color_manual("",
    labels = scales::parse_format(),
    values = c("#95B4CC", "#0018A9", "#FEC9C9", "#ED1C23")
  ) +
  xlab("Root Mean Square Error (RMSE)") +
  ylab("Correlation Coefficient (CC)") +
  scale_y_reverse()
ggsave("figures/fs03.png", g, dpi = 800, bg = "white", width = 8)
g

Fig S3. Evaluation of downscaled projections from global (CMIP6) and regional (CORDEX) climate models. Monthly precipitation and mean temperature from climate models were downscaled against CHELSA2 data for the period 1980-2005 for French Guiana, Ivory Coast and New Caledonia. We evaluated downscaled projected monthly precipitation against CHIRPS data and downscaled projected monthly mean temperature against GSHTD data for the period 2006-2019. The x-axis represents the root mean square error (RMSE) and the y-axis the correlation coefficient (CC), such that the best evaluation is located in the lower left corner with a RMSEP of 0 and a CC of 1. Each point represents the mean evaluation across months of a single climate model for scenario 2.6 or 8.5. Blue dots represent CMIP6 global climate models with scenario 2.6 in light blue and 8.5 in dark blue, while red dots represent CORDEX regional climate models with scenario 2.6 in light red and 8.5 in dark red. Grey lines connecting two dots represent matching scenarios for a single model.

Code

g <- read_tsv("results/evaluation/evaluations.tsv") %>%
  mutate(area = ifelse(area == "Côte-d'Ivoire", "Ivory-Coast", area)) %>%
  mutate(area = gsub("-", " ", area)) %>%
  filter(metric %in% c("CC", "RMSE"), variable %in% c("tas", "pr")) %>%
  filter(type == "downscaled", base_eval != "chelsa2") %>%
  filter(experiment %in% c("rcp26", "ssp126")) %>%
  mutate(base_eval = recode(base_eval,
    "chirps" = "precipitation (mm)",
    "gshtd" = "temperature (°C)"
  )) %>%
  mutate(area = paste(area, base_eval)) %>%
  select(-base_eval) %>%
  filter(month %in% c(6, 12)) %>%
  mutate(month = recode(as.character(month),
    "6" = "June",
    "12" = "December"
  )) %>%
  pivot_wider(names_from = metric, values_from = value) %>%
  group_by(model, variable, origin, area) %>%
  mutate(area = factor(area,
    levels = c(
      "French Guiana precipitation (mm)",
      "Ivory Coast precipitation (mm)",
      "New Caledonia precipitation (mm)",
      "French Guiana temperature (°C)",
      "Ivory Coast temperature (°C)",
      "New Caledonia temperature (°C)"
    )
  )) %>%
  mutate(origin = paste0(origin, "~2.6~W~m^{-~2}~", month)) %>%
  mutate(origin = factor(origin,
    levels = c(
      "CMIP6~2.6~W~m^{-~2}~June",
      "CMIP6~2.6~W~m^{-~2}~December",
      "CORDEX~2.6~W~m^{-~2}~June",
      "CORDEX~2.6~W~m^{-~2}~December"
    )
  )) %>%
  ggplot(aes(RMSE, CC, color = origin)) +
  geom_line(aes(group = model), alpha = .3, col = "black") +
  geom_point() +
  facet_wrap(~area, scales = "free") +
  theme_bw() +
  theme(legend.position = "bottom") +
  scale_color_manual("",
    labels = scales::parse_format(),
    values = c("#95B4CC", "#0018A9", "#FEC9C9", "#ED1C23")
  ) +
  xlab("Root Mean Square Error (RMSE)") +
  ylab("Correlation Coefficient (CC)") +
  scale_y_reverse() +
  scale_x_log10() +
  guides(col = guide_legend(nrow = 2))
ggsave("figures/fs04.png", g, dpi = 800, bg = "white", width = 8)
g

Fig S4. Evaluation of downscaled projections from global (CMIP6) and regional (CORDEX) climate models. Monthly precipitation and mean temperature from climate models were downscaled against CHELSA2 data for the period 1980-2005 for French Guiana, Ivory Coast and New Caledonia. We evaluated downscaled projected monthly precipitation against CHIRPS data and downscaled projected monthly mean temperature against GSHTD data for the period 2006-2019. The x-axis represents the root mean square error (RMSE) and the y-axis the correlation coefficient (CC), such that the best evaluation is located in the lower left corner with a RMSEP of 0 and a CC of 1. Each point represents the evaluation for June or December of a single climate model for scenario 2.6. Blue dots represent CMIP6 global climate models with June in light blue and December in dark blue, while red dots represent CORDEX regional climate models with June in light red and December in dark red. Grey lines connecting two dots represent matching scenarios for a single model.

Code

t <- read_tsv("results/evaluation/evaluations.tsv") %>%
  mutate(area = ifelse(area == "Côte-d'Ivoire", "Ivory-Coast", area)) %>%
  mutate(area = gsub("-", " ", area)) %>%
  filter(
    variable %in% c("tas", "pr"), type == "downscaled",
    metric %in% c("RMSE", "CC")
  ) %>%
  mutate(variable = recode(variable,
    "pr" = "precipitation",
    "tas" = "temperature"
  )) %>%
  group_by(area, origin, type, variable, base_eval, metric, model) %>%
  summarise(value = mean(value)) %>% # inter-months
  group_by(area, origin, type, variable, base_eval, metric) %>%
  summarise(sd = sd(value), mean = mean(value)) %>% # inter-model
  mutate(value = paste0(round(mean, 3), " (", round(sd, 3), ")")) %>%
  select(-sd, -mean) %>%
  pivot_wider(names_from = origin, values_from = value) %>%
  ungroup() %>%
  filter(base_eval != "chelsa2") %>%
  mutate(base_eval = toupper(base_eval)) %>%
  mutate(Evaluation = paste(base_eval, variable)) %>%
  rename(Country = area, Metric = metric) %>%
  select(Country, Evaluation, Metric, CMIP6, CORDEX) %>%
  arrange(Country, Evaluation, Metric)
write_tsv(t, "figures/t01.tsv")
read_tsv("figures/t01.tsv") %>%
  kable(caption = "Table 1. Mean evaluation of downscaled CMIP6 and CORDEX projections between 2006 and 2019 against CHIRPS precipitation and GSHTD temperature data in Ivory Coast, French Guiana and New Caledonia. Numbers indicate root mean square error (RMSE) or correlation coefficient (CC) with mean followed by standard deviation in parentheses. CMIP6 and CORDEX assessments are averaged across months, models and scenarios.") # nolint

Table 1. Mean evaluation of downscaled CMIP6 and CORDEX projections between 2006 and 2019 against CHIRPS precipitation and GSHTD temperature data in Ivory Coast, French Guiana and New Caledonia. Numbers indicate root mean square error (RMSE) or correlation coefficient (CC) with mean followed by standard deviation in parentheses. CMIP6 and CORDEX assessments are averaged across months, models and scenarios.

Country	Evaluation	Metric	CMIP6	CORDEX
French Guiana	CHIRPS precipitation	CC	0.663 (0.069)	0.596 (0.042)
French Guiana	CHIRPS precipitation	RMSE	56.059 (17.183)	63.97 (9.759)
French Guiana	GSHTD temperature	CC	0.336 (0.032)	0.316 (0.01)
French Guiana	GSHTD temperature	RMSE	0.569 (0.078)	0.578 (0.048)
Ivory Coast	CHIRPS precipitation	CC	0.909 (0.024)	0.894 (0.019)
Ivory Coast	CHIRPS precipitation	RMSE	21.873 (3.409)	24.498 (2.431)
Ivory Coast	GSHTD temperature	CC	0.662 (0.028)	0.671 (0.014)
Ivory Coast	GSHTD temperature	RMSE	0.674 (0.07)	0.747 (0.085)
New Caledonia	CHIRPS precipitation	CC	0.313 (0.019)	0.304 (0.009)
New Caledonia	CHIRPS precipitation	RMSE	67.197 (10.456)	70.323 (4.996)
New Caledonia	GSHTD temperature	CC	0.957 (0)	0.957 (0.001)
New Caledonia	GSHTD temperature	RMSE	0.667 (0.071)	0.742 (0.044)