Percentage of EFAS Formal Flood Notifications for which feedback was provided for 2020, aggregated per country (in colour). The total number of Formal Flood Notifications sent for each country during 2020 is shown on the map. EFAS partner countries for which no Formal Flood Notifications were issued during 2020 are shaded in grey.
Percentage of EFAS Formal Flood Notifications for which feedback was provided for 2020, aggregated per country (in colour). The total number of Formal Flood Notifications sent for each country during 2020 is shown on the map. EFAS partner countries for which no Formal Flood Notifications were issued during 2020 are shaded in grey.

by Marc Girons Lopez, EFAS Dissemination Centre

Formal Flood Notifications are the core of EFAS and it is therefore important that they are as accurate and relevant to the EFAS partners as possible. To this end, partner feedback is collected and analysed yearly. This way, potential weak spots can be identified and future developments of EFAS prioritised. The feedback submitted during 2020 is presented and analysed here together with feedback statistics from previous years (2016 – 2019).

With the implementation of the new system in 2019, the entire feedback collection system is now integrated in the EFAS-IS platform and partners may submit feedback directly from the link embedded in Formal Flood Notifications. If no feedback is provided within a reasonable time, a reminder is sent to the relevant partner. Quality control managers at Rijksvaterstaat (RWS), Slovak hydrometeorological institute (SHMU), and Swedish Meteorological and Hydrological Institute (SMHI) are responsible for monitoring the collection of feedback for Formal Flood Notifications. 

Even if the following analysis focuses on feedback provided for Formal Flood Notifications sent during 2020, it builds up on feedback statistics since 2016. However, since feedback collection has been continuously improving throughout the years, the completeness of some questions in this analysis may vary. Additionally, since some of the questions in the feedback form are not mandatory, some statistics may be based on a lower number of responses.

 

Summary of received feedback during 2020

A total of 240 Formal Flood Notifications were sent out during 2020 and 99 feedback reports were received, which accounts for 41% of all issued Formal Flood Notifications. This is the same percentage as for 2019 but a decrease compared to the year before (71%). Even if no detailed survey was performed on the reasons why partners did (not) provide feedback, it is conceivable that the decrease and following stagnation in feedback reporting was due to, first, the transition to a new feedback reporting system and, second, the significant increase in the number of Formal Flood Notifications being sent, which puts more load on partners. Even so, the provided feedback rate varied significantly among EFAS partners.

Issued EFAS Formal Flood Notifications and received feedback reports
Issued EFAS Formal Flood Notifications and received feedback reports.

The initial question in the feedback form is whether or not a flood event was observed in connection with a given Formal Flood Notification. The definition of a flood event is included in the question to help partners assess the event (i.e. return period equal or larger than 2 years). The 2-year return period was chosen as a definition for flood here as it allows to differentiate between correct rejections and flood events that happened but did not reach the 5-year return period threshold used in EFAS. In total, 38 out of 99 participants (38%) answered that a flood event was observed after a Formal Flood Notification had been sent out. This value is significantly lower than that from 2019 (61%), indicating an increased number of false alarms.

Participants responses to the question “Was the flood event observed?” of the feedback form
Participants responses to the question “Was the flood event observed?” of the feedback form.

The feedback form is adapted to whether a flood event was observed in connection to a given Formal Flood Notification or not, so relevant information for each case can be collected. Next, we are going to cover the feedback from those notifications for which a flood event was observed.

Feedback from observed flood events

Most of the participants who answered that a flood event had indeed occurred in connection to an issued Formal Flood Notification rated the accuracy of the EFAS information in terms of location as “As indicated in EFAS information” (76%). This is an improvement from 2019 (61%) and is related to a decrease in reported events being “In the wider region”. Overall, the location accuracy during 2020 was comparable to that of 2017 and 2018, and significantly better than that of 2016 and 2019.
 

EFAS performance in terms of accurately predicting the location of an event.
EFAS performance in terms of accurately predicting the location of an event.

Regarding the timing of the onset of the event, 34% of the participants stated that the start of the flood event happened on the day predicted by EFAS. This represents a significant decrease with respect to 2019 (61%) and 2016 (66%), which corresponds to an increase in those events that started 1-2 days earlier than predicted and ≥ 3 days later than predicted. Even if no clear pattern in forecast timing can be identified for the reported events in 2020 as a whole, the increase in the number of events that started earlier than EFAS predicted may limit reaction time, especially since Formal Flood Notifications may be sent with relatively short notice (at least with 48 hours in advance of the predicted start of an event).

EFAS performance in terms of accurately predicting the onset time of an event.
EFAS performance in terms of accurately predicting the onset time of an event.

Looking instead at the timing of the peak flow, the difference is in this case not so large with respect to 2019. Indeed, a similar number of participants stated that the flood peak was observed on the predicted day (40%) or 1-2 days earlier than predicted (25%). Nevertheless, while in 2019 EFAS had a clear tendency to predict the peak flow too late, under 2020 EFAS predicted the peak too early for a significant number of reported events (21%), which corresponds well with the statistics shown for the timing of onset. This question was first introduced with the new integrated feedback reporting system and no data is therefore available prior to 2019.

EFAS performance in terms of accurately predicting the peak time of an event
EFAS performance in terms of accurately predicting the peak time of an event.

In terms of the peak magnitude of predicted events, 24% of participants reported that the actual flood magnitude was comparable to the EFAS predicting. Additionally, 45% answered that the magnitude was less or much less severe than the EFAS prediction, and 22% stated that the flood was more or much more severe than the EFAS prediction. While the distribution is, at large, comparable to that of 2019, during 2020 there was a significant increase of events whose peaks were significantly overestimated by EFAS predictions. While it can be considered positive that EFAS does not tend to underestimate the peak flow magnitude, the values for 2020 are far from those from 2018, where close to 60% of participants reported that the peak magnitude was comparable to the EFAS prediction.

EFAS performance in terms of accurately predicting the peak magnitude of an event.
EFAS performance in terms of accurately predicting the peak magnitude of an event.

The lead time of EFAS notifications varied greatly between the different flood events, with most notifications being sent up to three days before the actual start of the event. This should not come to a surprise given the large differences in predictability of different weather patterns leading to flooding in different parts of Europe. Nevertheless, considering that one of the criteria for issuing Formal Flood Notifications is that the lead time to the onset of an event needs to be longer than 48 hours, it is significant that so many notifications result in much shorter lead times, giving very little reaction time in some cases. This change from previous years may be correlated with the increase number of events that were predicted 1-2 days too late (see previous figure regarding timing of onset). It is also important to notice here the significant fraction of events that were predicted with 9 or more lead days.

Participants response to the question “What was the actual lead time?” of the feedback form.
Participants response to the question “What was the actual lead time?” of the feedback form.

Regarding the severity of the reported flood events, 42% of participants stated that the return period of the event was less than a 5-year flood. Since the criterion for sending out a Formal Flood Notification requires an exceedance of a 5-year return period for runoff, it is remarkable that so many events were reported to be less severe than that. Additionally, when compared to previous years, during 2020 there was a significant increase in severe floods (return periods between 20 and 99 years, 21%).

Participants response to the question “What is the return period of the observed flood event?” of the feedback form.
Participants response to the question “What is the return period of the observed flood event?” of the feedback form.

It is however important to notice here that the time periods that partners use to calculate return periods may vary significantly from EFAS. For instance, the 5-year return period in EFAS is often lower than the 5-year return period threshold that is used in Sweden by the SMHI. This may be due to the quality of historical forcing data, the hydrological model performance, and the different time periods used in the return period analysis. In addition, EFAS based return periods are calculated based on simulated discharges, whereas partners are more likely to base their thresholds on discharge observations. Calculating return periods from simulated discharge values may lead to systematic biases between observed and simulated discharge values at certain locations. Consequently, it would be better for the partners to evaluate the Formal Flood Notifications including also a comparison between simulated discharge in EFAS and their recorded observed discharge values.

The main drivers behind flood events in 2020 (highest ranked causes) were reported to be extreme rainfall (47% of participants), long-lasting rainfall (37% of participants), and snow melt (16%). These causes were, together with soil saturation, also the most important secondary drivers. Extreme rainfall is the only driver that has been reported as being relevant for a significant percentage of reported flood events throughout the different reporting periods. The percentage of reported events mainly caused by long-lasting rainfall during 2020 was comparable to that of 2019, and significantly larger than for previous years. Finally, soil saturation was a relevant secondary driver of many reported events throughout the different reporting periods.

Participants response to the question “What caused the flood event?” of the feedback form.
Participants response to the question “What caused the flood event?” of the feedback form.

Finally, partners were asked about the perceived added value they got from the EFAS Formal Flood Notification. Most participants reported that the EFAS Formal Flood Notification conveyed an added value to their activity. Indeed, about 80% of participants considered that notifications conveyed medium to significant added value, hinting to a positive reception of EFAS predictions by their users. This question was introduced in 2017 and no data is therefore available for 2016.

Participants response to the question “How would you rate the added value of the forecast?” of the forecast form. A value of 1 corresponds to little to no added value while a value of 5 corresponds to a significant added value.
Participants response to the question “How would you rate the added value of the forecast?” of the forecast form. A value of 1 corresponds to little to no added value while a value of 5 corresponds to a significant added value.

Feedback from non-observed flood events

If no flood was observed in connection with a given Formal Flood Notification, partners were asked if they had any explanation why the forecasted flood event did not actually take place. Possible reasons were listed: reservoir operation, ice jam, forecasted precipitation did not occur or fell in a different area, and not enough snowpack melt. No responses were unfortunately obtained to this question for the events reported during 2020 (this is not a mandatory question). Nevertheless, out of those who responded in previous years, which were not many either, the most common answer was “other”, followed by “not enough precipitation”. Overall, the limited number of responses to this question by point to the difficulty of attributing specific causes to false alarms.

Participants response to the question “If no flood, do you have an idea why the event did not occur?” of the feedback form.
Participants response to the question “If no flood, do you have an idea why the event did not occur?” of the feedback form.

The added value of false alarms was obviously low, but there were anyway partners that appreciated receiving such notifications. No responses to this question for false alarms were recorded prior to the transition to the new integrated feedback collection system and therefore no data is available prior to 2019.

Participants response to the question “How would you rate the added value of the forecast?” of the feedback form. A value of 1 corresponds to little to no added value and a value of 5 corresponds to a significant added value.
Participants response to the question “How would you rate the added value of the forecast?” of the feedback form. A value of 1 corresponds to little to no added value and a value of 5 corresponds to a significant added value.

Summary
The above analysis allows drawing some key messages as follows:

  • Feedback was provided for about a third of all the disseminated Formal Flood Notifications during 2020. This was a similar percentage than that of 2019 and a decrease from previous years, and might be attributed to the increased number of Formal Flood Notifications in recent years.
  • Formal Flood Notifications were generally perceived to convey an added value to the partners.
  • An increased percentage of the Formal Flood Notifications sent out during 2020 resulted in false alarms when compared to previous years.
  • The accuracy of the Formal Flood Notifications sent out during 2020 was perceived to be good and to a general degree comparable to previous years.
  • The return period of the majority of the observed flood events in connection with a Formal Flood Notification was less than 5 years, which is below the lowest EFAS threshold.
  • The main drivers behind flood events in 2020 were extreme rainfall and long-lasting rainfall and, to a lesser extent, snow melt.
  • Based on the lack of responses to why certain forecasted events did not occur, it may be deduced that establishing causes to false alarms is not obvious.