Percentage of EFAS Formal and Informal Flood Notifications for which feedback was provided for 2022, aggregated per country (in colour). The total number of Formal and Informal Flood Notifications sent for each country during 2022 is shown on the map. EFAS partner countries for which no Formal and Informal Flood Notifications were issued during 2022 are shaded in grey.
Percentage of EFAS Formal and Informal Flood Notifications for which feedback was provided for 2022, aggregated per country (in colour). The total number of Formal and Informal Flood Notifications sent for each country during 2022 is shown on the map. EFAS partner countries for which no Formal and Informal Flood Notifications were issued during 2022 are shaded in grey.

 

by CEMS-Flood Analytics and Dissemination Centre

 

EFAS Flood Notifications (Formal, Informal, and Flash Flood Notifications) are the core of the EFAS service and for this reason it is important to ensure that they are accurate and relevant to the EFAS partners. The collection of feedback about Formal and Flash Flood Notifications started in 2016 and 2020, respectively. Since 2022 feedback is collected for Informal Flood Notifications as well. The criteria for the issue of Formal, Informal, and Flash Flood Notifications are explained here. Feedback on issued notifications contribute to identify potential weaknesses and help to identify and prioritize future developments of the EFAS system.

EFAS partners provided valuable and informative feedback to Formal, Informal, and Flash Flood Notifications issued during 2022. This article presents the analysis of the feedbacks collected during 2022, as well as a comparison with the feedbacks collected in the previous years. Some of the key messages stemming from the analysis are:

  • Feedback was provided for only 15% of the disseminated Formal and Informal Flood Notifications. The analysis showed that the partners are not yet used to provide feedback for Informal Flood Notifications. A protocol to issue feedback reminders for Formal Flood Notifications is already in place, a similar one for Informal Flood Notifications is under development. The response rate for Flash Flood Notifications was slightly higher: 19% for 2022.
  • Flood events were observed for 49% of the issued EFAS Formal/Informal Flood Notifications; compared to the last years feedback response, this result showed a small decrease of the number of false alarms and therefore an improvement. 
  • The main driver behind most riverine flood events was extreme rainfall, followed by long-lasting rainfall, snow melt and soil saturation. 
  • Formal and Informal Flood Notifications for flood events that were observed in reality were perceived to have a high added value to EFAS partners. The added value of false alarms was low.  
  • The accuracy of the Formal and Informal Flood Notifications was perceived to be good in terms of location, timing, and magnitude:  55% of responders stated that the location of the event had been as indicated in EFAS information; 71% of responders stated that the flood peak was observed on the predicted day, which is a little less but very similar compared to last year’s response; 48% of responders stated that the peak magnitude was comparable to the EFAS prediction (this percentage was higher than in the previous 3 years and similar to the high value in 2018). 
  • The number of Flash Flood Notifications was much lower in 2022 (763 Flash Flood Notifications) than in 2021 (1540 Flash Flood Notifications).  There was a high number of false alarms for Flash Flood Notifications, with 71% of the notifications not being observed in reality in 2022 (for reference, the percentage of false alarms was 61% in 2021).  

 

Methodology for the analysis of the feedbacks

This article presents the analysis of the feedback collected during 2022, and it provides a comparison with the feedback collected in the previous years (2016-2021). However, it is important to note that feedback collection methodology has been continuously improved and it became more complete and targeted throughout the recent years. For instance, additional questions have been included and the feedback retrieval has been streamlined. Another central example of change was the transition from an external service to a system which is integrated in EFAS-IS in mid-2019. Furthermore, as mentioned above, feedback collection for Informal Flood Notifications and Flash Flood Notifications were only introduced 2022 and 2020, respectively. Thus, the completeness of some of the questions included in the analysis may vary, hindering in some cases comparisons between different years. In addition, some of the questions in the feedback questionnaire are not mandatory causing a variability in the number of responses to the different questions. EFAS partners provided valuable and informative feedback to Formal and Informal Flood Notifications issued during 2022. The questions to be answered were the same for Formal and Informal Feedback, hence, those types of notifications could be analysed jointly. Although less comprehensive than the feedback on Formal/Informal Flood Notifications, feedback on flash floods was additionally received from the partners and analysed.


The former analysis of the feedback on notifications in 2021 highlighted the need for methodological improvements to the analysis protocol. The reasoning was that there are EFAS partner countries where several partners are receiving the same notification. Each partner is strongly encouraged to provide their feedback to that notification because the individual assessment of the notification from each partner is highly valuable. This can (and should) result in multiple feedbacks for one notification. Furthermore, up to January 2023, it was not possible to see if feedback has already been given for a certain notification, so it could have happened that partners unintentionally gave feedback to one flood event more than once. The sub-optimal analysis protocol used until 2021 could result in a number of received feedbacks which is higher than the total number of issued notifications. This direct comparison may potentially create a false perception of receiving a high number of feedbacks when compared to the number of notifications sent.


Ways to correct this artefact of the analysis protocol used up to the year 2021 were explored (Feedback on EFAS notifications for 2021 | Copernicus EMS - European Flood Awareness System). Further corrections were applied in the analysis for the year 2022 and will be used from now on. Specifically, the statistics on “Issued Formal and Informal Flood Notifications versus received feedback” and the map of provided feedback per country were corrected in order to account for the artefact above. In this article, the analysis was corrected by counting, for each country, the number of notifications that received feedback, neglecting if it was more than one feedback for the respective notification. It is important to note that this aggregation of the feedbacks was applied only to the graph (Figure 1) and the map (Figure 2), conversely, each one of the multiple feedbacks for one Formal/Informal Flood Notification has been included in the subsequent analysis: it is here reiterated that each individual assessment from each partner is valuable. Furthermore, in January 2023, a feature was added to the feedback platform enabling the user to see if feedback was already provided and which partner provided it. Clearly, consequences of this recent new feature are not seen yet in the analysis for the year 2022. 
 

 

Analysis of received feedback for Formal and Informal Flood Notifications during 2022

A total of 312 Formal and Informal Flood Notifications were issued during 2022, this number is the benchmark against which to compare the number of feedbacks. This benchmark is higher than in the previous years, because Informal Flood Notifications were only included since 2022. Counting them separately, it would be 136 Formal Flood Notifications, being less than in the previous years, and 176 Informal Flood Notifications. In total, 15% of all issued Formal and Informal Flood Notifications received feedback, neglecting the multiple feedbacks by applying the revised method of counting.


Counting the percentage of feedback separately for the two types of notification for the year 2022 using the revised methodology, there would be 41 Formal Flood Notifications that received feedback (30%) and 8 Informal Flood Notifications that received feedback (5%). Looking at these numbers, it needs to be contemplated that reminders to give feedback were sent to the partners only for Formal Flood Notifications but not for Informal Flood Notifications. Feedback reminders for Informal and Formal Flood Notifications are under development and will be sent out in the near future within this year. 


Figure 1 presents the number of EFAS Flood Notifications for which feedback was requested compared to feedback reports received. It applies the old method of counting, including multiple feedbacks for one notification, for the years 2016 to 2022 in order to be able to compare the numbers using the same method of counting. In the year 2021 using still the former analysis method, feedback reports were received for 59% of the Formal Flood Notifications. Using the same method as in the years before for 2022, 41% of the Formal Flood Notifications received feedbacks, which is a noticeable decrease compared to the year 2021 (Figure 1).


For the year 2022, the new, revised method of counting was applied. The results are included in the Figure 1 (orange coloured bar). The revised method was not only improved by neglecting the multiple feedbacks but it also included the Informal Flood Notifications and feedbacks along with the Formal Flood Notifications and feedbacks.

Number of EFAS Flood Notifications for which feedback was requested compared to feedback reports received. Former method of counting is applied to the years 2016 – 2022, looking only at Formal Flood Notifications. *In addition, the graph shows results of the revised method of counting for the year 2022, neglecting multiple feedbacks and including even the collection of feedback for Informal Flood Notifications that was started during 2021.
Figure 1. Number of EFAS Flood Notifications for which feedback was requested compared to feedback reports received. Former method of counting is applied to the years 2016 – 2022, looking only at Formal Flood Notifications. *In addition, the graph shows results of the revised method of counting for the year 2022, neglecting multiple feedbacks and including even the collection of feedback for Informal Flood Notifications that was started during 2021.

The provided feedback rate varies not only between years but also among EFAS partners (Figure 7). The question why some partners did not provide feedback was investigated by the EFAS partner survey for the reference year 2022 (the detailed report is available from EFAS partner survey 2022). The outcomes of the EFAS partner survey for the question related to notifications feedback were:  
•    62% of the EFAS partners that answered the survey, responded that they generally provide feedback on the notifications, while 36% stated that they do not provide feedback and 2% responded that they do not know.  
•    The most common reasons for not providing feedback were that there were no/few notifications/events in the area (6 participants), or a lack of time (2 participants), or that sending feedback is not implemented in the working procedure (2 participants).  
•    Few participants claimed that the notifications were too general and uncertain and that it is difficult to provide feedback in terms of hit or miss, when the notification informs about the probability of flooding. 
 

Figure 2. Percentage of EFAS Formal and Informal Flood Notifications for which feedback was provided for 2022 (using the revised method of counting), aggregated per country (in colour). The total number of Formal and Informal Flood Notifications sent for each country during 2022 is shown on the map. EFAS partner countries for which no Formal and Informal Flood Notifications were issued during 2022 are shaded in grey.
Figure 2. Percentage of EFAS Formal and Informal Flood Notifications for which feedback was provided for 2022 (using the revised method of counting), aggregated per country (in colour). The total number of Formal and Informal Flood Notifications sent for each country during 2022 is shown on the map. EFAS partner countries for which no Formal and Informal Flood Notifications were issued during 2022 are shaded in grey.

 

Assessing the connection between Formal/Informal Flood Notifications and observed flood events  
 

For the feedback analysis of the different questions, all feedbacks were taken into consideration, also when there were multiple feedbacks for one notification. In this way all valuable, individual information was included. The initial question in the feedback form was whether the flood event for which a Formal/Informal Flood Notification had been sent, was observed in reality. The definition of a flood event was specified in the question (i.e. return period equal to or larger than 2 years) to help partners assess the event. The 2-year return period was chosen as a definition for flood in this question as it would allow to differentiate between correct rejections and flood events that happened but did not reach the 5-year return period threshold. In total, 31 out of 63 respondents (49%) answered that a flood event was observed in connection with an issued Formal/Informal Flood Notification (Figure 3). This value is higher compared to the values from the two previous years (2021: 31% and 2020; 38%), moving again towards the high observed rates in 2017 (69%), 2018 (70%) and 2019 (61%).

 

Figure 3. Participants responses to the question “Was the flood event observed?” of the feedback form.
Figure 3. Participants responses to the question “Was the flood event observed?” of the feedback form.

If a Formal/Informal Flood Notification was issued and a flood event was observed in connection with the notification, the partners were asked to provide further information on the accuracy and lead time, as well as on the severity and on the probable causes of the event. The results are presented in the following paragraph.

 

Feedback for observed flood events

Observed flood events: How accurate were EFAS forecasts and notifications?

The accuracy of the observed EFAS Formal/Informal Flood Notification was assessed with respect to location, timing of the onset, timing of the peak and peak magnitude.

Most of the responders rated the accuracy of EFAS notifications in terms of location as “As indicated in EFAS information” (17 out of 31; 55%) (Figure 4). This is a similar value from the previous analysis (56% in 2021), but is at the same time a decrease from all other years analysis (i.e. 76% in 2020).  The lower value is related to the significant increase in reported events being observed in an “Adjacent catchment” or "In the region but not in the adjacent catchment". A number of events were also reported to have occurred in the wider region.

Figure 4. EFAS performance in terms of accurately predicting the location of the event.
Figure 4. EFAS performance in terms of accurately predicting the location of the event.

Regarding the EFAS accuracy in terms of the timing of the onset of the event, 68% of the responders stated that the start of the predicted flood event happened on the day predicted by EFAS (Figure 5). This was a decrease compared to the previous year (81% in 2021), yet a similarly high value as in 2016 (67%) and 2019 (61%).  Furthermore, 13% of the responders reported that the onset was 1-2 days later than it was predicted in 2022, and 19% of responders stated that the start of the event was 1-2 days earlier than it was predicted. No events were reported to have started 3 or more days later or earlier than predicted by EFAS.

Figure 5. EFAS performance in terms of accurately predicting the onset time of the event.
Figure 5. EFAS performance in terms of accurately predicting the onset time of the event.

The timing of the peak flow was also predominantly predicted with good accuracy in 2022 (Figure 6). 71% of responders stated that the flood peak was observed on the predicted day, which is a little less but very similar compared to last year’s response. There were no peaks reported to have occurred 3 or more days earlier or later in 2022. Only 1 responder stated to not know how accurate the timing of the peak was, this was similar to the previous years. This question was first introduced with the new integrated feedback reporting system and therefore no data is available prior to 2019.

Figure 6. EFAS performance in terms of accurately predicting the peak time of the event.
Figure 6. EFAS performance in terms of accurately predicting the peak time of the event.

When it comes to the accuracy in predicting the peak magnitude, a high percentage of responders (48%) stated that the peak magnitude was comparable to the EFAS prediction (Figure 7). This percentage was higher than in the previous 3 years and similar to the high value in 2018. While many responders in 2021 (51%) were stating that they did not know how accurate the notification was, there was a significant decrease seen in 2022 with only 16% not knowing the magnitude. Few responders stated that the peak magnitudes were either more severe (16%) or less severe (16%) than predicted by EFAS. Only one responder stated that the peak was much less severe (3%). None stated that the peak was much more severe than the EFAS prediction, which was the same in 2021, thus confirming the significant improvement when compared to previous years.

Figure 7. EFAS performance in terms of accurately predicting the peak magnitude of the event.
Figure 7. EFAS performance in terms of accurately predicting the peak magnitude of the event.

The lead time aspect was analysed separately for Formal and Informal Flood Notifications, because Informal Flood Notifications can have a lead time of under 48 hours, while Formal Flood Notifications are required to have a lead time of over 48 hours. The observed lead time for the EFAS Formal Flood Notifications varied greatly between flood events, with most notifications being disseminated up to three days before the actual start of the event (Figure 8), with 48% of the notifications having a lead time of two days. Given the large differences in predictability of different weather patterns leading to flooding in different parts of Europe, the variety in lead time is not surprising. Nevertheless, considering that one of the criteria for issuing Formal Flood Notifications is a lead time to the start of the onset of the event being more than 48 hours, it is noteworthy that a number of notifications result in a much shorter lead time to the observed event. This gives very little reaction time in some cases and was supposedly prevented by having the criteria of sending the Formal Flood Notification up till 48 hours before the onset of the event. 


Most of the Informal Flood Notifications were found to have a lead time of one day (3 out of 6; 50%), while 2 responders stated that there was a lead time of zero days and 1 responder stated that the lead time was three days (Figure 9). The feedback response rate for Informal Flood Notifications was very low, and does not cover a big variety of feedbacks. It is important to increase the amount of feedbacks for Informal Flood Notifications, in order to have a significant number of answers to base this analysis on.   
 

Figure 8. Partners' response for Formal Flood Notifications to the question "What was the actual lead time (i.e. days between receiving EFAS notification and observed onset of event)?" of the feedback survey.
Figure 8. Partners' response for Formal Flood Notifications to the question "What was the actual lead time (i.e. days between receiving EFAS notification and observed onset of event)?" of the feedback survey. 
Figure 9. Partners' response for Informal Flood Notifications to the question "What was the actual lead time (i.e. days between receiving EFAS notification and observed onset of event)?" of the feedback survey.
Figure 9. Partners' response for Informal Flood Notifications to the question "What was the actual lead time (i.e. days between receiving EFAS notification and observed onset of event)?" of the feedback survey.

 

Observed flood events: How valuable were EFAS forecasts and notifications?

Looking at the Figure 10 showing the result of received feedback according to the added value, where 1 is the lowest and 5 is the highest, most Formal/Informal Flood Notifications received a response stating a high added value (32% for a rating of 5 and 29% for a rating of 4). Although more partners perceived a high added value in 2021, the number of partners perceiving a high added value in 2022 was still higher than in 2017 to 2020. There were only few responses reporting a low added value in 2022 (6% for a rating of 1 and 3% for a rating of 2). This question was introduced in 2017 and therefore no data is available for 2016.

Figure 10. Partners' response to the question "How would you rate the added value of the forecast?" of the forecast survey for instances where a flood event was observed in connection with the Formal/Informal Flood Notification. A value of 1 corresponds to little to no added value and a value of 5 corresponds to a significant added value.
Figure 10. Partners' response to the question "How would you rate the added value of the forecast?" of the forecast survey for instances where a flood event was observed in connection with the Formal/Informal Flood Notification. A value of 1 corresponds to little to no added value and a value of 5 corresponds to a significant added value.

 

Observed flood events: magnitude and drivers

In 2022 there were less responses compared to the previous year not knowing about the severity of the event in terms of return period (23% in 2022, 51% in 2021). 32% of the respondents classified the event as having a return period of 2-4 years (Figure 11). That was a significant increase compared to the previous year 2021, but more similar to the percentage of the years 2020 and 2019. Considering that EFAS Formal Flood Notifications are triggered by a forecasted magnitude with a return period of 5 years or higher, this result of the survey could indicate that the EFAS overestimated the magnitude of the 32% of notifications that were classified by the responders to have a 2-4-year return period. With 26% the return period of 5-9 years got the next highest percentage of received feedback. There were also 3 responders stating an event with a return period of 10-19 years, and 3 responders stating an event of 20-99 years. There was no response having observed an event with a return period of over 100 years.

Figure 11. Partners' response to the question "What is the return period of the observed flood event?" of the feedback survey.
Figure 11. Partners' response to the question "What is the return period of the observed flood event?" of the feedback survey.

It is important to note that EFAS return periods are calculated based on simulated discharges, whereas most EFAS partners might base their thresholds on river discharge observations. It is therefore advised for partners to perform a bias analysis of the EFAS simulations with respect to their observed discharge values in order to adequately evaluate and use EFAS return periods. Additionally, other factors such as the quality of the historical forcing data, the hydrological model performance, and the extent of the time series used in the return period analysis may also have an impact on the computed thresholds.


The main drivers leading to flood events in 2022 (highest ranked causes) were reported to be extreme rainfall (56% of the respondents), long-lasting rainfall (25%), snow melt (13%) and soil saturation (6%) (Figure 12). These causes were also the most important secondary drivers. Extreme rainfall is the only driver that has been reported as being relevant for a significant percentage of reported flood events throughout the different years (from 2016 to 2022) and long-lasting rainfall is also mentioned as a driver in each year. Soil saturation is a relevant secondary driver of many reported events throughout the different years and snow melt has occurred to sometimes be a primary and sometimes a secondary driver. 
 

Figure 12. Partners' response to the question "What caused the flood event? If more than one cause, the alternatives are ranked from 1 to 4 (graph shows number of each cause and rank)." of the feedback survey.
Figure 12. Partners' response to the question "What caused the flood event? If more than one cause, the alternatives are ranked from 1 to 4 (graph shows number of each cause and rank)." of the feedback survey.

 

Feedback for non-observed flood events

If a flood event was not observed in connection with an issued Formal/Informal Flood Notification, partners were asked to provide an estimate as to why no event was observed and also on the added value of the notification. 
 

Most responders (69%) stated that the most likely reason for no flood event being observed in connection with an EFAS Formal/Informal Flood Notification was that either precipitation was overestimated by the forecast or that the precipitation fell elsewhere (“Not enough precipitation” in Figure 13). 25% of responders indicated that upstream reservoirs had dampened the flooding. There were 3% of responders conveying that there was not enough snowmelt for a flood event and 3% of responders claiming that it was another reason that no flood was observed.
 

Figure 13. Partners' response to the question "If no flood, do you have an idea why the event did not occur (reservoirs, precipitation as snow, precipitation fell in other area, forecasted precipitation did not occur, snow did not melt as fast as predicted, etc.)?" of the feedback survey.
Figure 13. Partners' response to the question "If no flood, do you have an idea why the event did not occur (reservoirs, precipitation as snow, precipitation fell in other area, forecasted precipitation did not occur, snow did not melt as fast as predicted, etc.)?" of the feedback survey.

The added value of false alarms in 2022 was as expected low (Figure 14). No responses to this question for false alarms were recorded prior to the transition to the new feedback collection system and therefore no data is available prior to 2019.

Figure 14. Partners' response to the question "How would you rate the added value of the forecast?" of the forecast survey for instances where a flood event was not observed in connection with the Formal/Informal Flood Notification. A value of 1 corresponds to little to no added value and a value of 5 corresponds to a significant added value.
Figure 14. Partners' response to the question "How would you rate the added value of the forecast?" of the forecast survey for instances where a flood event was not observed in connection with the Formal/Informal Flood Notification. A value of 1 corresponds to little to no added value and a value of 5 corresponds to a significant added value.

 

Analysis of the feedbacks for Flash Flood Notifications during 2022

The analysis of the feedback received for Flash Flood Notifications shows that there have been more than 2 times as many Flash Flood Notifications been issued in 2021 (1540) compared to 2022 (763). The response rate was similar in both years with 17% in 2021 and 19% in 2022 (Figure 15). This graph applies the former method of counting; therefore, multiple feedbacks may still be included in the counts. This will be corrected in the next years feedback analysis. 

Figure 15. Flash Flood Notifications sent versus feedback received.

 

In 2022, 28% of the Flash Flood Notifications were observed in reality, which is a decrease compared to 2021 (38%) (Figure 16).

Figure 16. Participants responses for Flash Flood Notifications to the question “Was the flood event observed?” of the feedback form.
Figure 16. Participants responses for Flash Flood Notifications to the question “Was the flood event observed?” of the feedback form.