by CEMS-Flood Analytics and Dissemination Centre
Formal Flood Notifications are the core of the EFAS service and for this reason it is important to ensure that they are accurate and relevant to the EFAS partners. It is therefore of utmost interest to collect partner feedback specifically for these notifications so that their quality and relevance can be monitored. Additionally, feedback may contribute to identify potential weak spots and help to identify and prioritise future developments of the EFAS system.
EFAS partners provided valuable and informative feedback to Formal Flood Notifications issued during 2021. This article presents the analysis of the feedbacks collected during 2021, as well as a comparison with the feedbacks collected in the previous years. Some of the key messages stemming from the analysis are:
- Feedback was provided for roughly one third of the disseminated Formal Flood Notifications. This is a decrease in the response rate with respect to 2020 and reasons for this will be analysed in the upcoming annual survey for the year 2022
- The main driver behind most flooding events was extreme rainfall, followed by long-lasting rainfall
- Formal Flood Notifications were perceived to have a high added value to EFAS partners, with a substantial improvement from previous years
- However, the number of false alarms significantly increased during 2021 with respect to previous years. This tendency, if not corrected, might lead to a decreased trust in the EFAS notifications in the future.
- The accuracy of the notifications was perceived to be good. Albeit the uncertainty in flood peak magnitude, the partners appreciate the accuracy of the predicted timing of the onset of the event and peak flow.
Methodology for the analysis of the feedbacks
This article analyses the feedback collected during 2021, contrasting it with data from previous years (2016-2020). Nevertheless, it is important to note here that since feedback collection has been continuously improving and becoming more complete and targeted throughout the recent years, the completeness of some of the questions included in the analysis may vary, hindering in some cases comparisons between different years. A central example of these changes was the transition from an external service to a system which is integrated in EFAS-IS in mid-2019.
Other modifications, such as the inclusion of additional questions, and the streamlining of feedback retrieval, have also been implemented. It is important to note that some of the questions in the feedback form are not compulsory, which may lead to the statistics for certain questions being based on a lower number of responses. Moreover, the analysis of the feedback on notifications in 2021 highlighted the need for methodological improvements to the analysis protocol. The reasoning is briefly explained below.
There are EFAS partner countries where several partners are receiving the same notification. Every partner is strongly encouraged to provide feedback to that notification once: the individual assessment of the notification from each partner is highly valuable. Due to the current sub-optimal analysis protocol, such a situation can lead to a higher number of feedbacks received than total notifications sent. This artifact leads to the false perception of receiving a high number of feedbacks when compared to the number of notifications sent. Nevertheless, since each feedback is important, ways to correct this artifact in order to provide accurate statistics of the number of feedbacks received as opposed to the number of notifications sent are being explored.
In this report, an intermediate, not final solution is presented. During this year’s analysis, it was decided to reduce the number X, feedback received per country i, to the total number Y, notifications sent per country i, if Xi > Yi. Such a solution has the purpose to provide a more realistic ratio between number of feedbacks and number of notifications.
For transparency, this report presents the two different ways of counting in figures 1a and 1b below. It is here noted that since the feedback received is aggregated over a year, the intermediate solution of this report does not prevent showing potentially too high numbers of feedback received for single countries and consequently even in the average over all countries as presented in figures 1 to 2. Ideally, the percentage of notifications that received feedback should be computed and technical solutions to achieve this aim are being explored.
Analysis of received feedbacks during 2021
147 Formal Flood Notifications were sent out during 2021. According to the intermediate solution proposed above, 37% of all issued Formal Flood Notifications received a feedback (Figure 1a). In the former analysis, this would have been 59% (Figure 1b). Looking at the new Figure 1a, this is roughly of the same order of magnitude as 2019 (35%) and 2020 (39%). It is worth noting that the feedback response rate was slightly higher during 2020 and at the same time the total number of notifications sent was much higher than during 2021 (240 in 2020 compared to 147 in 2021, compare Figure 1a). Moreover, the response rate in 2021 shows a strong decrease compared to the year 2018 where the same number of notifications was sent, and more than 60% of them received feedback.
To date, there is, however, no detailed analysis on why some partners did not provide feedback, this question should be studied in more detail, possibly in connection with the annual survey. The provided feedback rate varies not only between years but also among EFAS partners (Figure 2) and a more detailed analysis would be needed in order to understand these patterns and to increase the feedback sent to EFAS.
Assessing the connection between Formal flood notifications and observed flood events
The initial question in the feedback form was whether a flood event was observed in connection with a given Formal Flood Notification. The definition of a flood event was included in the question (i.e. return period equal to or larger than 2 years) to help partners assess the event. The 2-year return period was chosen as a definition for flood in this question as it would allow to differentiate between correct rejections and flood events that happened but did not reach the 5-year return period threshold. In total, 27 out of 87 respondents (31%) answered that a flood event was observed in connection with an issued Formal Flood Notification (Figure 3). This value is slightly lower than that from the previous analysis (2020; 39%), and significantly lower than the year before (2019: 61%), indicating a continued increasing trend in the number of false alarms.
If a flood event was observed in connection with an issued Formal Flood Notification, the partners were asked to provide further information on the accuracy, lead time, and added value of the notification, as well as on the severity and most likely causes of the event.
Feedback for observed flood events
Observed flood events: how accurate were EFAS forecasts and notifications?
Most of the responders rated the accuracy of EFAS information in terms of location as “As indicated in EFAS information” (15 out of 27; 56%) (Figure 4). This is a decrease from the previous analysis (2020; 76%), and is related to a significant increase in reported events being "In the wider region". A small number of events were reported to have occurred in an adjacent catchment. Overall, the location accuracy of EFAS in 2021 was comparable to that of 2016 and 2019.
Regarding the EFAS accuracy in terms of the timing of the onset of the event, 81% of the responders stated that the start of the predicted flood event happened on the day predicted by EFAS (Figure 5). This represents a significant increase with respect to the previous analysis (2020; 34%) and even surpasses the accuracy rates of 2016 and 2019. No events were reported to have started 3 or more days later than predicted by EFAS.
The timing of the peak flow was also mostly predicted with good accuracy in 2021 (Figure 6). Indeed, 81% of responders stated that the flood peak was observed on the predicted day. This represents a significant increase from previous years, when most flood peaks occurred earlier than predicted. This question was first introduced with the new integrated feedback reporting system and no data is therefore available prior to 2019.
When it comes to the accuracy in predicting the peak magnitude, a high percentage of responders (51%) stated that they did not know how accurate the notification was. This high uncertainty rate is only comparable to that of 2016 and a steep increase from recent years. Otherwise, most responders stated that the peak magnitudes were either comparable (22%) or less severe (19%) than predicted by EFAS. Only a few responders stated that EFAS underestimated the peak flow magnitudes (7%), which is a significant improvement when compared to previous years.
The lead time given by EFAS Formal Flood Notifications varied greatly between flood events, with most notifications being disseminated up to three days before the actual start of the event (Figure 8). This should come to no surprise given the large differences in predictability of different weather patterns leading to flooding in different parts of Europe. Nevertheless, considering that one of the criteria for issuing Formal Flood Notifications is that the lead time to the start of the onset of the event is more than 48 hours, it is remarkable that so many notifications result in a much shorter lead time, potentially giving very little reaction time in some cases. This pattern also seems to conflict with the responses provided to the questions on the predicted timing of the event which, to an overwhelming extent, pointed to a very high accuracy.
During 2021 partners perceived that EFAS Formal Flood Notifications conveyed a significantly higher added value for them (Figure 9). Indeed, the percentage of responders that considered that received EFAS Formal Flood Notifications had a high added value (values of 4 and 5 in the questionnaire) significantly increased with respect to previous years (2021: 78%; 2020: 31%). This question was introduced in 2017 and not data is therefore available for 2016.
Observed flood events: how much valuable were EFAS forecasts and notifications?
Roughly 50% of the respondents stated that they did not know how severe the flooding event was, which is the same percentage for the same answer in for the question on magnitude accuracy (Figure 10). Out of the responders that had information on the flood magnitude, it is most noticeable the drastic reduction of forecasted events with magnitudes under 5-year return period in 2021 (18%) with respect to previous years (e.g. 2020: 42%). The response rate for such events is comparable to that of 2017 and more in line with the guidelines for issuing Formal Flood Notifications, which require a forecasted magnitude with a return period larger than 5 years.
Observed flood events: magnitude and drivers
Here it is important to note that EFAS return periods are calculated based on simulated discharges, whereas most EFAS partners might base their thresholds on discharge observations instead. It is therefore advised for partners to perform a bias analysis of the EFAS simulations with respect to their observed discharge values in order to adequately evaluate and use EFAS return periods. Additionally, other factors such as the quality of the historical forcing data, the hydrological model performance, and the time periods used in the return period analysis may also have an impact on the resulting values.
The main drivers behind flooding events in 2021 (highest ranked causes) were reported to be extreme rainfall (90% of the respondents) and long-lasting rainfall (7%), and snow melt (3%) (Figure 11). These causes were, together with soil saturation, also the most important secondary drivers. Extreme rainfall is the only driver that has been reported as being relevant for a significant percentage of reported flood events throughout the different reporting periods, and it even became more prominent during 2021, in which long-lasting rainfall is mostly listed as a secondary driver. Soil saturation is a relevant secondary driver of many reported events throughout the different reporting periods.
Feedback for non-observed flood events
If a flooding event was not observed in connection with an issued Formal Flood Notification, partners were asked to provide an estimate as to why no event was observed and also on the added value of the notification.
Most responders stated that the most likely reason for no flooding event being observed in connection with an EFAS Formal Flood Notification was that either precipitation was overestimated by the model or that the precipitation fell elsewhere (Figure 12). A small percentage of responders indicated that upstream reservoirs had dampened the flooding.
The added value of false alarms in 2021 was as expected low, but partners tended to appreciate such notifications more than in the previous year (Figure 13). No responses to this question for false alarms were recorded prior to the transition to the new feedback collection system and therefore no data is available prior to 2019.