Evaluation criteria form a standard against which progress and achievement can be measured. They are usually captured in specific evaluation indicators and metrics, and can be used in conjunction with a baseline study of existing conditions.
- An indicator provides evidence that a condition exists or certain results have or have not been achieved, and can be either quantitative or qualitative.
- A metric refers to a unit of measurement that is quantitative.
Questions to consider
- Refer back to the objectives of the intervention and the Adaptation Logic Model – do the metrics and indicators help you to understand whether the objectives have been met?
- Consider and thoroughly test the logic behind your chosen indicators. Are they fit for purpose? Would they be more robust if worked into a package of indicators?
- How might changes in availability of data over the study period affect what can be measured, and when? This may affect which metrics you choose.
- Resist the temptation to distil your findings into a single number – this may be attractive to policy makers but does it tell them the full story?
- Remember that while metrics may be objective, the choice of indicators is not; these may reflect a particular framing of climate change. For example, a business may develop metrics to look at the economic viability of an adaptation action rather than examine the social distribution of benefits. Consider and challenge your own framing so it provides you with as full a picture as possible, as well as meeting your organisational needs.
- Quantitative metrics are attractive, but should be balanced with qualitative data which examines the facts behind the figures.
- Do the metrics you have chosen reflect a particular idea of success? Do you need to consider success from the point of view of other stakeholders or community members? For example, the success of a project to increase green space in urban areas could be measured in terms of reduced impact of the urban heat island effect, increased biodiversity or increased recreational space. All may be valid success measures depending on an individual’s perception
Process and outcome indicators
There are two distinct types of indicator:
“A process-based approach seeks to define the key stages in a process that would lead to the best choice of end point, without specifying that point at the outset”. (Harley et al, 2008.)
“An outcome-based approach seeks to define an explicit outcome, or end point, of the adaptation action”. (Harley et al. 2008.)
Process indicators are often used in the context of adaptation as frequently the point where the outcome can be evaluated has not yet occurred; as a result it can be difficult to apply an outcome-based approach. By using process indicators it is possible to consider whether the direction of travel is correct given the current information. For example, it may not be possible to determine whether a 20-year project will deliver adaptation benefits in a socially equitable way in Year Three, however, we could ensure the design of the project includes engagement of all social groups.
Developing evaluation metrics
Metrics are useful as evaluation criteria as they are objective, transparent and can be easily reproduced. The use of sound metrics allows for comparison with other types of adaptation actions or those delivered in other places and can enable the comparison of adaptation across spatial and temporal scales. Metrics can also be used to provide simple progress checks which can be understood by a wide range of users.
There are also disadvantages as there are no direct metrics with which to measure the adaptation process itself, so proxy indicators are used. Such measures can also be influenced by a range of social, economic and environmental factors outside the adaptation process. For example, a reduction in insurance claims in a flood prone area may be due to insurance companies refusing to insure properties, rather than the properties being better protected as the result of an adaptation project.
Metrics must be chosen carefully as part of a balanced package of indicators. They should be supported by a detailed evaluation of the reasons behind these data, and a more qualitative analysis of the impacts and reasons for an adaptation intervention. It is essential that we monitor what is important in improving our understanding, not only what is measurable.