“Driving impact: making the case for better evaluation of CCB”

In the “Accra Call” of the Global Conference on Cybersecurity Capacity Building (GC3B) in Accra/Ghana in November 2023, the global CCB community committed to “accelerate efforts to improve the measurement of cyber capacity building results [….] by actively and systematically integrating methodologies and good practices from the international development field”. 

This call was supported by two events organised at the conference by the Cybersecurity Capacity Centre for Southern Africa (C3SA), the University of Oxford Global Cyber Security Capacity Centre (GCSCC), Integrity and Royal Holloway University London (RHUL), in support of the Global Forum on Cyber Expertise’s Strategy and Assessments Working Group (GFCE WG):

• A panel discussion “Driving impact: making the case for better evaluation of CCB” [full video], making the case for CCB actors to invest resources in IE; and
• A complementing practitioner workshop’s (“Improving CCB Evaluation Practice - Knowledge Sharing and Community Building”), whose purpose was to build a community of practice to foster collaboration between the different actors and to identify lessons, tools, and solutions for better CCB evaluation and metrics.

Following are the key themes and issues expressed by CCB actors* during those sessions:

Cyber Capacity Building Principles

The following key principles for Cyber Capacity Building (CCB), and CCB Impact Evaluation (CCBIE), were recognised.
 
Local Ownership is a key principle of CCB, and is achievable through aligning CCB activities with local needs and capacity assessments, gap analysis, local context considerations, co-design, involvement of local expertise, recipient budgetary prioritisation, inclusion of recipients in donor’s budget planning, and the communication of (desired) impact to recipients.  
 
Inclusivity is important in order to break down barriers, minimise siloed working practices, and enable a more interdisciplinary approach between fields such as economics, development, engineer and computer sciences. CCBIE can, for example, factor in normative approaches from established development practices to avoid learning the same lessons at cost. Inclusivity will bring benefits from different perspectives and experiences, for example digital financial inclusion, where the development practitioners will be very different from the cyber-security teams. Identification of and engagement between stakeholders is key, requiring transparency regarding stakeholder objectives, (specifically donor and recipient), at international, regional and national levels.
 
Sustainability is essential, as CCB and CCBIE are not a one-off activities but require long-term engagement and resourcing. Sustainability is closely aligned with local ownership, and is negatively impacted by silo-ed working practices. Sustainability was specifically seen as important from an evidence-base perspective, as it would enable longitudinal data gathering, which is needed to measure long-term CCB outcomes and impacts, and also at aggregate level, which is currently missing. This would in turn enable evidence based, scalable, better coordinated and replicable CCB interventions.
 
Affordability requires CCB budget prioritisation for direct funding for projects, and also for CCBIE. Participants highlighted the lack of funding and resources for evaluation and the cost burden on implementers.

CCB Drivers

The panel discussion and workshop identified several drivers for cybersecurity and cyber capacity building. One example given was the minimization of cyber harms, linking cybersecurity capacities to the reduction of cyber harm across different intervention types, within different parts of the cyber ecosystem. Other examples for capacity building drivers emerge indirectly through International Development goals such as digital financial inclusion, with the narrative around driving investment in digitalization requiring security of those systems, from design to regulation to recourse mechanisms. Drivers are also a result of security failures, for example cyber security became a major consideration for the broader financial sector following the Bangladeshi bank heist.

Data, Metrics and Indicators

The development of robust data, metrics and indicators, which will improve the analysis of CCB, was identified as a top priority for CCBIE:

Data availability: Data availability for both CCB and CCBIE depends in part on the specific context, and differs between developing and mature economies, in both capacity and legislation/regulation around collection. In general, however, despite specific cases (for example the number of cyber incidents per country, and certain programme data) sufficient data exists according to workshop participants. However there is a need to improve current data collection, integrity, validation, consistency, and access. Challenges of data collection and processing were due to a number of factors, including a lack of incentives, resources, availability of certain data, infrastructure for secure data handling, clear benefit, and lack of skills and/or regulation for the collection of specific data such as digital evidence. Data collection is currently seen as an extra (ie not an integral) part of a programme, and the cost of acquisition/analysis as a burden, specifically on implementers. Trust issues and lack of political good will were also identified as impediments to data sharing, both from recipient as part of an intervention, and back to recipient following interventions. Trust was also seen as impacting public participation in data acquisition.

Data quality: As well as the importance of contextualised data and issues around misaligned intervention data, other aspects of data quality were identified as issues. For example, data needs to cover observables - ie People Processes & Technology indicators, in order to measure intended change, and should disentangle perceived change vs actual change.

Data sources: Participants suggested using public data sets that measure system vulnerabilities from a network through to a country level. There is also a need for simplified and automated data, multiple data sources, along with a more qualitative approach. Other data sources could include AI scrapped public reporting and contextual sentiment analysis, and data captured from large events (ie conferences) to demonstrate tangible outcomes, impacts, and benefits. There is also a distinction to be made between cyber security data, and CCB programme data, ie regarding project outputs and outcomes). And, similar to data availability, there is a need to clearly understand the difference in data sources depending on the specific context of CCB.

Metrics: There is a need for better understanding regarding developing metrics, for example around resilience, ie not just about preventing attacks, but about recovery from attacks. This was mentioned specifically in the context of FinTech. Better understanding of and metrics regarding trustworthiness, and broader social factors that drives trust in the system, is also required. Importantly, metrics need to be sustainable over a sufficient period of time to demonstrate longer term expected outcomes, impacts and benefits.
Indicators: Indicators need to improve the understanding and identification of the links between high level data and individual interventions. Therefore, there is a collective need to come up with “good” indicators, which could take a sectoral approach, ie financial sector. As an example, a high-level indicator for FinTech could be a measure of the elements that keep the system reliable. Interventions should have measurable goals for medium/long term impact across-sector.

Frameworks and Evaluation: “Not everything that is measured matters, and not everything that matters can be measured”

In order to better integrate with International Development, and to access additional funding, CCB requires monitoring and evaluation to be done using the same language of Theories of Change and inputs, outputs, outcomes and impacts, and needs to link ToC to Impact Evaluation. Third parties should be required to conduct evaluation, but it is also important to educate and capacitate implementers to better integrate monitoring and evaluation into interventions.

Frameworks, indices and models are important to support decision making processes and can provide evidence of cybersecurity effectiveness, which in turn would help prioritization. Common standards open the possibility to communicate among countries, share successes and put pressure on governments to perform.

There is a need to use existing models (CMM, SIM3, GCI) to engage with recipients, when initiating new projects. Within existing frameworks, there is a need to find commonalities and correlations, and how different frameworks, methodologies and measurements complement/contradict each other. Some are based on inputs, not outputs/outcomes/impacts. There is also a lack of standards regarding what success looks like in medium and long-term interventions – for example the change in cyber workforce, number of attacks, reduction in cyber harms, etc.

Importantly, context and intervention-specific nuances are not captured by generic framework questions, so there is also a need to create a toolkit for evaluating common types of projects – ie CERTs, training, cybercrime, legislation, with separate M&E frameworks for cybersecurity programmes depending on the types of programmes. Context is also important, and panelists suggested a sectoral approach with subsets of indicators and standards for minimum requirements for countries.

Evidence, Impact, and Benefits

Access to development funding necessitates the need to follow a development model for outcomes and impacts, creating clear intervention paths (via road-mapping), and creating evidence that is needed to both contribute to the policy and decision-making processes, and demonstrate links between cybersecurity and SDGs. From a donor perspective – there is a need to demonstrate how funding is both enhancing donor-security as part of the cyber eco-system, and also creating impacts as part of the International Development SDG agenda.

Though there is evidence that cybersecurity matters, e.g. regarding GDP or growth of specific industries which correlate with cyber incidents, there is a need for more research and better understanding of the causal effects and the implicit and explicit, direct and indirect cost of cyber harm, and to consider the full spectrum of cyber harm, ie at the individual, organisation and national level. For example, better understanding is required of the economic effects of cyber incidents, through to consequences for the production chain, sales, and reputational losses, especially in banking and finance.

As already mentioned in CCB drivers, such disaggregation of harm would allow for an examination of what capacities are needed to minimize cyber harm at every point, as well as a disaggregation of different types of CCB interventions. This would help build understanding of the benefits of cybersecurity and CCB investments from a cost-benefit perspective, working with beneficiaries to demonstrate that funding is generating evidence, and evidence is driving measurable interventions.
Participants suggested a layered approach to examining impact (ie strategic, operational, tactical) can be taken, along with a menu of impacts to guide recipients. This will help link cybersecurity to higher level impacts such as SDGs, which would help demonstrate to donors why this funding is needed.

Coordination, Communication and Awareness Raising

Coordination and deconfliction, in particular between donors, is important, due to the risk of over-capacitation in some areas, as different donors come with variants of the same capacity building or missing neglected areas of need.

Communication and awareness raising was mentioned as crucial, both at the technical level for governments, but also more widely in non-technical terms for citizen. Building a cyber-resilient society with shared responsibilities is not the responsibility of the government alone, and therefore communication is necessary with all stakeholders, for example media, private sector, and academia, so that all possible stakeholders that can contribute.

Panelists highlighted the importance of sharing success stories, lessons learnt and evidence regarding the benefit of CCB, in particular from a government and private sector perspective. Panelists also highlighted that the narrative around CCB and its impacts, and closer working with recipients, are key to getting the buy-in for obtaining the data which can demonstrate that funding is generating evidence in a timely manner.

Future work & Recommendations

There is a need to disentangle cybersecurity, as well as the different components of CCB (for example different types of interventions), in order to understand better the benefits of both CCB and CCB Impact Evaluation. There is an opportunity to map the ecosystem in a sectorial approach, ie hard/soft infrastructure, people, etc, to see what kind of resources are required, then use a more interdisciplinary approach to better manage CCB resources. Similarly, regarding best use of resources, it is important to improve coordination and deconfliction at the international level, and to streamline the funder process – for example via a centralised PoC. Finally, there is a need to measure the outcomes and impacts of CCB at both project and aggregate levels, in both the short and long term, in order to inform future directions and improve coordination of CCB.

Developing a positive narrative about the benefits of evidence-based CCB is critical. As a community, we need to develop theories of CCB to understand its benefits, informed by approaches from other (development) fields to decide on what and how impact is measured (e.g. cyber harm, costs, economic impact, resilience, trust).

There’s a need to improve existing data collection, integrity, validation and consistency (e.g. cyber incidents per country, CCB outputs and outcomes to measure impact, etc). This will require better understanding of the importance and relevance of certain types of data, and will require the involvement of key stakeholders from various disciplines in the gathering, analysis and interpretation of the data. However, there are challenges for certain types of data, for example, due to the incentives around not disclosing cyber incidents, and there is a role for governments to play in standardising, and allowing access to, different types of data.

Along with better data on the number of cyber incidents, more research is required on the implicit and explicit, direct and indirect, financial/non-financial costs of cyber incidents. Importantly, cyber incidents are not purely stoachastic. They are driven by determinants – economic, political, social, the breach function and the value at risk. So more research is required on how to predict, mitigate and determine the cost of cybersecurity incidents, with the potential that, with more research and development and more data, we may be able to have sufficient knowledge to more accurately predict, and more effectively mitigate, cybersecurity incidents in the future.

If you are interested to learn more about the project and get involved, please write to carolin.weisser@cs.ox.ac.uk

^{* Following organisations contributed to the planning and delivery of the sessions:
Blacksmiths Group, Carnegie Endowment of International Peace, Cyber Czar, Cyber4Dev, GFCE, GIZ, Global Partners Digital, Government of North Macedonia, Instituto Brasileiro de Ensino, JICA, KPMG UK, Michigan State University, NetHope, Oceania Cyber Security Centre (OCSC)/Monash University, Organization of American States (OAS), Royal United Services Institute for Defence and Security Studies (RUSI), Tecnológico de Monterrey, Tel Aviv University, UK FCDO, UNIDIR, and World Bank.
[2] https://www.youtube.com/watch?v=u_ar0i7InlY}