This time we have invited Global Cyber Security Capacity Centre (GCSCC) at Oxford University and Royal Holloway University London to share about the work they have done so far in identifying measures and frameworks for evaluating cyber capacity building projects, initiatives, and efforts.
Huge thanks to Carolin Weisser Harris, Lead International Operations at GCSCC and Co-Lead Task Force Strategy & Assessments at GFCE and Phil Sheriff, PhD Researcher in Cyber Security at Royal Holloway, University of London for taking their time to elaborate on the research they do and the results they expect.
The question of what works and what doesn’t work in CCB is at the core of our research over the last ten years and with the development and deployment of the Cybersecurity Capacity Maturity Model for Nations (CMM) we have contributed to a better understanding and worked intensively with key actors in CCB. One of them is the Global Forum on Cyber Expertise (GFCE) and as a contribution to the Working Group Strategy and Assessments we started this work which looks at challenges of evaluating impact of CCB activity from a systematic and operational focus at maturity and positive cybersecurity outcomes, but also higher level outcomes such as the Sustainable Development Goals (SDGs) or GDP.
GCSCC has published a number of papers that have been written looking at whether or not CCB matters at the macro level, for example linking GDP to cybersecurity maturity. To understand better the link to shorter term outcomes, and to understand how CCB can integrate concepts from the International Development Community, we have held interviews with practitioners and experts in both CCB and MERL, examining drivers, and the challenges of data and evaluation, from the perspectives of donors, implementers, and target audiences. The findings have been put into a paper, drafted in cooperation with Royal Holloway University London and presented at to the GFCE community and participants of the Global Conference on Cyber Capacity Building (GC3B) where we organised a panel discussion and workshop. The results of the workshop are available here.
Getting all parties to understand the importance of systematic evaluation of CCB and committing to support it are probably the most important, if not the greatest, challenges. If understanding and appreciation of evaluation is enhanced, then a number of current challenges will be easier to overcome, in terms of funding, resourcing and enabling, i.e. donors need to fund evaluation, implementers need to resource it, and recipients need to enable it (in terms of data access). This will only come once all parties are convinced of the requirement for better evaluation, and resource/enable it accordingly.
Probably the greatest challenge (i.e. once resources are put in place, implementers are upskilled, and recipients are happy to allow data access over time) is creating a strong evidence base for better CCB interventions over time.
This requires common terminology and understanding, standardising across a wide range of intervention types, ideally learning lessons from areas such as international development best practice.
Outputs, outcomes and impacts are often viewed as a function of time and/or scale. However, they actually represent different processes, and recognising this is key to better measure them.
Outputs are the immediate deliverable of an intervention, i.e. number of people trained, equipment deployed, conference attendance, strategies/legislation drafted/implemented.
An outcome is a behaviour change that results from the outputs. For example, for a security related training intervention, it could be the improvement in security, indicated by the reduction in proportion of successful attacks. For legislation, it could be the number of investigations/prosecutions that result from a new/amended law.
For impact – this is the resultant longer-term effect – for example an increase in economic benefit, increased GDP, etc. that result from a more secure cyber eco-system which is more enabling from an enterprise perspective. From a legislation/cybercrime perspective, it could be an overall reduction in cybercrime figures, or a reduction in cyber harm.
In conjunction with Royal Holloway University London and Integrity Global we are undertaking research along four different activity strands, starting with the creation of a typology for CCB interventions. Such activity will be iteratively co-created with a wide range of stakeholders. At the same time we will be researching evidence gaps, and looking at specific case studies that can be used to exemplify outputs/outcomes/impacts, identifying data sources, metrics and indicators that can be used for specific intervention types.
Carolin Weisser Harris is the Lead International Operations at the Global Cyber Security Capacity Centre (GCSCC), based at the Department of Computer Science of the University of Oxford.
In this role, Carolin is responsible for stakeholder engagement and the deployment of the centre’s Cybersecurity Capacity Maturity Model for Nations (CMM). She has co-authored a number of CMM reviews in Africa, Asia and Europe and contributed to best practice guides and research outputs in the field of cybersecurity capacity-building. Carolin is also part of the leadership of the working group “Strategy & Assessments” of the Global Forum on Cyber Expertise (GFCE) and led the development of CYBIL, the GFCE’s Cyber Capacity Knowledge Portal.
Phil Sheriff is a PhD student at Royal Holloway University London, studying the metricing and evaluation of international cyber capacity building programs. Prior to joining the CDT, he worked for both the FCDO and Australian DFAT in Pakistan, Colombia, Bosnia and Thailand, as well as roles in London and Canberra. Prior to that he spent seven years in the British Army. He has an MSc in Information Security from RHUL, an MA in Defence Studies and International Relations from St Andrews, and a BA in Physics from Oxford.