Abstract
This Policy Brief analyses some of the key issues related to the lack of timely evidence that can assist policymakers in tracking the progress towards the achievement of the Sustainable Development Goals (SDGs). The quality of official statistical information still lags behind the data needs, and barriers to the universal achievement of similar levels of data capture persist. There is a need to balance international comparisons with contextualised understanding of local needs to ensure that policies, drawing on progress data, are evidence-based. The G20 can lead a concerted rethink on more agile approaches to monitoring progress and understanding how to pivot effectively based on evidence and insights. Recommendations include triangulating summative, formative, and developmental evaluation; seeking hidden, unintended consequences of adopting specific indicators; repurposing existing data to improve the currency of information; and incorporating greater use of leading indicators.
1. The Challenge
Many options for tracking and assessing progress in the UN Sustainable Development Goals (SDGs) have been created over the years.[a] Nonetheless, it is acknowledged that information poverty restricts the effectiveness of tracking achievements. This Policy Brief focuses on SDG 4 and SDG 5 for illustrative purposes, but several points will be of relevance to discussions about the other SDGs.
The quality of official statistical information still lags behind the data needs, particularly in developing economies. While not denying the importance of statistics, their quality, defined by the United Nations (UN) as relevant; accurate and reliable; timely and up to date; accessible and clear; and coherent and comparable—[1] is critical if they are to contribute effectively to decision-making. It is easy to be seduced by hard numbers, and forget to look behind them. Developing countries struggle to update statistics regularly. For instance, the Gender Inequality Index currently relies upon data that in some cases dates back to 2013.[b] Sachs et al. (2021)[2] addressed both informational poverty and the time lag between the reference year and the monitoring year, analysing what is available on a country level for monitoring specific SDGs. Only 40 percent of the UN member states have the data available for the planned indicators for SDG 5 (gender equality), with an average reference year of 2016 in the monitoring year of 2021.[3] Furthermore, 58 percent of the countries have the necessary statistical data for monitoring SDG 4 (quality education) with the same average reference year, five years behind the monitoring year of 2021.
A range of barriers hinder the universal achievement of similar levels of data capture. Handling the difficulties in maintaining consistent and comparable meanings of indices and indicators requires effective international collaboration (Stotsky et al. 2016).[4] However, despite international efforts, it is unlikely that the issues will be resolved in the short term. A recent report highlighted a number of reasons behind these challenges, including difficulties that are inherent in stakeholder collaboration and coordination; lack of adequate human and financial resources to consistently collect and maintain data; gaps in the data required; and difficulties in producing appropriately disaggregated statistics that will provide additional insights for understanding and decision-making.[5] As the 2030 target nears, the persistent time lag between the need for policy decisions to optimise progress and the availability of relevant data as well as the need to have more agile access to evidence to support decision-making requires a redefinition of our approach to building an informational base that currently relies mainly on hard data from official statistical sources.
The scope and the provenance of specific indicators need to be understood to ensure that these adequately capture what we need to know. There is now greater awareness of informational poverty due to lack of resources and missing data points, and important lessons have been learnt with regard to the need for collaboration to reduce it. Informational poverty, however, is connected not only to the (non-)availability of statistical data, which is often related to the income levels of countries, but also to the informational correctness of the indicators selected.
For example, one crucial contention relates to the definition of GDP—its make-up only partially represents the monetary value of all finished goods and services made within a country in a given period of time. Importantly, it does not include women’s work with regard to caring for family (children, the elderly, other family members). Currently, this is captured neither by GDP nor by any other statistical indicator (e.g. that related to employment). This failure to fully represent the hidden work of women has been on the international agenda for some time now, and continues to raise concern, considering its negative impact on gender equality (SDG 5).[6] A more appropriate assessment of the progress for SDG 5 requires redefinition of the content of GDP. Identifying a broader set of potential measures is an avenue currently being explored by the High Level Forum on Official Statistics.[7]
There is a need to balance international comparisons with contextualised understanding of local needs to ensure that policies are evidence-based. This Policy Brief draws on the analogy of a road map. Reaching a person’s destination involves triangulating their current position, based on a number of data points, figuring out where they went wrong and autocorrecting, and being alert to accidents and traffic snarls ahead so that they can recalculate the best route, (sometimes) adjusting the destination itself. Only knowing how many kilometres one has travelled reveals nothing about how close the person is to the destination or whether they are on the right path.
2. The G20’s Role
The G20 can act as an impetus for a concerted rethink on more agile approaches to monitoring progress and understanding how to pivot effectively, based on evidence and insights. The G20 outlined its commitment to the 2030 Agenda for SDGs via its own Action Plan, highlighting the power of advocacy and the ability to support initiatives as the key strengths of this forum.[8] This makes the G20 an ideal platform for spearheading efforts to build our knowledge of alternative and complementary approaches to monitoring and tracking progress. The challenges are significant: inherent in tracking progress in a way that balances comprehensiveness with oversight and country comparisons with contextualised understanding, and provides a real impetus to understanding performance and, more importantly, the pathways to improvement.[9]
While addressing approaches to monitoring progress and informational poverty does not directly speed up the process of achieving the SDGs, it does improve our ability to make decisions, assess the relative efficacy of actions, and tackle systemic barriers to change. This in turn positively impacts the design of effective, evidence-based policies and interventions on global, regional, national, and subnational levels. Broadening the informational base with perceptual data will provide additional insights into women’s economic and social activities, and shorten the time lag between the monitoring year and the average reference year, which is critical for more effective and efficient interventions in the process of achieving the established goals.
3. Recommendations to the G20
The G20 should advocate for a triangulated approach, which combines findings from summative, formative, and developmental evaluation. Summative evaluation, generally, leads to the type of results that will tend to have more in common with the typical quantitative indicators that are used to identify progress. These do have a role to play, but tend to be lagging indicators, as highlighted earlier. Formative evaluation is not only more focused on process, but it is also concerned with more qualitative questions and understanding what is at work while one is still moving towards the goal. Developmental evaluation goes a step further,[10] encompassing both formative and summative evaluation, and drawing on a wide range of qualitative and quantitative data to make sense of what is happening within a given context, and support better decision-making. Ansell et al. (2022)[11] highlight the fact that developmental evaluation can be more appropriate when, for example, evaluating co-creation processes, given their fluidity and emergent qualities.
The G20 should act as a forum for sharing experiences of adopting specific indicators to ensure that potential hidden, unintended consequences are identified and enable rapid course correction. For example, Sampark in India has direct experience of the unintended negative consequence of tracking dropout rates in schools. Children from migrant communities are more vulnerable than non-migrant children as they lack access to education and other kinds of support.[c] Chances of children attending school are known to diminish if the family’s standard of living is low, the number of siblings is high, and the mother is needed to work as a labourer to support her family (Roy 2015).[d]
Sampark’s centres act as bridge schools for children as young as six months old, thereby reducing the potential burden of care on older siblings, who are freed up to focus on their education. The children are also mainstreamed into local government schools. When migrant construction workers’ children were not admitted to schools, Sampark traced the cause to the anticipated dropout rate of this group, an indicator, which becomes a black mark on school records. Sampark was able to resolve this issue by proposing that every child leaving school should receive a transfer certificate, which confirms the legitimacy of the child’s departure, enabling them to subsequently enrol in another government school. As a result, over the past three years, 160 children, who would otherwise not have been able to access education, were able to do so, thanks to transfer certificates.
The G20 should seek ways to repurpose existing data to improve the currency of information for tracking the progress towards SDGs, including the Global Entrepreneurship Monitor (GEM) perceptual data, and advocate for the same. Reliance must be placed on the collaboration among different sources of indicators/data, relevant for monitoring the progress towards achieving the SDGs. An effective partnership requires a common understanding of the governing conceptual framework as well as a rigorously designed and implemented research methodology. To stimulate discussion on how to reduce informational poverty in monitoring the achievements of the SDGs and to illustrate the power of repurposing existing information for additional insights, a group of researchers from the Global Entrepreneurship Monitor carefully analysed the relevance of GEM indicators for this purpose (Singer et al. 2022).
GEM is the world’s biggest survey, based on collecting perceptual data on entrepreneurship activities, using a standardised conceptual framework and a set of standardised surveying tools. The GEM indicators satisfy the criterion regarding the Tier II SDG indicators, as categorised by the UN (indicators are conceptually clear, and have an internationally established methodology, and standards are available, but data are not regularly produced by at least 50 percent of the UN member countries). Even with its limitations, GEM perceptual data should be considered a valuable contribution to the ‘freshness’ of the reference year in relation to the monitoring year.
The G20 should take the lead in developing studies that provide better understanding of how and why gender matters by delivering more sophisticated and nuanced insights. Singer et al. (2017)[12] used GEM data to analyse if and how gender influences differences in being innovative in entrepreneurial activities. The aim of this analysis was to identify the statistically relevant influencers to improve the understanding of the gender-sensitive patterns of innovativeness in the Croatian economy.
The assumed differences between ‘innovative’ and ‘non-innovative’ women and men were tested via three research hypotheses, looking at the difference between innovative women and innovative men as well as that between innovative and non-innovative men and women, respectively. Independent variables included personal demographics, personal attributes and societal values, firm demographics, and firm innovation strategies Dependent variables included gender, newness of products, newness of technology, and exposure to competition.
Statistical analysis confirmed all three hypotheses, providing a good starting point for further work, testing whether predicting growth potential by using intentions or some other identified influencing independent variables is gendered. At the same time, this is a commendable example of how the GEM data can be used to provide more sophisticated insights into how gender plays a role in entrepreneurship activities, and how to intervene in this process to release hidden potential.
The G20 should highlight the value of fostering individual agency and focus on influencing shifts in consciousness and tracking appropriate changes in individual behaviour, with a view to sharing effective interventions. The G20 Action Plan[13] refers to “bold transformative steps through both collective and individual concrete actions at international and domestic levels.” This is an important reminder of the individual responsibility to pursue the SDGs and contribute to the cumulative effect on the achievement of the SDGs. To avoid a typical ‘tragedy of the commons’ scenario, we must foster individual agency. With this in mind, the Inner Development Goals (IDG)[14] has been a transformative skills framework, co-created on the premise that the acquisition of the associated mindset and skill set will help individuals take action towards achieving the SDGs.[15] Building on this idea, in a recent report, the Conscious Food Systems Alliance (CoFSA) highlighted the importance of the inner drivers to human behaviour and the “narrative of separation” that governs the actions of many of us, suggesting that individual shifts in consciousness will change individual behaviour and support the development of regenerative food systems.[16]
This Policy Brief is taking this idea one step further by using the IDG framework, effectuation theory,[17] and the sense of agency scale[18] as proxies for measuring the impact of providing financial and non-financial support to individuals, participating in self-reliant groups or setting up their own ventures. Part of the rationale for this approach involves the desire to identify leading, as opposed to lagging, indicators. If we are able to use leading indicators to estimate the impact of SDG-related interventions and the potential progress towards achieving the SDGs, we have richer and more immediate data to help us learn from the interventions more quickly and make improvements along the way.
The G20 should promote the use of GEM data on entrepreneurial capacity as an example of a leading indicator. Singer et al. (2023)[19] showed how GEM data may reduce informational poverty while providing important insights into an individual’s capacity for entrepreneurial activity across the world, measured by different indicators enhanced via gender, education, and age. Registered businesses are disaggregated into three types of entrepreneurial activity—start-ups with fewer than 42 months of activities; established businesses, beyond 42 months; and entrepreneurial employee activity—to determine whether and how gender, age, and education play roles in such activities.
Importantly, the GEM conceptual framework defines entrepreneurial activity in terms of the interaction between an individual and their entrepreneurial institutional context. This is turn encompasses access to finance, education, research, professional services, simplicity of the regulatory framework, and supportiveness of government policies. The research of Singer et al. confirmed that context, education, age, and gender matter, thereby providing inputs for more effective and efficient evidence-based policies.
Attribution: Yolanda Gibb et al., “Innovative Approaches to Assessing Achievement of SDGs,” T20 Policy Brief, July 2023.
Endnotes
[a] The SDG tracker, accessed at https://sdg-tracker.org, provides a visual evolution of the key aspects of each SDG.
[b] This, according to the International Monetary Fund, was assessed on 22 March 2023.
[c] According to the 2011 Census, there are over 92.95 million migrant children in India. UNICEF India. (2020). “Child Migration in India” (Policy Brief). Retrieved from https://www.unicef.org/india/media/3416/file/Child-migration-India2020-policy-brief.pdf.
[d] Children migrating with their parents can reach dropout rates of 80 percent. Roy, A. K., Singh, P., and Roy, U. N. (2015). “Impact of Rural-urban Labour Migration on Education of Children: A Case Study of Left behind and Accompanied Migrant Children in India.” Space and Culture, India, 2, no. 4, 17–34. https://doi.org/10.20896/saci.v2i4.74.
[1] United Nations, UN National Quality Assurance Frameworks Manual for Official Statistics, (United Nations, 2019), https://unstats.un.org/unsd/methodology/dataquality/un-nqaf-manual/.
[2] Jeffrey D. Sachs, Christian Kroll, Guillame Lafortune, Grayson Fuller, and Finn Woelm, Sustainable Development Report 2021, 1st ed. Cambridge University Press, 2021, https://doi.org/10.1017/9781009106559.
[3] Jeffrey D. Sachs, Christian Kroll, Guillame Lafortune, Grayson Fuller, and Finn Woelm, Sustainable Development Report 2021.
[4] Janet Gale Stotsky, Sakina Shibuya, Lisa Kolovich, and Suhaib Kebhaj. “Trends in Gender Equality and Women´s Advancement” Working Paper, IMF, February 2016, https://www.imf.org/en/Publications/WP/Issues/2016/12/31/Trends-in-Gender-Equality-and-Womens-Advancement-43697.
[5] UNECE, Measuring and Monitoring Progress towards the Sustainable Development Goals, (UNECE, 2020), https://unece.org/unece-and-sdgs/publications/measuring-and-monitoring-progress-towards-sustainable-development-goals.
[6] Christian Alonso, Mariya Brussevich, Era Dabla-Norris, Yuko Kinoshita, and Kalpana Kochhar, “Reducing and Redistributing Unpaid Work: Stronger Policies to Support Gender Equality”, Working Paper 19, no. 225, IMF, October 2019, https://doi.org/10.5089/9781513514536.001.
[7] United Nations Statistical Commission, “Statistical Measures beyond GDP”, High Level Forum on Official Statistics, 54th Session of the United Nations Statistical Commission, Monday 27th February 2023. Accessed at https://unstats.un.org/UNSDWebsite/statcom/session_54/side-events/Side-event-Beyond-GDP.pdf.
[8] G20 China, G20 Action Plan on the 2030 Agenda for Sustainable Development, (G20 China, 2016), https://www.b20germany.org/fileadmin/user_upload/G20_Action_Plan_on_the_2030_Agenda_for_Sustainable_Development.pdf.
[9] G20 China, G20 Action Plan on the 2030 Agenda for Sustainable Development, (G20 China, 2016), https://www.b20germany.org/fileadmin/user_upload/G20_Action_Plan_on_the_2030_Agenda_for_Sustainable_Development.pdf.
[11] Christopher Ansell, Eva Sørensen and Jacob Torfing, “Evaluating processes, outputs and outcomes to learn and improve,” In Co-creation for sustainability. Emerald Publishing Ltd. (2022),151-164.
[11] Michael Quinn Patten, “Developmental Evaluation”, Evaluation Practice 15, no. 3 (October 1994), 311–19. https://doi.org/10.1177/109821409401500312.
[12] Slavica Singer, Šarlija, Nataša, Pfeifer, Sanja, and Oberman Peterka, Sunčica, “Gender Patterns of Businesses with Growth Potential in Croatia”, in Technology, Commercialization and Gender, eds P. Wynarczyk and M. Ranga, Palgrave Macmillan, 2017, 101-140, DOI 10.1007/978-3-319-49923-9_5.
[13] G20 China, G20 Action Plan on the 2030 Agenda for Sustainable Development, (G20 China, 2016), https://www.b20germany.org/fileadmin/user_upload/G20_Action_Plan_on_the_2030_Agenda_for_Sustainable_Development.pdf.
[14] Inner Development Goals: Background, method and the IDG framework, (IDGs, 2021) https://static1.squarespace.com/static/600d80b3387b98582a60354a/t/61aa2f96dfd3fb39c4fc4283/1638543258249/211201_IDG_Report_Full.pdf.
[15] European Parliament, Implementation and delivery of the sustainable development goals, European Parliament Resolution of 23 June 2022, (European Parliament, 2019-2024) https://www.europarl.europa.eu/doceo/document/TA-9-2022-0263_EN.pdf.
[16] Legrand T., A. Jervoise, C. Wamsler, C. Dufour, J. Bristpow, J. Bockler, K. Cooper, T. Corçao, N. Negowetti, T. Oliver, A. Schwartz, G. Steidle, S. Taggart, L. Søvold, J. Wright (2022), Cultivating inner capacities for regenerative food systems – rationale for action. United Nations Development Programme, https://www.undp.org/sites/g/files/zskgke326/files/2022-09/CoFSA%20Rationale%20Report.pdf.
[17] S. D. Saraswathy (2008), “Effectuation: Elements of Entrepreneurial Orientation,” Cheltenham: Edward Elgar Publishing, doi.org/10.4337/9781848440197, https://www.scirp.org/(S(czeh2tfqw2orz553k1w0r45))/reference/referencespapers.aspx?referenceid=2788004
[18] Adam Tapal, Ela Oren, Reuven Dar, and Baruch Eitam, “The sense of agency scale: a measure of consciously perceived control over one´s mind, body and the immediate environment” Frontiers in Psychology, Vol. 8, Article 1552, (September 12, 2017), https://www.frontiersin.org/articles/10.3389/fpsyg.2017.01552/full#:~:text=The%20Sense%20of%20Agency%20Scale%20(SoAS)%20was%20developed%20as%20a,body%2C%20thought%20and%20immediate%20environment.
[19] Singer, Šarlija, Pfeifer, and Oberman Peterka, “Gender Patterns of Businesses with Growth Potential in Croatia”.
20 Singer, Šarlija, Pfeifer, and Oberman Peterka, “Gender Patterns of Businesses with Growth Potential in Croatia”.
