Page 0: Page 1: 450 400 Average house prices (£ 000's) 350 300 250 200 150 100 50 England Five Host Boroughs Greenwich Hackney London Newham Tower Hamlets Waltham Forest 1995-01 1996-01 1997-01 1998-01 1999-01 2000-01 2001-01 2002-01 2003-01 2004-01 2005-01 2006-01 2007-01 2008-01 2009-01 Meta-Evaluation of the Impacts and Legacy of the London 2012 Olympic Games and Paralympic Games Jonathan France and Anna Woodham June 2nd 2011 2010-01 0 Page 2: Contents • Overview of the meta-evaluation – Objectives – Phases – Methods • Planning and prioritisation • The evaluation approach • Key areas for further research 2 Page 3: Study objectives Comprehensive and systematic meta-evaluation of the Impacts and Legacy of the London 2012 Olympic Games and Paralympic Games • Considering Sport, Economic, Social and East London, plus Disability and Sustainability • Outputs, intermediate results, and outcomes, and the additional impact and VFM of legacy investments • Tangible and intangible/intended and unintended effects • Lessons learnt around planning and delivery of a lasting legacy • Influencing role (interim evaluation) • Inform and advance the evidence base on hosting mega-events • To develop and enhance methods of meta-evaluation (ESRC supported) 3 Page 4: Overview of the 3 phases Phase 1: May 2010 – April 2011 Project Initiation • Project Initiation Document • Project Plan • Project set-up tasks Phase 2 : Feb 2011 – March 2012 Report 3 (Baseline) • Baseline • Counterfactual Phase 3: April 2012 – March 2013 Report 6 (Post Games Evaluation) • Baseline & counterfactual update • Evaluation synthesis • Primary research Report 4 (Interim Evaluation) Report 1 (Scope) • Identification of project scope • Research questions • Logic chains • Evaluation synthesis • Primary research • Initial modelling • Impacts and lessons learnt to date • Economic modelling • VFM • Lessons learnt • Longer-term impacts • Meta-evaluation methods • Data strategy Report 2 (Methodology) • Meta-evaluation methodology • Lessons learnt Phase 4 planned for (March 2013-2020) 4 Page 5: Contents • Overview of meta-evaluation • Setting up and planning the meta-evaluation – Project scope – Logic chains – Research questions • The evaluation approach • Key areas for further research 5 Page 6: Defining scope • Legacy initiatives: – New public programmes and investments developed as a result of Games – Existing programmes and investments refocused, expanded, or accelerated – Private and third sector investments inspired by the Games • Distributional impacts: – Geography (East London, Nations and Regions) – Groups (disabled, BME, young people, gender) • Temporal impacts: – Pre, during and post Games  Thoughts refined through desk research, DCMS workshop, stakeholder consultation and logic chains. 6 Page 7: Logic chains Rationale and Objectives London 2012 involves a major regeneration programme providing jobs, homes, infrastructure, training and opportunities for UK businesses. This helps overcome a coordination failure whereby firms and individuals are unwilling to invest in the area even though if they all did together they would benefit. Outcomes/impacts - Economic benefits – increased GVA, job creation and jobs safeguarded - Better health outcomes - Better quality of life/ improved wellbeing - More cohesive and inclusive communities - More sustainable communities. Gross to Net Conversion -Need to take account of additionality extent to which opportunities would not have been provided or not be available from other sources in the counterfactual scenario. -Need some consideration of extent to which programme of intervention is responsible for observed outcomes. -Adjustments for displacement, leakage, substitution, crowding out and multiplier effects, as relevant. The Games will also: - act as a showcase for British expertise and capabilities, overcoming similar problems with coordinated promotion to encourage inward investment, tourism and improve export potential; - provide a focal point for national identity, representing Britain and British culture to the world and creating civic and national pride that inspires new attitudes and behaviours; - inspire greater participation in sport, providing positive externalities through improved health. Activities - Sport - Harnessing the UK's passion for sport - Economy - Exploiting opportunities for economic growth - Social - Promoting social engagement and participation - East London - Driving the regeneration of East London. Outputs - Engagement with individuals (sports, culture and employment support) - Improvements to sports infrastructure - Engagement with businesses - Creation of opportunities (volunteering, disabled people etc) - Marketing and awareness campaigns - Building of venues according to new sustainable construction standards. Results - New participants in sport and cultural activity - People into jobs and gaining qualifications - More new businesses and more contracts traded with existing businesses - New inward investment to East London and UK - Increased visitor numbers to London and UK - Greater sense of community cohesion and inclusion - More sustainable behaviour of individuals and businesses - Improved accessibility for disabled people in accessing sport, culture employment and transport. 7 Page 8: Research questions Designed to tease out what the impacts have been in legacy areas participation in sport and physical activity the development of competitive and elite sport economic impacts, particularly in terms of employment and GVA CROSS CUTTING Changes in attitudes to disability; increased participation of disabled people in sport, economy, volunteering and culture Contribution to sustainable development Effects on well-being Effects on the international profile and reputation of the UK, London and East London social impacts, particularly in terms of volunteering and development of the 'big society' contribution to the regeneration of East London Benefits to target groups/communities value for money Whether impacts have been sustained Lessons learned about how to maximise the benefits to the host country and city from the staging of mega-events, particularly in terms of organisational lessons and change Page 9: This is a meta-evaluation • Requires robust programme evaluations, that meet needs of Meta-Evaluation, so… 1. Identify gaps or lack of granularity 2. Influence evaluations • Evaluation Steering Group • Work with Legacy Boards • Internal and external seminars 3. Ongoing assessment of relevance and robustness of project-level evaluations 4. Last resort develop own plans for primary research 9 Page 10: Contents • Overview of meta-evaluation • Setting up and planning the meta-evaluation • The evaluation approach – Framework – Value for money – Challenges • Key areas for further research 10 Page 11: Framework for assessing impact • Builds upon existing evaluation frameworks: – – – – Treasury Green Book input – output – outcome model Regional Development Agency Impact Assessment Framework Magenta Guidance PWC 2012 Games Evaluation Framework • Each logic model grounded in theory of change and common outcome measures, which can then be tested out through evaluation evidence • Issues of deadweight must be tested and will apply to all themes; displacement and substitution more to economic themes • Relevant impact assessment frameworks then devised for each theme, but top down and bottom up analysis required in all cases 11 Page 12: Sport: Logic chain Rationale and Objectives The 2012 Games provides an unparalleled opportunity to catalyse investment in new and improved sports infrastructure, as well as inspiring people to do more and achieve more in sport. Social and economic benefits can be derived from health benefits. Elite achievement in sport can deliver important benefits in terms of boosting national pride, providing sport role models, encouraging community and elite participation in sport, and improving the UK's reputation and influence abroad. The GOE’s Sports Legacy Strategy Paper, set out five promises: - Places: Transforming the places where people play sport, making the benefits of London 2012 visible in cities, towns and villages across the country - People: Inspiring people to make sport happen at the local level - Play: create the sporting opportunities and challenges that give everyone the chance to become part of the mass participation legacy. Outcomes/impacts - Increased participation in in active sport - Increased life expectancy and better health - Increased excellence and elite sport achievements - Increased sporting influence abroad Gross to Net Conversion Need to take account of additionality extent to which opportunities would not have been provided or not be available from other sources in the counterfactual scenario. Need some consideration of extent to which programme of intervention is responsible for observed outcomes. Activities Legacy activity is to take place in three areas: -Places: Upgrading up to a thousand local sports clubs and facilities;· investing in iconic multi-sport facilities; protecting and improving hundreds of playing fields. -People: Recruiting, training and deploying 40,000 sports leaders -Play: Motivate over 100,000 adults to test themselves in multiple Olympic and Paralympic sports;; investment in sport for disabled people. - Support for Elite and Internationalsport Outputs - Youths involved in Olympic and Paralympic style school sport competitions - Adults and young people involved in community sport and physical activity programmes - Coaches trained to support sports - Hours of volunteering for sports - Investment in elite sport and NGBs - Young people engaged in international sport - Bids for other sporting events Results - Increased access to sport opportunities, sport satisfaction, club membership - New participants in sport and less drop-off - Increased confidence of pupils, higher school attendance, and academic motivation - Better pathways and support to elite sport - Higher level of sport participation for disabled - Increased access to sport volunteering and higher interest and commitment in volunteering - Increased influence abroad - Mega sporting event bids won 12 Page 13: Data sources: collation; QA; synthesis; and aggregation NATIONAL EVALUATIONS National evaluations with a regional dimension eg. CompeteFor PRIMARY AND SECONDARY DATA Secure additional questions in surveys Commission studies if needed PROJECT MONITORING Project/programme monitoring information Input data per project MACRO ECONOMIC MODELLING Impacts of construction expenditure Defining counterfactual Estimating secondary (indirect and induced) impacts QUALITATIVE DATA Case studies 13 Page 14: Examples • Sport and social themes:  Amendments to national culture and sport surveys in England, facilitating statistical tests of association between sports participation and variables measuring engagement in the Games; medals performance; time-series data/scenarios for volunteering  Bottom-up evidence from programme evaluations; post Games venue planning; selective primary research for Games Makers (survey), disability (media analysis) and sustainability (case studies) • Economy and East London • Mix of statistical techniques (differences in differences/control areas; economic modelling); amendments to national tourism surveys; UK trade and investment data • Bottom-up evidence from programme evaluations; stakeholder interviews (SAV); academic studies; resident and business surveys 14 Page 15: Valuing benefits Techniques include: • Benefits transfer (other studies) • Econometric modelling Benefits/Outputs 2012 Games (with legacy) 2012 Games (Without legacy) No 2012 Games Baseline • Revealed preference techniques • Stated preference techniques On benefits transfer, some benchmarks available through the DCMS CASE programme: • Doing sport once a week generated Subjective Well Being equivalent to £11,000 increase in household income; • Health cost savings of doing sport are £5,000 for a 30-39 year old. 15 Page 16: Thematic challenges • Sport – Measuring the sustainability of participation (or use of proxies) – Methodologies to analyse elite sport networks are underdeveloped – Top-down sport for development highly criticized and under-evaluated • Social – Lack of participant research – access to LOCOG Games Maker database – Wider influences, e.g. Big Society • Economy – Aggregating the short-term and ongoing impact of the Games on economy – Assessing the longer-term influence of the Games on image/competitiveness; – need to influence tourism agencies to capture impacts over time – wider impact of the Games on inward investment and export trends (longitudinal approach preferred) 16 Page 17: Broader challenges • Questions have been prioritised, but scope still vast and complex • Internal organisation and systems • What is additional? • Influencing, building support (e.g. LOCOG, academic community), access • Synthesis or aggregation of outcome data? • Assessing additionality • Developing counterfactuals without evaluations • Developing counterfactuals with evaluations • Geographical and E&D sensitivities • Mega-events have not previously been measured in such a comprehensive way: no real precedent 17 Page 18: Contents • Overview of meta-evaluation • Planning the meta-evaluation • The evaluation approach • Key areas for further research 18 Page 19: Areas for further research: Theme East London Primary Data Requirements • Household survey and focus groups with residents • Private sector leverage • Stakeholder lessons from park conversion and convergence Economic • Visitor experience survey • FDI attracted • Sustainability of employment outcomes • Impact of infrastructure investments • Elite Sport contribution Sport Community Engagement Disability and Sustainability Nations and Regions • Inspire Programme outcomes • Games Maker survey • Perceptions of disability • Additional small scale primary research • Pre-Games Training Camps impact • Local Authority 2012 activities and outcomes 19 Page 20: How you can get involved • Share knowledge on existing and planned 2012 evaluations and research • Share findings of 2012 project level evaluations and research • Influence surveys and research work planned • Views and input on methodology and alternative approaches • Contact: anna.woodham@culture.gsi.gov.uk 20 Page 21: Page 22: Questions… 22 Page 23: