Impact Evaluation of Income Support Program for the Poorest Draft Methodology The objective of this note is to propose a methodology to conduct an assessment of the interventions under the Income Support Program for the Poorest (ISPP). The impact evaluation (IE) study will provide evidence on the operation of the program as well as the impact of cash transfers on (i) the socioeconomic conditions and food security of beneficiary families; (ii) child nutrition; (iii) child cognitive development, and (iv) readiness for school. The assessment will consist of three components: (i) an impact evaluation using baseline and endline information on a sample of beneficiary and control households (household survey, growth monitoring, and cognitive development tests); (ii) qualitative assessments throughout the implementation of the project (community surveys and focus group discussions), and (iii) a mid-term evaluation of children’s cognitive performances in beneficiary and control households (Bayley test only) if funding is sufficient. The evaluation will be carried out between June 2017 and June 2020, and the results of the assessment will provide feedback to improve the program design and operation for the next cycle of scale up. ISPP implementation The ISPP aims to provide income support to the poorest mothers in 43 Upazilas1 of Bangladesh, while (i) increasing the mothers’ use of child nutrition and cognitive development services, and (ii) enhancing local level government capacity to deliver safety nets. To achieve its objective, the project will finance quarterly cash transf ers to up to 600,000 eligible households conditional on utilizing the following services: (i) up to 4 ante-natal care (ANC) visits by pregnant beneficiaries; (ii) monthly growth monitoring and promotion (GMP) of children from 0-24 months; (iii) quarterly GMP for children from 2 to 5 years of age, and (iv) monthly attendance at child nutrition and cognitive development (CNCD) awareness and counselling sessions by all enrolled mothers. Table 1: Timeline of ISPP implementation Jan 2017/Jun Jan 2018/June Jan 2019/Dec Jun 2020 Jun 2020 2020 2020 Phase 1 Phase 2 Phase 3 Selection of treatment areas 1Upazila = sub-district. The upazilas were selected on the basis of high poverty and high probability of malnutrition. Implementation of ISPP will be phased-in, and progressively cover all 43 Upazilas: 15 will start receiving transfers from July 2017 (Phase 1), 14 will start receiving transfers from July 2018 (Phase 2), and 14 will start receiving transfers from July 2019 (Phase 3). All Upazilas will receive the transfers until program closure, i.e. June 2020. Table 1 provides a summary of the phased-in implementation. Upazilas were randomly selected into each of the three phase groups (see Annex 1 for a complete breakdown).2 Applications A public information campaign (PIC) about the intervention will be put in place in all upazilas, ahead of registration and enrollment (January-June 2017 for Phase 1; January-June 2018 2018 for Phase 2, and January-June 2019 for Phase 3). The PIC will inform potential beneficiaries about the program, eligibility rules, where to enroll, what documentation to bring, etc. Selection of beneficiaries Only the poorest households are eligible to participate in the ISPP program. As such, poverty scores of applying households will be the first determinant in eligibility. The PSC, which uses a proxy-means test (PMT)3 score to approximate the household’s level of income/poverty, is currently being implemented by the Bangladesh Bureau of Statistics (BSS), and is expected to be available ahead of household enrollment in Phase 1. A unique eligibility threshold is expected to be implemented across all Upazilas, corresponding to the bottom 30 percent of the population (or an income level below BDT 1,533.8 per person per month): households with a PMT score below -60 points (equivalent to an income level below BDT 1,533.8 per person per month) will be eligible to participate in the program, while households with a PMT score above -60 points will not be considered poor enough. Out of the eligible households, only families with at least one pregnant women and/or at least one child below the age of 5 will be enrolled in the program. The number of potential beneficiaries will then determine the total budget allocation needed for each participating Union, and thereby help predict the budget allocation needed for control unions when they enter the program in Jan/May 2018 (Phase 2) and Jan/May 2019 (Phase 3).4 2 Simple (non-stratified) random draw. 3 PMT is a targeting method used across many countries. It determines the eligibility of households for social assistance through the calculation of a poverty score based on observable, measurable and verifiable household characteristics such as household de mographics, dwelling and durables owned. The selection of observable variables, and associated weights will be done using the latest household income and exenditure survey (HIES 2010). 4 Because there is no prior information at the union level regarding the number of potential beneficiaries, applications will happen prior to budget allocation in the first year (2017). In the following years, budget allocation will be based on the initial estimated number of beneficiaries and will hence be done prior to the application process. Using ISPP design for the identification strategy The identification of treatment and counterfactual groups for the impact evaluation will use the design of ISPP: (i) the rolling out of the transfers in three phases, and (ii) the eligibility threshold. The proposed identification strategy will focus on Phase 1 and Phase 3 Upazilas only. 15 upazilas – that is 159 unions – will be covered by ISPP from January/February 2017 to June 2020. In addition, 14 upazilas – that is 135 unions – will be phased into ISPP two years later.5 The treatment group will consist of eligible households in Phase 1 unions: those who apply to the program and whose poverty score is below the eligibility threshold. Using the phased-in implementation: Treatment and Control 1 Difference-in-differences and randomized control trial The control group (Control) will consist of eligible households in Phase 3 unions, i.e. households whose poverty score is below the eligibility threshold in areas where the program is not available until the last year of implementation. The proposed identification strategy (difference-in-differences (DiD) combined with randomized control trial (RCT)) is based on the assumption that the randomization into phase groups at the Upazila level guarantees that eligible households in Phase 1 and Phase 3 unions are similar before program implementation, and that all changes in the outcomes of interest between the two groups can be attributable to ISPP. Using this identification strategy will ensure that there are no spillover effects between treatment and control groups. Data collection and sampling Main questions to be answered The proposed IE will collect and analyze baseline (July/December 2017) and endline (July/December 2019) data at the household level to provide insights on the operation of the CCT program, as well as seek to answer the following questions: does participation in ISPP lead to an increase in household consumption, food and protein intake, and dietary diversification? Does it improve anthropometric outcomes? Does it improve a set of child cognitive and language developmental outcomes (measured by Bayley Scales of Infant and Toddler Development-III, and the Wechsler Preschool and Primary Scale of Intelligence)? Does it lead to increased and early enrollment in primary school? Does it lead to improved women’s decision -making ability? 5 For the purpose of the IE, a randomization at the union level is preferred over a randomization at the Upazila level, as it gives more power and reliability to the final results of the IE. Unfortunately, from an implementation perspective, that option was not feasible, and randomization could only be implemented at the Upazila level. The evaluation will also look into the relative cost-effectiveness of each arm of the intervention (ante-natal care, growth monitoring and promotion, cognitive development) with respect to their impact on child development outcomes and readiness for school. This is particularly important given the innovative payments system, which is likely to have significant implications on the efficiency and cost of implementing CCTs. For both, baseline and endline, data collection will consist of a household roster, full consumption questionnaire, health and nutrition module, anthropometric measurement, and child cognitive developmental outcomes for children aged 0 to 24 months at the time of the baseline.6 In addition, focus groups discussions and facility surveys will be conducted to provide complementary information. Finally, data on cognitive development only may be collected between the full-fledged baseline and endline surveys to monitor the progress of children towards ECDN outcomes. This exercise will depend on securing additional funding. The timeline of the program is such that the IE team chose not to conduct a midterm household survey as most outcomes of interest are unlikely to change within 1.5 years (evidence from the Shombhob pilot). The endline is scheduled for Spring/Summer 2019 just before Phase 3 upazilas start receiving ISPP transfers, to make sure they are not yet affected by the program. In addition, conducting a follow-up survey mid- 2019 will enable the team to finalize the IE report a few months before ISPP closes (June 2020), and will inform the implementation team on what works and what does not before any future scale up of the program. Sampling Different options were considered by the evaluation team, preliminary power calculations were conducted using the following assumptions: 28 Upazilas (14 in Phase 1 and 14 in Phase 3), a minimum detected effect (MDE) of 0.13 (based on previous estimates from the literature), a significance level of 0.05, and an intra- cluster correlation (ICC) of 0.02 to 0.15.7 Based on power calculations using the clusersampsi command in Stata, and assuming 6 clusters per upazila the minimum sample size would range from 1,176 to 6,384 households altogether. The impact evaluation team decided to keep 6 clusters per Upazila – i.e. a total of 168 clusters – each of which would consist of 20 households. The total sample size 6 While the program is available to all children up to 60 months of age, the IE team decided to focus on the youngest children for 2 reasons. Firstly, researchers have identified the first 1,000 days of a child’s life – from pregnancy through a child’s secon d birthday – as the most critical window of time that sets the stage for a person’s intellectual development and lifelong health. Indeed, evidence from the Shombhob pilot show that improvements in anthropometric measures can only be seen among the youngest cohorts that benefited from the program (0-12 months at enrollment). Second, children who are older that 24 months when the baseline is conducted will be older than 60 months when the endline survey, and will have already exited the program. 7 The data from HIES shows that ICC ranges from 0.015 to 0.17 (per capita consumption, i.e. the variable we use to look into the impact of the program on poverty reduction) in the upazilas that will be covered by ISPP and for which there is information in HIES 2010. consists of 3,360 households (6*20*28). This size is well above the minimum sample size of 1,176 corresponding to an ICC of 0.02, and corresponds to an ICC of 0.127. Sampling will be conducted among households with pregnant mothers and/or children less than 24 months of age. While the program is available to all children up to 60 months of age, the IE team decided to focus on the youngest children for 2 reasons. Firstly, researchers have identified the first 1,000 days of a child’s life – from pregnancy through a child’ s second birthday – as the most critical window of time that sets the stage for a person’s intellectual development and lifelong health. Indeed, evidence from the Shombhob pilot show that improvements in anthropometric measures can only be seen among the youngest cohorts that benefited from the program (0-12 months at enrollment). Second, children who are older than 24 months when the baseline is conducted will be older than 60 months when the endline survey, and will have already exited the program. Cost estimates (using a 4,000 household sample) Estimated baseline budget: USD 570,400 Estimated midline budget: USD 309,400 (if funding becomes available) Estimated endline budget: USD 580,000 Estimated total budget: USD 1,104,000 (if funding becomes available: 1,413,400) Timeline Table 2: Timeline of ISPP implementation 01- 02- 03- 01- 02- 03- 01- 02- 03- 06- 17 17 17 18 18 18 19 19 19 20 Phase 1 Phase 2 Phase 3 06-17: 06-19: Baseline Endline HH survey + HH survey + Bayley Bayley Public Information Campaign Application/selection (and rolling) Enrollment of beneficiaries (and rolling) Transfers (and rolling) Final report for Impact Evaluation Qualitative surveys (FDGs, community and facility surveys): throughout implementation Annex 1: List of Upazilas by phase Phase Division District Upazila Unions Mymensingh Jamalpur Melandaha 11 Mymensingh Jamalpur Bakshiganj 7 Mymensingh Jamalpur JamalpurSadar 15 Mymensingh Mymensingh Dhobaura 7 Mymensingh Mymensingh Muktagachha 10 Mymensingh Mymensingh Phulpur 10 Mymensingh Mymensingh Tarakanda 10 PHASE ONE Mymensingh Mymensingh Haluaghat 12 Mymensingh Mymensingh Nandail 12 Mymensingh Sherpur SherpurSadar 14 Rangpur Gaibandha GaibandhaSadar 13 Rangpur Gaibandha Gobindaganj 17 Rangpur Kurigram Raumari 5 Rangpur Kurigram Chilmari 6 Rangpur Kurigram Bhurungamari 10 Total 15 159 Mymensingh Jamalpur Sarishabari Upazila 8 Mymensingh Mymensingh Ishwarganj 11 Mymensingh Mymensingh Fulbaria 13 Mymensingh Mymensingh Trishal 12 Mymensingh Sherpur Nakla 9 Rangpur Gaibandha Saghata 10 PHASE TWO Rangpur Gaibandha Sundarganj 15 Rangpur Gaibandha Sadullapur 11 Rangpur Gaibandha Palashbari 9 Rangpur Kurigram Ulipur 13 Rangpur Kurigram Phulbari 6 Rangpur Kurigram Rajarhat 7 Rangpur Kurigram Nageshwari 14 Rangpur Nilphamari Jaldhaka 11 Total 14 149 Mymensingh Jamalpur Madarganj 7 H H R A T P E E E S Mymensingh Jamalpur Dewanganj 8 Mymensingh Jamalpur Islampur 12 Mymensingh Mymensingh Gaffargaon 15 Mymensingh Mymensingh Gauripur 10 Mymensingh Mymensingh Mymensingh Sadar 13 Mymensingh Mymensingh Bhaluka 11 Mymensingh Sherpur Sreebardi 10 Mymensingh Sherpur Nalitabari 12 Mymensingh Sherpur Jhenaigati 7 Rangpur Gaibandha Fulchhari 7 Rangpur Kurigram KurigramSadar 8 Rangpur Kurigram Char Rajibpur 3 Rangpur Lalmonirhat Hatibandha 12 Total 14 135 Grand Total 43 443 Annex 2: Distribution of households’ consumption Table 2.1: Rural Dhaka and rural Rajshahi Note: the vertical lines represent the 30th and 40th percentile cutoffs (respectively BDT1,553.8 and BDT1,767.5 per person per month). Source: HIES 2010. Table 2.2: Households form the pilot program - Jaldhaka Note: the vertical lines represent the 30th and 40th percentile cutoffs (respectively BDT1,553.8 and BDT1,767.5 per person per month). Source: Shombhob impact evaluation baseline.