Measuring Impact

At BAP we've been, and remain, committed to effectively monitoring and evaluating our impact. In 2015-2016 we measured how customers used their bicycles and set out to analyze their financing standing, access to healthcare, and household health and hygiene. The study recently wrapped up and we'll publish the results soon enough. 

We're refining our M&E data techniques and have honed in on a continuous collection of income / expenditure metrics. This coincides perfectly with the data we collect and analyze to determine credit worthiness. In our application and appraisal process our Loan Officers investigate how much people spend, save, earn, and invest. Since we are doing this for ALL customers in the beginning we decided to use this as the baseline of our M&E. Thirty percent of each disbursement cohort will be tracked over time and receive an end line appraisal for comparison.

The document below explains our goals, methodology, and assessment techniques in more details. 

Objective

To gain a better understanding of economic benefits of asset financing for rural farmers.

Theory of change

BAP’s theory of change (documents below) outlines the pathway in which asset financing to smallholder farmers creates an increased financial standing and higher income. For example, with bicycles we see savings associated with using the bicycle as a means of transport as a means to increased financing standing. For, oxen and plough, however, the ability to increase yields leads similarity to increased financing standing.

The theory of change is an important tool as it guides BAP’s monitoring and evaluation strategy to identify key pathways to our organizational goal. Increased financial standing is measured by assets, expenditures, and income.  The increased income and capital associated with the asset received is likely to have four main effects: higher expenditures on household goods (i.e., improved quality of life); increased agricultural inputs; increased chance of starting an off-farm business; and more capital to pay for education.

Impact assessment approach

BAP will assess the impact of its programming by comparing key indicators from a sample of beneficiaries before and after they receive a bicycle. We will also be conducting a post-end line survey for larger products. Despite the popularity of randomized control trial (RCT) approaches to impact assessment, BAP will not, at this time, use such an approach. BAP is currently maximizing its resources by distributing selecting beneficiaries in a targeted, non-random manner. Without randomly selecting participants to join the program an RTC is not a viable option.

BAP is looking into the possibility of using new clients entering the program as a control group to ensure any changes are attributed to the introduction of the bicycle. This would ensure if an entire area had unusual good farming season, we would be able to see hiring incomes in both our end-line group as well as the new clients joining the group. There are several hurdles to this method as it assumes that the new clients joining BAP and our former clients were within the same income group when joining BAP. If BAP’s geographic area, partners, or methods to finding suitable bicycle loan groups does not change throughout the year, this may be a feasible option for BAP.

Sample size

We use Yamane’s (1967) standard simplified formula for calculating sample sizes. We calculate this at a 95% confidence level. Therefore, for our annual population of bicycle 900 bicycle recipients we plan to sample 277 participants.

For oxen and plough, the participants will be higher. If our total population size for the year is 55, we will survey 48 of the participants.

Data collection methods and technologies

Survey

BAP will be collecting quantitative data using our already existing application and appraisal process. These surveys include basic household biographic information as well as expenditure, assets, income, and agriculture outputs. Therefore, all BAP clients would have taken the baseline survey in the form of these applications and appraisals. This will greatly assist BAP in saving time and resources. See the supporting questions to view which questions from our existing application and appraisal BAP will be using for impact evaluation purposes.

End-line clients will be randomly selected from each loan disbursement cohort. The end-line will mirror the baseline and include additional questions focusing on our services.

As the bicycle clients loan is for 10 months, clients will receive their end-line survey 12-months after to ensure that we do not see changes purely related to the season. The purpose of these surveys is to ensure that BAP clients are experiencing positive changes relative to their state prior to receiving a BAP loan.

Technology

BAP will use SurveyCTO, an offline survey tool, for the end-line insert. However, BAP for the baseline and end line appraisals we will be using the same technology BAP uses for the regular program. At this time, BAP is using paper surveys due to the complexity of information asked.

Acknowledgments

Our end-line insert uses questions from the SEEP network’s “Assessment Tools for Microfinance Practitioners.” BAP has also received guidance in formulating its M&E from Dr. Brian Thiede, Assistant Professor of Rural Sociology at PennState, and Dr. Admos Chimhowu, Senior Lecturer at the Global Development Institute at University of Manchester.