Measurement of PYD Constructs:

What processes or strategies are critical for adaptation of reliable and valid PYD tools to different contexts?

How do we determine whether a measure found to be valid and reliable in one context is appropriate to be adapted to a different context?
What are the costs and benefits of adapting an existing valid and reliable tool compared to creating a new tool more tailored to the context?

The last few years have seen several advances in measures of PYD constructs in a variety of contexts. However, the YouthPower Action review and inventory of measures of soft skills demonstrated that not every measure is appropriate for program evaluation when a) measuring change over time, and b) comparing PYD constructs between program participants and some appropriate comparison group. Several steps are required to determine if a measure is appropriate for adaptation to a new context. As Galloway et al point out, it is important to assess the evidence that the measure is valid and reliable. Determining if the measure is appropriate for the purpose intended is also important (e.g., program evaluation based on group comparisons). So far, we have found no information specifically about the costs and benefits of measurement tool adaptations to new context vs. the creation of an entirely new tool.

Consistent and reliable measurement of soft and life skills

Much of the PYD programming in LMICs focuses on helping young people develop soft and life skills which will allow them to be more successful in all aspects of their lives. YouthPower Action published a report on “Key Soft Skills for Cross-Sectoral Youth Outcomes” which highlighted the importance of and the need for consistent and reliable measurement of these skills for program monitoring and evaluation. They subsequently published “Measuring Soft and Life Skills in International Youth Development Programs: A Review and Inventory of Tools,” which reviews measurement tools of the seven identified skills and is based on key criteria for use in international youth development programs. This extremely valuable review of measures identified 74 measurement tools that are free (no monetary costs) to access and designed for youth between the ages of 12 and 29. The three primary skills with cross-sectoral impact were positive self-concept, self-control, and higher order thinking skills. The four additional skills were identified within one or two of the three sectors were social skills, communication, goal orientation, and empathy. The reviewers applied seven criteria that produced a rubric for scoring each measure. The value of this approach is not that it identified the ‘best’ measure or even the ‘most useful’ measure, but that it highlights the wide range of available measures and provides a way of thinking about how they are alike and different from each other. The criteria focus on ease of administration and evidence of validity and reliability, as well as appropriateness for youth development programs in LMICs targeting youth developmental outcomes of current interest.

Importantly, the review also highlights the different purposes for measuring soft skills and how the important features or qualities of a measure will be different for different purposes. These differences are especially relevant when conducting program evaluations because not all good measures are good at detecting change over time or distinguishing between groups. Finally, this review and inventory of measures provides recommendations for the development of a self-report instrument designed specifically for use across programs and contexts so that much needed comparisons can be made. The recommendation also includes the development of supporting non-self-report measures and strategies for reducing bias in self-reports. While the report is limited to the seven skills identified for impact in three related sectors (i.e., work force development, sexual and reproductive health, and violence prevention), it makes a strong contribution to the field of measuring PYD in LMICs in general by drawing attention to a set of challenges shared by PYD measures:

over reliance on youth self-report
lack of consensus on terminology and definitions
lack of adequate evidence for tool validity and reliability
response options and scoring methods that are sensitive to change over time
developmental appropriateness across the wide range of ages considered to be ‘youth’

Adapting the Developmental Assets Profile Across Contexts

In 2017, Scales, Roehlkepartain, and Shramko reported on the use of the 58-item Developmental Assets Profile (DAP) in 31 countries over the preceding decade. They describe the process of translating and adapting the questions and discuss the primary problems encountered in doing so as well as provide detailed tables on reliability and validity in each country. With some exceptions, the measure performed well in terms of internal consistency, short-term stability, and convergent and external validity. World Vision International and the Search Institute collaborated to create a short version of the DAP which could be more easily administered in emergency situations, such as refugee camps and war zones where the full 58-item DAP is impractical. In a 2015 article in the Journal of Adolescent Health, Scales and colleagues reported promising psychometric results on the 13-item Emergency Developmental Assets Profile. Applying four criteria to each of the original items (i.e., representation of each of the eight original asset categories, strong correlation with the full DAP, ease of translation to many languages, and acceptable reliability across cultures) resulted in 10 items selected for testing. These were supplemented with items developed based on qualitative interviews with children in emergency situations in five countries in the Middle East and Africa. The final list of items was tested using data from Syrian youth in refugee camps in Iraq, Jordan, and Lebanon, as well as Philippine youth displaced by typhoon Haiyan in 2013. The items were found to be internally consistent, be reliable across contexts, and have moderately convergent validity. Multiple translations of the full DAP are now available. So many youth live in economically-volatile, politically-unstable, and/or high-conflict areas that the development of a brief version of the DAP to use in emergency or humanitarian crises fills an important gap.

Measuring the Five Cs

Compassion International (CI) has undertaken a longitudinal evaluation of their faith-based programming for youth in El Salvador explicitly based on PYD and applying the Lerner and Lerner model of the Five Cs of PYD—Competence, Confidence, Character, Caring, and Connection. They have recently reported psychometric properties on their measure of PYD using the 34-item short form measure of the Five Cs of PYD with comparisons between program recipients and similar youth not currently receiving their services. They found strong support for measurement invariance across age groups (i.e., 9-11-year old and 12-15-year old) and genders for this measure. These were preliminary results within a single country with limited diversity among participants, but this project will evolve to include longitudinal data cross several countries as CI continues to evaluate its programming for youth living in poverty across the world.

Next Steps Towards Answering This Question

Given the progress made in the measurement of self-report strategies especially in the Assets domain (i.e., soft and life skills), we focus more attention on measures of the enabling environment. The environment is a particularly important but challenging domain to measure. For example, we often rely on youths’ reports of feeling safe and supported, and those perceptions are very important to understanding the role the environment plays; however, it may not reflect whether the environment is actually safe and supportive. Below are some suggested next steps:

Development of observation-based measures and use of community level data to assess important qualities of the environment more objectively. PYD measures are highly dependent on youths’ self-reports because some of the constructs are based on beliefs and perceptions. However, the over reliance on self-report measures leads to potential bias toward social acceptability. Self-reports often reflect what one thinks is the ‘right’ answer and/or how a person is feeling at the time they complete the survey or interview. Measures using observations or community-level data might help to differentiate between actual safety and the perception of safety or danger.
Better formative understanding of multiple aspects of the enabling environment. Social norms are a good example of an aspect of the enabling environment that needs more theoretical clarification before improved measurement is likely. Unlike the case of soft skills where the field is poised to develop a standard language and measure to use across contexts and cultures, for some qualities of the environment, standardization of measurement is premature. Next steps should include more clarity and specificity about the theoretical models used and why, and comparison of different measurement strategies.

Download the Learning Agenda.