** Archive **
WIAGL 8 (WIA Youth Performance Measures)
Workforce Investment Act Guidance Letter No. 8
December 5, 2003
TO: Local Elected Officials and WIA Administrative Entities
FROM: Thomas J. Hayes, Director Ohio Department of Job and Family Services
SUBJECT: WIA Youth Performance Measures

I.Purpose

To provide local elected officials and WIA administrative entities information regarding Workforce Investment Act (WIA) Youth Performance Measures.

II.Effective Date

Immediately

III.Guidance Statement

This guidance has been prepared to help local areas understand the WIA Youth Performance Measures. We expect that the performance measures will be meaningful indicators of how our workforce system is serving its customers, but performance measures should never become an end in itself. Rather, the focus should be on the needs of the individual and not performance measures themselves. Local programs should be a product of well-designed and well-implemented systems, responsive to local needs and both building on and expanding local opportunities.

We hope that the process of designing local programs will be highly collaborative, focusing on customer needs and the resources available for meeting those needs. Within that process, the performance measures should be seen as parameters within which programs must operate- much like allocation size and applicable laws and regulations.

The WIA performance measures are one component in the comprehensive WIA performance system. Together components focus attention on: the importance of performance accountability at all levels, the interdependence of partner programs and the right of the public to valid consumer information and quality service. But most of all, the performance

measures are important because they represent the aspirations of our customers - for skill development and acquisition of meaningful credentials, for stable employment over time, for improved earnings, and for positive experiences when they seek our help.

Youth Performance Measures - General

The US Department of Labor (DOL) has outlined seven specific measures for youth and youth are also included in the Customer Satisfaction measures for Participants and Employers. There are different measures for different age groups.

Older Youth (19 - 21)

Entered Employment

6 Month Retention

6 Month Earnings Change

Credential Rate

Younger Youth (14 - 18)

Skill Attainment Rate

Diploma or Equivalent Rate

Retention

Those for Older Youth are more employment oriented, though they recognize the importance of further education. Those for Younger Youth are less employment oriented and focus on skills and education development, though they reward program directions that should result in better employability. A more detailed look at each of these measures will take place later in the document.

The USDOL selected these seven measures for some of the following reasons:

1.The research shows what gets measured is what gets done

2.Effective youth programs require longer term interventions with youth

3.Goal setting is a valuable life skill and is a learned behavior

4.They wanted focus on skill attainment, educational advancement, credentials and job placement

5.Follow-up was important to insure continued success and development

This document is intended as guidance for local areas but the most current and comprehensive federal information on WIA youth performance measures is contained in three federal documents: USDOL Training and Employment and Guidance Letters (TEGL's) No. 7-99 , 8-99 and 7-01. These TEGL's and other timely information can be viewed and/or downloaded from http://wdr.doleta.gov/directives/.

Registration and Exit

Registration. All youth receiving services of any level under the WIA Title I youth funding stream must be registered. Registration is the process of collecting information to support a determination of eligibility. Local areas can collect information as part of the pre-registration process to determine whether the youth would benefit from participation in WIA services prior to registration.

An individual can be registered for both Adult and Youth services simultaneously if they meet the eligibility requirements for both funding streams. In the case of a youth participant who also becomes an adult participant, the individual will be counted in both the applicable youth measures and adult measures. Neither set of exit-based measures will be applied until that participant has exited both adult and youth services (as well as all other WIA Title I or partner services). This is because exit-based measures assess a participant's outcome following their full range of services, even when those services are from different program areas.

Classification as a younger youth or older youth is done at registration and does not change regardless of the individual's age at exit. The Individual Service Strategy (ISS) goals are appropriate as long as the youth continues to receive services identified in their ISS and has not attained the outcomes established for that age group.

Examples:

1.If an individual is 16 on the registration date and the services are Youth funded - counts in Younger Youth performance measures

2.If individual who is 19 is served in both Youth and Adult funds - they count in both Older Youth and Adult performance measures

3.If individual is enrolled between ages 14 - 18 and turns 19 while enrolled - counted in Younger Youth performance measures at exit

Once a youth registers for WIA Youth Services, the local area can get credit for partner services. To encourage integration of services and recognize shared contributions toward outcomes, USDOL recommends the following strategy for tracking and reporting across WIA Title I funded (WIA funded programs) and other workforce investment partners.

Once an individual has registered for WIA funded services, programs can claim credit for outcomes on core measures attained by participants who receive non-WIA funded services (such as those offered by One-Stop partner programs).

The non-WIA-funded partner programs that are included in the performance measurement system must be party to local Memorandums of Understanding (MOU).

Each partner program must have the capacity to track registered WIA participants until the individual exits all WIA and non-WIA-funded partner services. Care must be taken to maintain current records on service participation through case management or tracking systems. Some methods for tracking participants across programs include specifying the non-WIA funded services in the individuals' ISS, coordinating services across WIA and non-WIA funded services, and providing follow-up services to individuals.

Point of Exit. For all of the measures (except the younger youth skill attainment rate and employer customer satisfaction measure) the term "exit" is used to determine when to count an individual in a specified reporting period. There are two ways to determine exit during a quarter:

1.A participant who has a date of case closure, completion or known exit from WIA-funded or non-WIA funded partner service within the quarter (hard exit) or

2.A participant who does not receive any WIA-funded or non-WIA funded partner service for 90 days and is not scheduled for future services except follow-up services (soft exit). This definition is designed to encourage program operators to keep in close contact with all youth enrolled in the program until they complete all services.

Participants who have a planned gap in service of greater than 90 days should not be considered as exited if the gap in service is due to a delay before the beginning of training or a health/medical condition that prevents an individual from participating in services. Service providers should document any gap in service that occurs with a reason for such a gap in service. Participants who exit from services because they are incarcerated, deceased, or are reservists called to active duty and choose not to return to WIA, or have a health/medical condition that prevents the individual from participating in services, should be excluded from the measures. Once a participant has not received any WIA funded or partner services for 90 days, except follow-up services and there is no planned gap in service or the planned gap in service is for reasons other than those specified above, that participant has exited WIA.

Timing of Performance Measures and UI Wage Data

Timing of Performance Measures. There are two types of time periods that affect WIA Youth Performance measures. There are "real time" measures (Younger youth skill attainment, Younger Youth High School or equivalent, and customer satisfaction) and "long term" measures (Older Youth - entered employment, retention, earnings and credential and Younger Youth retention rate). The final results of the long term measures will not be known until some time after the individual exits the program.

Due to the use of Unemployment Insurance (UI) wage data as a source of much of the data, reporting for WIA, outcomes are reported quarterly. Exits are counted during calendar quarters and not necessarily on the exact date that they are reported in the system. The Exit Quarter is the calendar quarter in which the last date of service takes place. Performance calculations are based on the Exit Quarter, with the exception of younger youth skill attainment rate and employer customer satisfaction measure. An individual who exits in January will be counted as exited during the January - March quarter so that the 3rd quarter after exit would be October - December of that same year. (See attachment A). Each individual becomes part of an exit cohort, a group who are determined to be "exiters" within a particular quarter and are looked at together for measurement purposes. When performance measures are calculated the exit cohorts are kept together as a group as time passes depending on each measure.

As explained earlier, the exit date is the last date of WIA funded or partner service received (except follow-up services). For a soft exit, the exit date cannot be determined until 90 days has elapsed from the last date of service. At that point, the exit date recorded is the last date of service and not at the end of the 90 day period. The exit quarter is the quarter in which the last date of service (except follow-up services) takes place. If a participant exits WIA and receives additional WIA services, (other than follow-up) after exiting, that participant is treated as a new participant for purposes of the core measures and will be included in the appropriate measures. The definition of exit applies to all of the core measures except the younger youth skill attainment and employer customer satisfaction measures. Please see Attachment B for a chart showing the timing of WIA Youth Performance Measures.

Unemployment Insurance (UI) Wage Data. All employers who are covered under the Unemployment Insurance Act are required each quarter to report the earnings of each of their employees to the ODJFS. This data is maintained in a database organized by employer account code and employee Social Security Number. The data is used to verify the employers, quarters, and amounts of earnings, when employees file a claim for an unemployment insurance benefit.

UI Wage Records are the primary data source for entered employment, retention, earnings change, employment portion of credential rate and employment portion of younger youth retention rate. If employment is not found in UI Wage Records, local areas may use supplemental data. However, UI Wage records are the only data source for earnings change measures.

For the WIA performance measures, a customer is considered "employed" in a given quarter if they have any reported earnings for that quarter, regardless of the amount earned or hours worked.

The WIA measures use six quarters of wage record data: the 2nd and 3rd quarters prior to the quarter of registration, and the 1st, 2nd, and 3rd quarters following exit. There can be up to a two quarter delay in which the wages are earned and the quarter in which the data on those earnings are available. For example: earnings for the 1st post exit quarter for WIA registrants who exit during the July through September quarter of 2002 may not be available until the April through June quarter of 2003 . Clearly, UI wage record based measures are not useful for day-to-day program management.

Therefore we recommend creating an information system for day-to-day management, and to project performance on longer-term measures. In order to manage program goals, a case management system is needed to constantly keep in touch with participants; goals for vendors need to be clearly defined; and a strong network of vendors and partners is needed.

A percentage of workforce is not covered by Unemployment Insurance, which has led USDOL to permit - but not require - the use of supplemental data sources for reporting employment. Ohio has recently gained the ability to access federal and other states wage data that expands the amount of data that is available for performance measures.

USDOL has issued guidelines regarding the use of of supplemental data in USDOL TEGL 7-99. Please follow those guidelines if you are using supplemental data for performance measures.

Follow-Up

After a youth has exited the program the local area is required to provide at least 12 months of follow-up services. The types of services provided and the duration of the services must be determined based on the needs of the individual. Please see WIA regulation section 664.450 for more information on follow-up services.

Much of the data reported for youth performance measures are not part of the UI wage data reporting system. That data includes: exit date; high school diploma or GED; credentials; skill attainments; attendance in secondary school, post-secondary education, advanced training; military service; qualified apprenticeship and those not employed at registration. Some of these data elements can be reported up to almost a year after the exit date.

Example:If an individual exits on January 7, 2003 they could be excluded from the Earnings Change for Older Youth if they are in post-secondary education of advanced training in the 3rd quarter after exit. The third quarter after exit will not end until December of 2003.

Reporting data on these elements requires acquiring information from the participant and other sources. The best way to gather this information is through follow-up. If the information is not gathered and reported it is likely to have a negative effect on performance measures.

Closely tracking youth after exit can not only contribute to more successful long-term outcomes for youth, but also allow for the ongoing data collection that is required to measure performance for youth. Without tracking the youth's progress after exit, local operators would face the burden of paying for costly follow-up surveys to track performance outcomes.

Do make sure that you have good collateral contact information on all youth, so that you will be able to capture performance information after they leave the program.

Follow-up is not only essential for accurate data reporting but retaining and keeping youth engaged in the program and following up on their goals and success.

Credential

The federal definition of credential is flexible which allows the state and local areas to expand on the definition. Ohio has defined a credential as follows:

Credential - A nationally recognized degree or certificate or State or locally recognized credential. Credentials include, but are not limited to, a high school diploma, GED or other recognized equivalents, post-secondary degrees/certificates, recognized skill standards, and licensure or industry-recognized certificates. Local Boards should encourage certificates to recognize successful completion of the training services listed above that are designed to equip individuals to enter or re-enter employment, retain employment, or advance into better employment. Local areas should include all State Education Agency recognized credentials.

As stated above the local area has great flexibility in defining a credential. It includes locally determined credentials that help get or retain a job. Local areas have the opportunity to work with employers to develop meaningful credentials. It could include: soft skills training that are recognized by a third party that helps an individual get a job, a driver's license needed to get a job, passing proficiency tests, etc. Local areas should document what is acceptable as credentials in their area.

To get credit for the credential measure for Older Youth Credential Rate the credential must be coupled with another Older Youth positive outcome. All of the credential-based measures incorporate some lag time between exit and when the credential must be obtained. In the case of the Younger Youth and the diploma or equivalent rate, the credential must be obtained by the end of the 1st quarter after exit. In all of the remaining credential based measures, the credential must be obtained by the 3rd quarter after exit. However, credential attainment can occur anytime during participation but must occur and be reported by the end of the third quarter after exit.

More detailed information on credentials will be contained in a specific guidance under separate cover.

Specific Youth Performance Information

When looking at the formulas for Youth performance measures it is important to look at the parameters for each measure. Those parameters will tell who is included and/or excluded from the measure. The parameters will be in italics.

Older Youth (Age 19-21) Entered Employment Rate

Parameter: Of those who are not employed at registration and who are not enrolled in post-secondary education or advanced training in the first quarter after exit:

Numerator: Number of older youth who have entered employment by the end of the first quarter after exit

divided by

Denominator: Number of older youth who exit during the quarter

What is this measure?Clearly, this is an employment-related measure. It measures the number of youth who didn't have a job before services and were employed by the end of the 1st quarter after exit. The measure looks at what happens after a young person exits services. It recognizes the importance of education for older youth. Youth who exit but are in post-secondary education or training are not included in the measure.

This measure looks at the number of older youth who have entered employment by the end of the first quarter after exit divided by the number of older youth who exit during the quarter. The individual is considered employed if UI wage records for the quarter after exit show earnings greater than zero. UI Wage records will be the primary data source for tracking employment in the quarter after exit. When other data sources are used, individuals should be counted as employed if, in the calendar quarter after exit, they did any work at all as paid employees, worked in their own business, profession, or worked on their own farm.

Who is included in this measure?This measure includes all older youth except those who meet the exclusions below. Individuals who are both employed and in post-secondary education or advanced training in the first quarter after exit are included in this measure.

Who is excluded from this measure?Individuals who are employed at registration are excluded from this measure. Individuals who are not employed, but are in postsecondary education or advanced training in the first quarter after exit are also excluded from this measure.

When is this measured?This is an EXIT measure. It is measured in the first quarter after the exit quarter. For example, if a participant exits any time from July 1 - September 30, 2002, the first quarter after exit is October 1 - December 31, 2002.

What period of time does this measure cover?This measure looks at the employment status of the individual at registration and also looks at the their employment status, and whether they are in postsecondary education or advanced training, the quarter after they exit.

What else should I know about this measure?

  • This measure is similar to the one calculated for Adults and Dislocated Workers.
  • If tracking and reporting is not done for those who are in post-secondary education or advanced training in the quarter after exit they will not be excluded from this measure.
  • Those employed at registration are not gathered from the UI database but are inputted into the system at the local level.
  • Those who are employed in the 1st quarter after exit and in postsecondary education or advanced training will be included in this measure.

Example: 100 youth exit (April - May 02)

20 employed at registration (January - March 02)

40 entered employment in 1st quarter after exit (June - August 02)

15 entered Post-Secondary Education and not employed in 1st qtr after exit (June - August 02)

5 entered Advanced Training and not employed in 1st quarter after exit (June - August 02)

10 entered employment and Post-Secondary Education or Advanced Training in 1st quarter after exit (June - August 02)

(40 + 10) / [(100-20) - (15 + 5)] = 50 / 60 = 83% Entered Employment Rate

Older Youth (Age 19-21) Employment Retention Rate

Parameter: Of those who are employed in the first quarter after exit and who are not enrolled in post-secondary education or advanced training in the third quarter after exit:

Numerator: Number of older youth who are retained in employment in third quarter after exit

divided by

Denominator: Number of older youth who exited during the quarter

What is this measure?This is another employment-related measure. It is longer-term, examining what happens 6 months after leaving services. Like the entered employment rate, information for this measure will be taken from the UI wage records unless supplemental data sources are needed. It also recognizes the value of further education. Youth who are not working but are enrolled in post-secondary education or advanced training 6 months after exit are not included in the measure.

This measures the number of older youth who were employed in the first quarter after exit and are employed in third quarter after exit divided by the number of older youth who exited during the quarter. Employment in the first and third quarters following exit does not have to be with the same employer. The individual is considered employed if UI wage records for the quarter after exit show earnings greater than zero. UI wage records will be the primary data source for tracking employment in the third quarter after exit. When other data sources are used, individuals should be counted as employed if, in the calendar quarter after exit, they did any work at all as paid employees, worked in their own business, profession, or worked on their own farm.

Who is included in this measure?This measure includes all older youth who are employed in the first quarter following exit except those specifically excluded as described below.

Who is excluded from this measure?This measure excludes older youth who were employed in the first quarter and not employed in the third quarter following exit, but are in post-secondary education or advanced training in the third quarter following exit.

When is this measured?This is an EXIT measure. It is measured in the third quarter after the exit quarter. For example, if the participant exits any time from July 1 - September 30, 2002, the 3rd quarter after exit will be April 1 - June 30, 2003

What period of time does this measure cover?This measure looks at the employment status of the individual the quarter after they exit, and whether they are in postsecondary education or advanced training and their employment status the 3rd quarter after they exit.

What else should I know about this measure?

  • This measure is similar to the one calculated for Adults and Dislocated Workers.
  • If tracking and reporting is not done through the 3rd quarter after exit for those who are in post-secondary education or advanced training, they will not be excluded from this measure.
  • This measure is based on those employed in the 1st quarter after exit regardless of employment at registration.
  • Employment retention is not limited to the same employer.
  • Those who are employed in the 1st quarter after exit and the third quarter after exit and also in postsecondary education or advanced training will be included in this measure.

Example: 100 youth exit (April - May 02)

40 entered employment in 1st quarter after exit (June - August 02)

10 entered Post-Secondary Education and not employed in 3rd quarter after exit (Jan - Mar 03)

5 entered Advanced Training and not employed in 3rd quarter after exit (Jan - Mar 03)

10 entered employment and Post-Secondary Education or Advanced Training in 1st quarter after exit (June - August 02)

20 + 10 / [40 - (10 + 5)] = 30 / 35 = 86% Employment Retention Rate

Older Youth (Age 19-21) Average Earnings Change in Six Months

Parameter: Of those who are employed in the first quarter after exit and who are not enrolled in post-secondary education or advanced training in the third quarter after exit:

Numerator: Total post-program earnings (earnings in quarter 2 + quarter 3 after exit) minus pre-program earnings (earnings in quarter 2 + quarter 3 prior to registration)

divided by

Denominator: Number of older youth who exit during the quarter

What is this measure?This is another long-term measure, looking at a young person's earnings 6 months after exit from services compared to the individual's earnings 6 months before receiving services. It is also an employment-related measure. Like the previous two, it recognizes the value of further education. Information for this measure is drawn from UI wage records only. No supplemental data sources can be used.

This measures total post-program earnings (earnings in quarter 2 + quarter 3 after exit) minus pre-program earnings (earnings in quarter 2 + quarter 3 prior to registration) divided by the number of older youth who exit during the quarter. This measure includes all older youth who were employed in the first quarter after exit. The UI wage records will be the only data source for this measure.

Who is included in this measure?Individuals who are employed in UI covered employment in the quarter after the exit quarter.

Who is excluded from this measure?Individuals who are not employed in the first quarter after exit. Individuals whose employment status (pre- or post-program) cannot be confirmed by UI Wage Record cross match, or through other accepted data sources. Individuals whose entry date is so far back in time that accessing pre-registration wage data is unfeasible or unreasonable are also excluded from the measure.

When is this measured?This is an EXIT measure. When calculating this measure data is collected from the 2nd and 3rd Quarters after exit and the 2nd and 3rd Quarters prior to registration. The 2nd and 3rd Quarters prior to registration are used because trends show that many participants experience intermittent or "stop-gap" employment immediately prior to registration.

What period of time does this measure cover?This measure looks at the employment status of the individual the quarter after they exit, and whether they are in postsecondary education or advanced training the 3rd quarter after they exit. This measure is based upon the wages the 2nd and 3rd quarters prior to registration as well as the 2nd and 3rd quarters after exit.

What else should I know about this measure?

  • This measure is similar the one calculated for Adults and Dislocated Workers.
  • If tracking and reporting is not done through the 3rd quarter after exit for those who are in post-secondary education or advanced training they will not be excluded from this measure.
  • This measure is based on those employed in the 1st quarter after exit regardless of employment at registration.
  • Wages are not limited to the same employer.
  • Those who are employed in the 1st quarter after exit and the third quarter after exit and also in postsecondary education or advanced training will be included in this measure.
  • The only wages that are tracked for this measure are UI wages.
  • Youth with no or low pre-program earnings will be much more likely to experience significant earnings gain.

Example: 100 youth exit (April - May 02)

40 entered employment in 1st quarter after exit (June - August 02)

20 still employed in the 3rd quarter after exit (Jan - Mar 03)

10 entered Post-Secondary Education and not employed in 3rd quarter after exit (Jan - Mar 03)

5 entered Advanced Training and not employed in 3rd quarter after exit (Jan - Mar 03)

10 entered employment and Post-Secondary Education or Advanced Training in 1st quarter after exit (June - August 02)

Total of $30,000 Earnings in quarter 2nd (July - August 01) and 3rd quarter (April - June 01) before registration of 30 exited youth who were employed in the first quarter after exit

$50,000 Total Earnings in quarter 2nd (Oct - Dec 02) and 3rd quarter (Jan - Mar 03) after exit of 30 exited youth who were employed in the first quarter after exit

$50,000 - $30,000 / [40 - (10 + 5)] = $20,000 / 35 = $571 Average Earnings Change

Older Youth (Age 19-21) Credential Rate Measure

Parameter: None

Numerator: Number of older youth who are in employment, post-secondary education, or advanced training in the first quarter after exit and received a credential by the end of the third quarter after exit

divided by

Denominator: Number of older youth who exit during the quarter

What is this measure?This is a new type of measure. It looks at whether those youth who left services and were in employment or further education when they left have received a credential during services or within 6 months after services. Like the last two measures, this is a long-term measure. It recognizes and rewards attainment of credentials. It is important to note that "credentials" may be defined by the State, the local area, or both. In addition to academic credentials such as Associate's or Bachelor's degrees, this measure could include such things as obtaining a Commercial Driver's License or earning a Child Development Credential.

This measures the number of older youth who are in employment, post-secondary education, or advanced training in the first quarter after exit and received a credential by the end of the third quarter after exit divided by the number of older youth who exit during the quarter. A credential must be coupled with employment, entry into post-secondary education, or entry into advanced training. Credentials can be obtained while an individual is still participating in services.

Who is included in this measure?All older youth exiters will be included in this measure.

Who is excluded from this measure?There are no exclusions from this measure.

When is this measured?This is an EXIT measure that has two components. In the quarter after the exit quarter, entry into employment, post-secondary education, or advanced training is measured. In the third quarter after the exit quarter, attainment of a credential is measured.

What period of time does this measure cover?This measure looks at the time period for measuring credentials when the participant starts receiving services through the 3rd quarter after exit and for entry into employment, post-secondary education, or advanced training in the 1st quarter after exit.

What else should I know about this measure?

  • This measure is similar to the one calculated for Adults and Dislocated Workers.
  • If tracking and reporting is not done through the 3rd quarter after exit for those who are in post-secondary education or advanced training they will not be excluded from this measure.
  • The credential must be coupled with other Older Youth outcomes to be counted as a positive outcome.
  • All youth who exit are included in the measure
  • Credential can be received during the receipt of services or any time including through the 3rd quarter after exit
  • Credentials can be defined by the federal, state or local level depending on local need.

Example: 100 youth exit (April - May 02)

40 entered employment in 1st quarter after exit (June - August 02)

10 entered Post-Secondary Education in 1st quarter after exit (June - August 02)

5 entered Advanced Training in 1st quarter after exit (June - August 02)

50 had credentials before the end of the 3rd quarter after exit and another OY positive outcome in the 1st quarter after exit (Jan - March 03)

50 / 100 = 50% Credential Rate

Younger Youth (14 - 18) Skill Attainment Rate Measure

Parameter: Of all in-school youth, and any out-of-school youth assessed to be, in need of basic skills, work readiness skills, and/or occupational skills:

Numerator: Total number of basic skills goals attained by younger youth plus number of work readiness skills goals attained by younger youth plus number of occupational skills goals attained by younger youth

divided by

Denominator: Total number of basic skills goals plus the number of work readiness skills plus the number of occupational skills goals set

What is this measure?It is a "real time" measure, meaning that it looks at what is happening while youth are participating. It measures the attainment of basic academic, work readiness or occupational skills, comparing actual achievement to planned achievement.

Up to three skill attainment goals may set for each youth within a program year. The first skill attainment goal must be set on the registration date. Other goals may be set at this time or they may be set later. No more than 3 may be set, during the program year for performance purposes. Goals may involve any combination of the three types of goals. For example, a customer could have two basic skills goals plus one occupational skill goal. New skill attainment goals may be set on the anniversary date of the individual registration into WIA even if all previously set goals have not been attained.

Pre-assessment and post-assessment of skill goals are required. Use of standardized test procedures, such as standardized tests or performance-based assessments with standardized scoring methods are encouraged. Where such tests or assessments are not available assessment techniques must be objective, unbiased and conform to widely accepted, clearly defined criteria, be field tested for utility, consistency, and accuracy, and provide instructions for valid test administration. All data and methods used to determine achievement of skill attainment goals must be documented and are subject to audit.

Who is included in this measure?All in-school and out-of-school Younger Youth who have a basic skills goal, a work readiness skills goal, and/or an occupational skills goal and, during the reporting quarter, the goal was attained or one year has lapsed since the goal was set. A youth who is basic skills deficient must have at least one basic skills goal.

Who is excluded from this measure?Youth who do not have any skill attainment goals are excluded from this measure. Youth who have skill attainment goals set, but the one year anniversary of the date the goal was set is a future date are also excluded from this measure. There will always be more goals set than attained, but some of the "goals set" shouldn't be part of the denominator because the "12 months" hasn't expired.

When is this measured?Skill Attainment Rate is measured DURING program participation. Skill goals that are recorded as "Attained" during the reporting quarter are included in the numerator. Skill goals that were set and the one year anniversary of the date the goal was set falls within the reporting quarter, and the goal has not been recorded as attained are included in the denominator. There is flexibility in defining goals (Goal setting around what you want to do, what you want to have, what you want to be - within 30 days, 6 months, one year).

What period of time does this measure cover?This measure is reported by actual program year. Skill attainments are reported as they are achieved and do not have to wait until the 12 month period is reached. Failure to meet a goal should not be recorded until the goal attainment period has expired, in case the goal can be obtained.

What else should I know about this measure?

  • The local area has flexibility in setting a goal to be achieved. It should be something that can be achieved in a 12 month period. There is no standard such as a grade level a year mandated by federal or state policy. Special care should be taken with mentally disabled youth to make sure that goals are set that can be achieved within the year period.
  • First goal must be recorded as set on the registration date (even if not actually set until a short time later).
  • If youth exits before attaining a goal, it is considered a negative outcome. If the youth achieved a goal prior to exit, case management and follow-up may be used to gather these pre-exit attainments to record positive outcomes.
  • Three goals per year maximum.
  • Youth have one year to attain each goal (each goal is independent).
  • This measure counts skills not the number of youth. One youth can achieve several skill goals.
  • Local Areas might consider the denominator as the # of goals attained + # of goals not attained within one year + # of goals set for those who exit without attaining set goals
  • For Younger Youth who remain in school during the program, this is likely to be the only measure that will apply. Therefore it is essential that these youth be assessed and as appropriate, achievable goals are set for each year that they are in the program.
  • Target dates for accomplishing goals may not be more than a year from the date on which they are set, but they may be extended to accommodate periods of time with a planned gap of service as outlined by DOL in TEGL 7-99.
  • Although it may seem obvious, in order to be successful with this measure, there must be a local system in place to assess each youth, determine what are the appropriate goals, and a process to reassess each youth to determine when the goals have been achieved.
  • Do not exit the youth until you have verified that each of the set goals have been achieved, because once they are exited, you cannot go back and complete goals.

Example: 50 Skill goals attained

10 Skill goals not attained within one year

10 Skill goals set for those who exit without attaining set goals

50 / (50 + 10 + 10) = 50 / 70 = 71% YY Skill Attainment Rate

Younger Youth (14 - 18) Attainment Of Secondary School Diplomas And Their Recognized Equivalents Measure

Parameter: Of those who register without a diploma or equivalent:

Numerator: Number of younger youth who attained secondary school diploma or equivalent by the end of the first quarter after exit

divided by

Denominator: Number of younger youth who exit during the quarter (except those still in secondary school at exit)

What is this measure?This measure looks at whether those who participate in activities receive a high school diploma or equivalent. It only applies to those who do not already have this credential, and recognizes it as an important first step in employability. The measure is calculated a the end of the 1st quarter after exit, even if the diploma/equivalent was received during services. Note that in-school youth who leave services and are still in school, such as those who participate in a summer program and return to school, are excluded from the measure.

Number of younger youth who attained secondary school diploma or equivalent by the end of the first quarter after exit divided by the number of younger youth who exit during the quarter.

Who is included in this measure?Individuals who exit the WIA youth program and don't meet the exclusion listed below.

Who is excluded from this measure?Individuals who had a high school diploma or equivalent at registration and/or youth who remain in secondary school at exit are excluded from this measure.

When is this measured?This is an EXIT measure. It is measured in the first quarter after the exit quarter. For example, if a participant exits any time from July 1 - September 30, 2002, the first quarter after exit is October 1 - December 31, 2002.

What period of time does this measure cover?This looks at those individuals before they register to see if they have a high school diploma or its equivalent and attained their secondary school diploma or equivalent during program participation through the 1st quarter after exit to see if they have received these credentials.

What else should I know about this measure?

  • If the youth is still in school at exit they are excluded from this measure. The fact that they are in school must be reported for the individual to be excluded.
  • Those who have HS diploma or equivalent at registration are excluded from this measure.
  • Attainment of the diploma can occur anytime during participation.
  • Must attain diploma or equivalent by the end of the end of the 1st quarter after exit.

Example: 350 Those who registered without a high school diploma or equivalent (January - March 02)

100 youth exit (April - May 02)

45 Of those without HS diploma at registration who equivalent who received HS diploma or equivalent before the end of the 1st quarter after exit (June - Aug 02)

10 Youth who exit who are still in secondary school (April - May 02)

45 / (100 - 10) = 45 / 90 = 50% Diploma or Equivalent Attainment Rate

Younger Youth (14 - 18) Retention Rate Measure

Parameter: None

Numerator: Number of younger youth found in one of the following categories in the third quarter following exit: post secondary education, advanced training, employment, military service, qualified apprenticeships

divided by

Denominator: Number of younger youth who exit during the quarter (except those still in secondary school at exit).

What is this measure?The third and last measure for younger youth is the retention rate. As shown here, it recognizes a variety of positive outcomes for youth. It is a longer-term measure which looks at placement and retention in postsecondary education, advanced training, military service, unsubsidized employment, or qualified apprenticeships. The measure is determined by dividing the number of younger youth who, in the third quarter after the exit quarter, are in post secondary education, advanced training, unsubsidized employment, military service or qualified apprenticeships by the number of younger youth who exit during the quarter.

Who is included in this measure?Individuals who exit the WIA youth program and did not remain in secondary school at exit.

Who is excluded from this measure?Individuals who remain in secondary school at the time of exit are excluded from this measure.

When is this measured?This is an EXIT measure. It is measured in the third quarter after the exit quarter. For example, if the participant exits any time from July 1 - September 30, 2002, the 3rd quarter after exit will be April 1 - June 30, 2003.

What period of time does this measure cover?It is measured in the third quarter after the exit quarter. It also looks at those who are still in secondary school at exit.

What else should I know about this measure?

  • If the youth is still in school at exit they are excluded from this measure. They must be reported as in-school to be excluded.
  • All other younger youth are included in this measure.
  • Youth have to be in one of the 5 positive outcomes in the 3rd quarter after exit: employment, post-secondary education, advanced training, military or qualified apprenticeship.
  • Local areas must implement programs for younger youth that are likely to lead to one of the positive outcomes for this measure. Aside from employment, these outcomes are generally obtained after high school. This measure is directed at out-school-youth and older in-school youth who can be expected to continue their education or enter employment or military after high school.
  • Local areas that serve exclusively in-school youth in the 14-18 category who are not scheduled to graduate from high school while in the program should be cautious. Such designs will result in no students planned to be in the denominator for this measure. In such a situation, the only students actually entering the denominator may be those who dropped out of school and then dropped out of the program. This will result in failure of the measure.

Example: 100 youth exit (April - May 02)

75 Are found in one of the 5 federally recognized positive outcomes in the

3rd after exit (Jan03 - March 03)

10 Youth who exit who are still in secondary school (April - May 02)

75 / (100 - 10) = 75 / 90 = 83% YY Retention Rate

IV.Technical Assistance

There are a number of strategies local areas can undertake to better manage the performance of their youth programs. WIA has seven youth performance measures, six of which measure outcomes after program exit. The performance measures are accountability tools set by the federal government and not management tools. The measures may not "tell your story" or adequately reflect the health of your organization and programs. The best strategy for success is to maintain and monitor interim measures to assess progress. Five key interim measures to monitor are intake, participation in program elements, skill attainments, exit data, and follow-up services.

  • ·                     Intake:Comprehensive Individual Service Strategies (ISS) should be developed that include both short-term and long-term goals. ISS goals and strategies should be updated as short-term goals are achieved or the youth's needs change. Skill attainment goals should be established for all in-school and all out-of-school youth assessed as needing basic skills, work readiness skills and/or occupational skills training.
  • ·                     Participation In Program Elements:Active participant engagement in the ten youth program elements leads to positive outcomes. In addition, specific program elements have direct impacts on certain WIA outcome measures. Thorough assessment of each individual's needs and determining which program elements impact which performance measures and monitoring to ensure participation in these activities will increase successful program outcomes. Tracking youth participation will help manage the process of keeping participants on track for completion of short-term skill attainment goals and preparation to move into long-term employment, education, or training.
  • ·                     Skill Attainment:Closely tracking skill attainment is another interim way to monitor performance. Skill attainments, particularly in academic areas, positively correlate with both diploma acquisition and exit placement outcomes.
  • ·                     Exit Data:A key piece of data to monitor is the number of participants exiting the program. Because exit is the triggering event for six of the seven youth performance measures, it is extremely important to understand the definition of exit and to monitor youth participation and promptly capture "soft exits," youth who have not participated in any activity for 90 days, as well as hard exits. Active case management, identification of multiple family/friend contact information, and frequent contact with youth participants can both reduce the number of "soft exits" and increase access to these youth during the 12-month follow-up period. Programs should strive to insure that participants do not exit the program until they complete short-term skill attainment goals and are prepared to move into long-term employment, education, or training.
  • ·                     Follow-Up Services:Closely tracking follow-up services can not only contribute to more successful long-term outcomes for youth, but also allow for the ongoing data collection that is required to measure performance for youth. Without tracking the required follow-up services, local operators would face the burden of paying for costly follow-up surveys to track performance outcomes.

Local areas might also consider these suggestions when developing service strategies for the out-of-school youth population. A major challenge in serving out-of school youth is not only identifying them, but recruiting, retaining and keeping them engaged in the program. Training programs have not historically been designed to meet the needs of these youth. These programs often do not offer age-appropriate or other activities that interest youth, "get them in the door," and keep them enrolled in the program, until completion. Local program operators must focus on strategies to provide for employment, as well as addressing the need for basic skills, occupational skills, and work readiness skills. A three-pronged service strategy may be useful in serving this population:

  • Help them find gainful employment (full-time or part-time), and provide available support services, such as childcare. Also, help the youth understand that securing employment opportunities and increasing one's career potential is directly related to completing education and/or skills training, and attainment of educational/employment credentials.
  • While employed, provide the education and skills training needed by the youth to attain education/employment credentials, participate in post-secondary education, or become gainfully employed in a career with advancement opportunities.
  • Focus on retention of those out-of-school youth engaged in the required education/skills training activities until program completion.
  • Connecting youth, including those not in schools, to the One-Stop system, which may include: developing One-Stop Career Centers specifically designed to engage out-of-school youth; establishing convenient and extended hours for youth (e.g., youth who have jobs); conducting out-of-school youth-only forums and focus groups; and "informal presentations" by One-Stop staff in neighborhood places frequented by the out-of-school youth.

Examples of other strategies that focus on both in-school and out-of-school youth include: marketing youth-focused outreach materials; developing youth-based informational websites; One-Stop staff presentations to youth and their families in traditional and non-traditional places, such as convenience stores and laundromats; and training One-Stop front-line staff on the development of and elements required in the design of training programs for out-of-school youth.

Linkages to Partner Services. To encourage integration of services across programs (WIA Title I funded and non-WIA Title I funded) and recognize shared contributions toward outcomes, the following strategy for tracking and reporting on the core measures across programs can be used. WIA Title I funded youth programs can count participants who receive services provided by non-WIA Title I funded school-to-work (i.e., schools) and one-stop partner programs in the WIA core measures as long as the individual has been registered for WIA Title I youth services and:

a)is concurrently receiving WIA Title I funded youth services while receiving partner services;

b)is scheduled to receive WIA Title I funded youth services at a future date while receiving partner services or upon exit of the partner services; or

c)moves to partner services, and can be tracked while receiving and upon exit of partner services.

One approach for maximizing resources could include identifying service providers who already offer services that meet one or more of the program elements and are accessible to all eligible youth. In a competitive selection process, those service providers could be assigned more weight or scored higher if they agree to match WIA-funded services with some additional non-WIA funded services that address the ten program elements, as long as it is clearly stated in the Request for Proposal (RFP). Examples of the types of existing resources that can be tapped to complement youth programs under WIA include the local education system, public housing agencies, and programs such as Temporary Aid for Needy Families (TANF) and Welfare-to-Work.

Because the measures reward long-term attachment, multiple activities for one individual, and collaboration, goals may not be able to be passed on in a straightforward manner. Also, because official data collection will involve the UI wage records, which may not be readily available to local areas, local Youth Councils may have to set goals for vendors in a different way than before.

Local areas need to acknowledge that the data collected to determine goal attainment for the program as a whole will not be available on a timely basis, and is not sufficient to manage programs on a day-to-day basis. Therefore, an information system is needed for day-to-day management, and to project performance on longer-term measures. In order to meet goals, a case management system is needed to constantly keep in touch with participants; goals for vendors need to be clearly defined; and a strong network of vendors and partners is needed.

While it is a good idea to incorporate the WIA measures in the RFP process, there is some danger in this approach due to the long-term nature of the WIA outcome measure and delays associated with using the Unemployment Insurance wage records - the data source for many of the measures. There is a significant delay in the information available on these measures. Therefore, local areas may opt to consider the following two strategies to counteract these timeliness issues. First, local areas could use shorter term or interim outcome measures and process measures in conjunction with the WIA outcome measures to receive more timely feedback on performance of their service providers. Second, local areas may want to consider contracts longer than one year in order to allow more time for results against the WIA performance measures. A combination of these two approaches is also possible.

Setting Appropriate Goal(s). Within the skill attainment rate, three types of skills are being assessed within the Skill Attainment Measure: basic skills, work readiness skills, and/or occupational skills. All younger youth who are determined basic skills deficient (defined as an individual who has English reading, writing, or computing skills at or below the 8th grade level on a generally accepted standardized test or a comparable score on a criterion-referenced test) must have a basic skills goal that will be held accountable to in the skill attainment rate. If the participant is not basic skills deficient and therefore does not have a basic skills goal, the individual must have a work readiness and/or an occupational skills goal if they are an in-school youth. If the participant is an out of school youth (not in need of basic skills), it is a local option whether or not a work readiness skills goal and/or an occupational skills goal is necessary. In the example of a younger youth who only participates in the summer activity, exits, and returns to secondary school following the summer activity, if that participant is not deemed to be basic skills deficient, then a work readiness skills goal would be most appropriate for that youth. This work readiness skills goal could be tied to their summer youth experience in order to make it feasible to attain such a goal. If the participant is basic skills deficient, they must have a basic skills goal and it would be more appropriate to continue serving such a participant beyond their summer experience.

Enhance Program Quality. Local Boards, Youth Councils, and local programs should utilize quality practices as they plan and design a comprehensive and effective year-round youth services strategy. Principles that define quality programming for youth include:

  • Designing program activities to reflect program goals;
  • Ensuring the ongoing participation of caring adults;
  • Continuously improving the program based on data collection and analysis; and
  • Guaranteeing long-term follow-up to all youth participants.

Enhance Assessment Strategies. The objective assessment is a process that identifies service needs, academic levels, goals, interests, skill levels, abilities, aptitudes, and supportive service needs, as well as barriers and strengths. It includes a review of basic and occupational skills, prior work experience, employability potential, and developmental needs.

To determine whether a youth has met the skill attainment goals requires a pre-assessment and post-assessment of skill level. There are a wide range of tools that can be used to measure skill attainment goals for youth. Local assessment strategies should include some type of standardized assessment procedure such as a written test or a performance-based assessment with a standardized scoring method. See TEGL 7-99, which provides guidance on the WIA performance measures, for examples of the types of tests that may be used.

In some cases, a standardized test or a performance-based assessment may not be available (such as for work-readiness skills). The assessment methods used must be objective, unbiased and conform to widely accepted, clearly defined criteria, be field tested for validity, consistency, and accuracy, and provide for the training/preparation of all raters/scorers. Progress toward skill attainment goals can be tracked through case management or follow-up services if the goal attainment occurs during the exit quarter. While WIA allows flexibility, assessment strategies should include methods for documentation. Since participant records are subject to audit, States and local programs will want to ensure that staff receive training on how to document individual skill achievement and goal attainment.

Improve Recruitment Efforts, Enhance Retention and Completion of Individual Service Strategies. The youth development emphasis in WIA calls for effective strategies to recruit youth and then to engage and retain participants until they receive all needed services to successfully transition to adulthood and careers. Experience has shown that enrolling youth in occupational skills training and retaining them until completion of the program leads to better results. There are a number of steps that local programs can take to effectively recruit youth and then engage and retain youth in services.

First, it is important to make a positive connection with a young person from initial recruitment. This includes engaging youth who can talk positively about programs as peer advocates and recruiters; collaborating with community and faith-based organizations that already work with disadvantaged youth, especially out-of-school youth; and offering staff and youth incentives for recruiting new participants. Outreach and recruitment strategies can also tap into youth culture, such as public service announcements on local radio stations that appeal to young people.

A youth development approach recognizes that the young people who participate in WIA youth services will have diverse backgrounds and experiences and different types of developmental needs. It is important to recognize that most of these young people live in communities marked by poverty, violence, and illness. Not all youth mature at the same rate. Starting with their first visit to a program and the orientation process, youth want to understand how they "fit in." Some may want to immediately start skills training or a GED class; others might just be looking for somewhere to hang out for a few hours a week. In both cases, youth are "testing" how much they can trust the staff and the program. Local programs should be prepared to accommodate the wide variety of needs that youth bring during the early stages of their emerging relationship with the staff and program. Just like a new friendship, as the trust levels grow, so can the expectations that are placed on the youth.

The most effective youth programs promote a sense of membership and affiliation and a safe alternative to the streets. This includes strategies such as providing participants with membership cards, t-shirts, and opportunities to participate in planning activities, all of which contribute to retention. Programs that are co-located or connected to community centers that offer a range of recreational and cultural activities may be more successful in attracting and engaging young people, especially out-of-school youth. An effective individual service plan can play a critical role in ensuring that a young person stays engaged and completes the program. Adult mentorship, one of the ten required program elements, is key to the engagement and retention of youth during their enrollment. Other strategies that help programs to retain youth and maintain contact with them as part of follow-up are presented in the next section.

Expand and Further Develop Follow-Up Services. While local areas may conduct a broad range of follow-up activities under WIA (see TEGL 3-99), many face the challenge of developing and implementing a structure and/or system for follow-up. Indicators of a systematic approach to follow-up may include assigning follow-up activities to specific staff and developing a computer generated tickler file or some other type of system to ensure regular check-in with youth. In cases where partner organizations are conducting follow-up activities, local areas need to have a structure in place to monitor these activities. A systematic approach also entails using follow-up data and information to assess program effectiveness and improve program quality.

As local areas develop strategies for follow-up, they may find the operating principles listed below useful. These principles are drawn from "best practices" in the field of youth development.

(1)Develop a close mentoring relationship before and after placement. Structure staff assignments, schedules, and activities in a manner that will facilitate the establishment of a strong personal bond between the young person and the staff member providing follow-up services before the youth completes/leaves the program.

(2)Develop a systematic approach for maintaining contact and interaction with the young person during follow-up services. Since job loss or other set backs generally occur early in the post-program time period, it is important to provide intensive support and mentoring during the first part of the follow-up period. One successful model entails contacting the youth by phone:

  • each day before or after work/school/training for the first 5 days of placement in a job or advanced education/training. It is important to talk over everything that happened during work/school/training.
  • once a week for the next six months of employment/school/training.
  • once a month after the first six months of employment/school/training unless a personal crisis requires intensive contact.

(3)Provide engaging follow-up activities to help keep young people interested and connected. The activities need to be relevant which have to support the individuals goals (not just WIA) This could include: evening and weekend social, recreational, and cultural activities for informal support; meeting for a business lunch during the workday; skills upgrading classes and workshops in the evening (e.g., computer skills, GED); peer tutoring and mentoring; or support group meetings.

(4)Meet physical and emotional as well as vocational needs. It is important to maintain a network of services that support the whole person and help youth access those services. This network should include: medical services, housing, transportation, child care, and workplace clothing supplements.

(5)When the youth is employed, maintain a non-intrusive contact with employers. Follow-up staff should visit the job site as soon after the youth starts a job as possible and meet the employer and/or supervisor. During this visit, the follow-up staff should describe his/her role to the employer and provide a phone number. The follow-up staff should contact the employer again at the end of four to six weeks of employment for an update and periodically thereafter as needed.

(6)Incorporate the Youth voice. The youth need to give their input to insure buy-in and relevance to their needs.

Follow-up services can not only contribute to more successful long-term outcomes, but also allow for the ongoing data collection that is required to measure performance for youth under WIA. Without required follow-up services, local operators would face the burden of paying for costly follow-up surveys to track performance outcomes.

Enhancing Youth Connections to One-Stop Systems. WIA requires each local area to establish at least one comprehensive One-Stop center, specifies a wide range of required partners, and specifies the types of services that must be available. Youth programs funded under WIA are required partners in One-Stop systems. Many existing One-Stop systems were established before WIA and most of these One-Stop systems did not have outreach efforts to attract youth, pursue youth program partnerships, or provide youth services because the required core partners for that initiative primarily focused on adult customers. However, since youth programs are required partners under WIA, States should ensure that local One-Stop systems look for ways to better incorporate youth programs, develop a broader array of youth services, and reach out to a wider range of youth program partners than was typical in the past.

There are a number of ways in which local areas can enhance connections to youth and access to One-Stop systems. These include the following:

(1)Supporting Youth Through Organizational Design. One promising approach is co-locating youth program staff at the One-Stop center or designating staff to coordinate outreach and services for youth at One-Stop centers. Even when staff are not co-located, cross-training of youth program and One-Stop staff can be extremely helpful. At a minimum, youth program staff should participate in tours of One-Stop centers and One-Stop staff should visit youth programs.

(2)Marketing and Outreach Efforts to Recruit Youth. There are many strategies to encourage youth to use the One-Stop system. These include: establishing linkages with schools, community-based organizations, and faith-based organizations; conducting outreach efforts that target out-of-school youth; conducting special tours of the One-Stop centers for youth; and establishing linkages with School-to-Work systems.

(3)Customizing One-Stop Center Facilities and Self-Service Resources for Youth. Some One-Stop centers maintain information about youth activities and services, have separate resource rooms and/or have resources customized for youth customers. In addition, there are ways to help make facilities more "youth-friendly" and inviting to young people. Some local areas have established separate satellite centers targeted for youth or innovative satellite centers at places where youth spend time, such as secondary schools, libraries, parks and recreational facilities, and shopping centers.

(4)Linking to Existing One-Stop Services. In order to provide the ten required youth program elements under the WIA, local areas may benefit from some of the activities and services that are already available through other funding sources at One-Stop centers. For example, some One-Stops provide vocational and GED training. In terms of supportive services, One-Stops may have referral databases for childcare providers or services such as family planning.

Please contact the Bureau of Workforce Services (614) 644-7078 or E-Mail WIAQNA@odjfs.state.oh.us with any questions.

V.Reference

Workforce Investment Act Section 122 (c) (5)

20 CFR 663.530

Rescissions: None

Distribution: WIA Guidance Letter Expiration: Continuing

Attachment A

Click here to view WIA'S Performance Measures For Youth at a Glance. (Not archived)

Attachment B

Click here to view Key Definitions for WIA Youth Performance Measures. (Not archived)

Youth Flowchart/Pictures of Performance

Pictures of Performance (WIA Youth Measures). (Not archived)