In order to strengthen accountability across programs, CCMEP establishes a single lead agency responsible for meeting common performance measures. Performance measures are focused on improving employment and education outcomes for low-income youth ages 16-to-24 served by the program. CCMEP measures are based on those established by the U.S. Department of Labor for workforce programs funded by the Workforce Innovation and Opportunity Act (WIOA).

The Ohio Department of Job and Family Services (ODJFS) has been in discussions with our stakeholders throughout the development of CCMEP, including representatives from workforce development boards and county job and family services agencies. This memo provides clarification on a number of issues raised in those discussions about how ODJFS will manage performance and accountability in the program. The memo includes: A) a description of primary performance measures and consequences for failure to meet performance standards; and B) information about Temporary Assistance for Needy Families (TANF) work participation requirements for individuals receiving assistance through Ohio Works First (OWF) who are required to participate in CCMEP.

In general, the state's approach to performance management will be familiar to local area workforce development agencies administering WIOA-funded programs. Yet, some county department of job and family services (CDJFS) agencies may be unfamiliar with how WIOA performance metrics work. Likewise, many workforce agencies may be unfamiliar with TANF work participation requirements. Implementation of CCMEP will require local administrators to become familiar with aspects of both TANF and WIOA performance management.

Overview

First, as discussed in further detail in Part A of this memo, CCMEP performance standards will be established beginning program year 2017 to measure outcomes of the first cohort of CCMEP participants that leave the program (i.e., exit) in program year 2016 (See CCMEP Schedule of Performance table below). This memo provides additional information about the CCMEP primary performance measures, participants included in the measures, performance periods, negotiation of standards, determining success or failure and consequences for failing to meet performance standards.

Second, with regard to TANF work participation requirements applicable to OWF recipients, beginning July 1, 2016 county agencies will be monitored based on their work required population age 25 and over not participating in CCMEP. A new report will be created to establish a county rate with CCMEP participants excluded. Part B of this memo provides further explanation and detail about how the state will monitor TANF work participation in CCMEP and the risk associated with any financial penalties.

Part A: CCMEP Performance Management

1) What are the CCMEP primary performance measures?

CCMEP accountability will be tied to outcome measures tracked at the lead agency (county) level. There are seven primary performance measures for each lead agency outlined in paragraph (A) of rule 5101:14-1-07 of the Ohio Administrative Code (OAC) and detailed as follows:

(1) Employment 2nd quarter after exit: The percentage of program participants that are in unsubsidized employment during the second quarter after exit.

(2) Employment 4th quarter after exit: The percentage of program participants that are in unsubsidized employment during the fourth quarter after exit.

(3) Education, training or employment 2nd quarter after exit: The percentage of program participants that are in unsubsidized employment, education, or occupational skills training during the second quarter after exit.

(4) Education, training or employment 4th quarter after exit: The percentage of program participants that are in unsubsidized employment, education, or occupational skills training during the fourth quarter after exit.

(5) Credential attainment rate: The percentage of program participants that were in an education or training program while enrolled in CCMEP who obtain a recognized post-secondary credential, or a secondary school diploma or its recognized equivalent during participation in or within one year after exit from CCMEP. CCMEP program participants who obtain a secondary school diploma or its recognized equivalent shall be included in the percentage counted as meeting the criterion if such participants, in addition to obtaining such diploma or its recognized equivalent, have obtained or retained employment or are in an education or training program leading to a recognized post-secondary credential within one year after exit from CCMEP.

(6) Median earnings 2nd quarter after exit: The median earnings of program participants who are in unsubsidized employment during the second quarter after exit.

(7) Effectiveness in serving employers (i.e., job retention): The percentage of program participants who were in unsubsidized employment during the second quarter after exit who during the fourth quarter after exit are employed with the same employer.

The state also has established intermediate measures that are employment- and credential-focused to capture the impact of services while participants are in the program. Unlike primary performance measures, which are based on outcomes after a participant exits, intermediate measures capture the impact of services while participants are in the program. Intermediate measures are described in further detail in CCMEP Procedure Letter #1.

2) Which participants are included in the measures?

Each lead agency will have one set of common performance measures based on all individuals in CCMEP regardless of whether their services are paid for by WIOA or TANF. Lead agency performance will include each program participant that signs an individual opportunity plan (IOP) on or after July 1, 2016 as well as WIOA youth transitioned to CCMEP.

Measuring CCMEP performance is separate and distinct from measuring WIOA performance. In WIOA, local workforce area performance is based on the number of individuals that received a WIOA-funded service; in CCMEP the lead agency will have flexibility to determine which funding source is used (TANF and/or WIOA) to provide services to CCMEP program participants. Only those participants who receive a WIOA-funded service will be part of the DOL -WIOA youth performance measures tracked at the local workforce area level.

3) When does performance start?

Performance periods are established by program year and span a period of time from July through June. CCMEP will begin July 1, 2016 which is program year 2016. All of the CCMEP primary performance measures will be evaluated after a participant has exited the CCMEP program. Therefore, the first set of CCMEP performance standards will be for program year 2017. (See Appendix - Table 3. Performance Measure Cohorts for a detailed explanation of when performance will be measured).

CCMEP performance reports will be issued quarterly (i.e., July-September; October-December; January-March; April-June) and a final performance report will be issued annually. These reports will be shared with the local workforce development boards and the county CCMEP lead agency. Reports will include a scorecard with outcomes from the seven (7) primary performance measures as well as intermediate measures. (Appendix - Table 2. Example Performance Report provides a mock-up for illustrative purposes).

4) When will CCMEP participant outcomes for the primary performance measures be evaluated? (See Appendix - Table 3. Performance Measure Cohorts)

All seven of the primary performance measures will be evaluated after the participant has exited the CCMEP program. Different measures will be evaluated at different measurement periods based on the measurement definition (e.g., measure 1 looks at employment in the second quarter after exit and measure 2 looks at employment in the fourth quarter after exit). Since time is needed to wait for data to become available (i.e., all primary performance measures use wage record data which requires a two quarter delay to become available). For these reasons, measures 1, 3, and 6 will be evaluated four (4) quarters after the participant exits; and measures 2, 4, 5, and 7 will be evaluated six (6) quarters after the participant exits.

• For example, measure 1 seeks to determine if a program participant is in unsubsidized employment during the "second quarter" after exit. For a participant that exited in August 2016, their "exit quarter" would be July-September 2016 and the "second quarter" after their exit quarter would be January-March 2017. Wage record data will be used to determine if this participant was employed during the second quarter after their exit. Since there is a two (2) quarter delay to receive wage record data, the data needed to evaluate whether or not this participant was employed during the second quarter after their exit will not be available until July-September 2017. Measures 3 and 6 also seek to determine information during the "second quarter" after exit and measures 3 and 6 will also use wage record data, therefore the situation in this example applies to these measures as well. Hence, the statement in the above paragraph, "measures 1, 3, and 6 will be evaluated four (4) quarters after the participant exits."
• Measures 2, 4, 5, and 7 are similar in that wage record data will be used, but these measures seek to evaluate information during the "fourth quarter" after exit. For example, measure 2 seeks to determine if a program participant is in unsubsidized employment during the "fourth quarter" after exit. For a participant that exited in August 2016, their "exit quarter" would be July-September 2016 and the "fourth quarter" after exit would be July-September 2017. Again, there is a two (2) quarter delay to receive wage record data, therefore the data needed to evaluate whether or not this participant was employed during the fourth quarter after their exit will not be available until January-March 2018. Hence the statement above, "measures 2, 4, 5, and 7 will be evaluated six (6) quarters after the participant exits."
• Note that, in the examples above, the participant exited the program in PY 2016 but the primary performance measures were evaluated in PY 2017. CCMEP Performance Success or Failure will be determined in the period that a performance measure is evaluated, not when a participant was enrolled or exited. Therefore, in the examples above the participant's outcome will be part of the lead agency's PY 2017 performance report and held against their PY 2017 performance standard.

5) What are performance standards and how will standards be negotiated?

A standard is defined as a level or target to meet; this standard will be in the form of a percentage rate for all primary performance measures except number 6 (Median earnings 2nd quarter after exit), which will be in the form of a dollar amount.

Each lead agency will have an opportunity to negotiate its CCMEP performance standards with ODJFS. The local workforce development board and county lead agencies must work together when negotiating the CCMEP measures in order to ensure that the CCMEP standard does not jeopardize the local area's ability to meet the WIOA youth negotiated measures. CCMEP performance standards for PY 2017 primary measures will be negotiated in June 2016 prior to the program start date. (Appendix - Table 1. Example Performance Standards provides a mock-up for illustrative purposes).

Consistent with DOL guidance, note that for PY 2016 and PY 2017, primary performance measure number 6 (Median earnings 2nd quarter after exit) and primary performance measure number 7 (Effectiveness in serving employers) will not count towards the performance success or failure calculation. Data will be collected in order to establish a baseline to use in establishing a standard for PY 2018 and beyond. Additionally, lead agencies will be held accountable for an individual performance measure once a full year of outcome data are available for evaluation.

Hence, for PY 2017, only primary performance measure number 1 (Employment 2nd quarter after exit) and primary performance measure number 3 (Education, training or employment 2nd quarter after exit) will count towards the lead agency's performance success or failure evaluation. For PY 2018 and beyond, all primary performance measures will count towards the performance success or failure evaluation.

(See Appendix - Table 2. Example Performance Report for additional details).

6) What constitutes CCMEP performance success or failure?

In the interest of consistency, CCMEP performance success or failure will mirror Ohio's methodology for calculating the success or failure in meeting WIOA's primary performance measures. Lead agency performance reports will be made available by ODJFS quarterly and will include the success or failure results. Although performance reports will be made available on a quarterly basis, only the result for the annual report will count in determining lead agency success or failure (See Appendix - Table 2. Example Performance Report for additional detail).

Actual CCMEP performance will be evaluated for each individual measure relative to the lead agency's negotiated performance standard, using the Exceeds, Meets, and Fails criteria defined below.

Achieving a 'Meets' or 'Exceeds' on all individual measures constitutes lead agency success. Obtaining a 'Fails' on any individual measure constitutes lead agency failure.

• Exceeds - when the actual Lead Agency performance achieved against an individual performance measure is in excess of 105 percent of the performance standard for the measure
• Meets - when the actual Lead Agency performance achieved against an individual performance measure falls in the range of 90 to 105 percent of the performance standard for the measure
• Fails - when the actual Lead Agency performance achieved against an individual performance measure is less than 90 percent of the performance standard

Example: evaluating Employment 2nd Quarter after Exit with a standard of 80.0 percent:

An example of "Exceeds" for this measure is when the Lead Agency outcome is greater than 105 percent of the standard (i.e., greater than 84.0 percent); satisfying the requirement for 'Exceeds'.

An example of "Meets" for this measure is when the Lead Agency outcome is between 90 and 105 percent of the standard (i.e., between 72.0 and 84.0 percent); satisfying the requirement for 'Meets'.

An example of "Fails" for this measure is when the Lead Agency outcome is less than 90 percent of the standard (i.e., below 72.0 percent); satisfying the condition for 'Fails'.

7) What are the consequences of failing performance measures? `

• PY 2016 (July 1, 2016 - June 30, 2017): For the 1st year of CCMEP there will be no CCMEP program exiters to measure (see CCMEP Schedule of Performance table). Lead agencies will be expected to collect and report data relevant to intermediate measures. Lead agencies will receive quarterly reports and an annual calculated performance outcome for intermediate measures only.
• PY 2017 (July 1, 2017 - June 30, 2018): This will be the first year with primary performance measure data available for the exiters of the CCMEP program. During this second year of the CCMEP program, lead agencies will be expected to collect and report data relevant to all performance measures. Lead agencies will receive quarterly reports and an annual calculated performance outcome for primary performance measures, as well as outcomes for intermediate measures. In the event of a performance failure, the lead agency will be required to submit and be held to a corrective action plan.
• PY 2018 (July 1, 2018 - June 30, 2019): For the 3rd year of the CCMEP program, lead agencies will continue to be expected to collect and report data relevant to all performance measures. Lead agencies will receive quarterly reports and an annual calculated performance outcome. In the event of a performance failure, the lead agency will be required to submit and be held to a corrective action plan.
• PY 2019 (July 1, 2019 - June 30, 2020) and beyond: For the 4th year of the CCMEP program and beyond, lead agencies will continue to be expected to collect and report data relevant to all performance measures. Lead agencies will receive quarterly reports and an annual calculated performance outcome. In the event of a performance failure, the lead agency will be subject to corrective action, which may include financial penalty.

Part B: CCMEP and Ohio Works First Work Participation

To support the success of CCMEP, ODJFS has asked the U.S. Department of Health and Human Services (HHS) for a waiver from certain federal rules governing what counts toward work participation in the Ohio Works First (OWF) TANF program. 1 The state has not requested a waiver of work or work participation. OWF recipients are still required to engage in work-related activities for the federally required minimum number of hours and those who fail to comply are subject to sanctioning. OWF work-required individuals participating in CCMEP must adhere to the minimum twenty hour requirement as set forth in rule 5101:14-1-05 (CCMEP rule) of the OAC, or the hour requirement set forth in rule 5101:1-3-01 (OWF rule) of the OAC; whichever is higher. HHS continues to review the state's application and has not yet approved or denied the request.

Regardless of the status of the waiver, all individuals in CCMEP will be treated the same and receive a common client experience that adopts an individualized approach to case management focusing on improved employment and education outcomes for low-income youth served by the program. All participants, including those who are work-required and receiving assistance through OWF, will be assigned to activities based on a comprehensive assessment of their employment and training needs and barriers to employment. Caseworkers will jointly work with clients to develop an individual opportunity plan and have flexibility to assign only those activities that improve employment outcomes for participants.

To support this approach, beginning July 1, 2016 county agencies will be monitored based on their OWF work required population (age 25 and over) who are not participating in CCMEP. A new report will be created to establish a county rate with CCMEP participants excluded to continue to provide feedback and monitor county OWF work participation (i.e., 16-to-24 year olds will not be included in the ADCR county dashboard).

While CCMEP participants ages 16-to-24 will be excluded from county OWF work-participation reports, CCMEP participation hours will still be counted and reported by the state to HHS. Activities that work required individuals are engaged in through their individual opportunity plans must be verified and tracked in CRIS-E (including good cause hours, etc.) to ensure integrity in federal reporting and that proper notices are issued from CRIS-E in the event that a sanction of OWF benefits is proposed.

Will counties incur financial penalties for not meeting work participation?

Ohio will continue to be held by HHS to the requirement to meet the 50 percent all-family work participation rate and the 90 percent two-parent work participation rate for all work required families (assistance groups) receiving OWF. This includes assistance groups enrolled in CCMEP. State law provides that, should ODJFS incur any federal financial penalties for failure to meet federal work participation rate requirements, causal counties may be held financially responsible. This policy only applies in the event the state should be assessed a federal financial penalty.

With regard to any penalties associated with not meeting work participation, a causal county can be identified and risk a penalty if their overall caseload does not meet the 50 percent all-family rate or the 90 percent two-parent work participation rate. This can be determined as follows: (See Appendix - Table 4. Determination of Causal County for additional detail)

1)If OWF work required participants age 25 years and older did not participate (documented & verified) in a sufficient number of assigned core and non-core TANF work participation hours as described in paragraph (F) of rule 5101:1-3-01 of the OAC to meet the required number of hours of paragraph (C) of rule 5101:1-3-12 of the OAC; and

2)If OWF work required participants age 16-24 years did not participate (documented & verified) in a sufficient number of assigned work participation hours from CCMEP activities described in paragraph (C) of rule 5101:14-1-05 (CCMEP rule) of the OAC to meet the higher of either twenty hours per week or the required number of hours of paragraph (C) of rule 5101:1-3-12 of the OAC.

Note that counties are not absolved of responsibility for OWF work-required participants ages 16 to 24 in CCMEP. In alignment with the requested waiver, the difference with CCMEP enrolled OWF work-required participants vs. non-CCMEP enrolled participants is the flexibility of the work activity assigned.

OWF work-required participants age 16-24 enrolled in CCMEP are not held to the same assignment restrictions of core and non-core work participation hours as described in paragraph (F) of rule 5101:1-3-01 of the OAC. Rather, they are held to the CCMEP activities described in paragraph (C) of rule 5101:14-1-05 of the OAC. They are, however, held to the documented and verified participation of those activities to meet the higher of either twenty hours per week or the required number of hours of paragraph (C) of rule 5101:1-3-12 of the OAC. Failure to ensure participants meet their required minimum number of hours could place a county at risk of being a causal county.

Appendix

Table 1:Example Performance Standards

Table 2:Example Performance Report

Table 3:Performance Measure Cohorts

Table 4:Determination of Causal County