Fillable Disability Employment Initiative Project Implementation and Impact Evaluation (Department of Labor)

Fill Online, Printable, Fillable, Blank Disability Employment Initiative Project Implementation and Impact Evaluation (Department of Labor) Form

Use Fill to complete blank online DEPARTMENT OF LABOR (DC) pdf forms for free. Once completed you can sign your fillable form or send for signing. All forms are printable and downloadable.

Disability Employment Initiative Project Implementation and Impact Evaluation (Department of Labor)

On average this form takes 152 minutes to complete

The Disability Employment Initiative Project Implementation and Impact Evaluation (Department of Labor) form is 548 pages long and contains:

  • 0 signatures
  • 0 check-boxes
  • 608 other fields

Country of origin: US
File type: PDF

U.S.A. forms for Department of Labor

BROWSE DEPARTMENT OF LABOR (DC) FORMS

Related forms
Fill has a huge library of thousands of forms all set up to be filled in easily and signed.
Fill in your chosen form
Sign the form using our drawing tool
Send to someone else to fill in and sign.
find another form
 
Disability Employment Initiative Project Implementation and Impact Evaluation
Round 5 through Round 6 Grantees
Project Director
Douglas Klayman, Ph.D.
UAUTHORS
Social Dynamics, LLC
Gary Shaheen, Ph.D.
Douglas Klayman, Ph.D.
Aaron Searson, Ph.D.
Amy Rowland, MFA
Melissa Williamson, A.A.
Abt Associates
Sung-Woo Cho, Ph.D., Statistician
March 19, 2020
Submitted to:
U.S. Department of Labor
Office of Disability Employment Policy
200 Constitution Avenue, NW
Washington, DC 20210
Submitted by:
Social Dynamics, LLC
This report was prepared for the U.S. Department of Labor
(USDOL), Office of Disability Employment Policy, by Social
Dynamics, LLC, and Abt Associates under Task Order
Contract #DOL- OPS-15-A-0015. The views expressed are
those of the authors and should not be attributed to USDOL,
nor does mention of trade names, commercial products, or
organizations imply endorsement of same by the U.S.
Government.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
1
Table of Contents
I. EXECUTIVE SUMMARY .................................................................................................................................. 5
II. INTRODUCTION ............................................................................................................................................ 9
A. DISABILITY AND LABOR FORCE ENGAGEMENT ........................................................................................................ 9
III. DESCRIPTION OF THE PROGRAM ................................................................................................................ 10
IV. EVALUATION DESIGN .................................................................................................................................. 13
V. PROGRAM IMPLEMENTATION .................................................................................................................... 14
A. IMPLEMENTATION OF DEI SERVICE DELIVERY STRATEGIES ..................................................................................... 14
B. FINDINGS FROM THE SOCIAL DYNAMICS SYSTEMS CHANGE CODING RUBRIC ............................................................. 28
C. DEVELOPING DOMAINS AND INDICATORS FOR SYSTEMS CHANGE ............................................................................ 29
VI. GRANT IMPLEMENTATION ROUND 5 .......................................................................................................... 36
A. STATE: CALIFORNIA; FOCUS AREA: ADULTS WITH DISABILITIES ............................................................................... 36
i. Stated Goals and Objectives ..................................................................................................................... 36
ii. DEI Service Delivery Strategies ................................................................................................................. 37
iii. Implementation Summary ........................................................................................................................ 38
iv. Partnerships and Collaborations............................................................................................................... 39
v. Career Pathways ....................................................................................................................................... 41
vi. Outreach and Dissemination .................................................................................................................... 41
vii. Promising Practices: “Pathways for Success” ........................................................................................... 41
viii. Challenges and Sustainability .............................................................................................................. 43
B. STATE: ILLINOIS; FOCUS AREA: ADULTS WITH DISABILITIES .................................................................................... 44
i. Stated Goals and Objectives ..................................................................................................................... 44
ii. DEI Service Delivery Strategies ................................................................................................................. 44
iii. Implementation Summary ........................................................................................................................ 45
iv. Partnerships and Collaborations............................................................................................................... 46
v. Employer Partnership ............................................................................................................................... 47
vi. Career Pathways ....................................................................................................................................... 48
vii. Outreach and Dissemination .................................................................................................................... 49
viii. Promising Practices: “Pathways for Success” ...................................................................................... 50
ix. Challenges and Sustainability ................................................................................................................... 50
C. STATE: KANSAS; FOCUS AREA: ADULTS WITH DISABILITIES ..................................................................................... 51
i. Stated Goals and Objectives ..................................................................................................................... 51
ii. DEI Service Delivery Strategies ................................................................................................................. 52
iii. Implementation Summary ........................................................................................................................ 53
iv. Partnerships and Collaborations............................................................................................................... 54
v. Career Pathways ....................................................................................................................................... 57
vi. Outreach and Dissemination .................................................................................................................... 57
vii. Promising Practices: “Pathways for Success” ........................................................................................... 58
viii. Challenges and Sustainability .............................................................................................................. 58
D. STATE: MASSACHUSETTS; FOCUS AREA: ADULTS WITH DISABILITIES ........................................................................ 60
i. Stated Goals and Objectives ..................................................................................................................... 60
ii. DEI Service Delivery Strategies ................................................................................................................. 61
iii. Implementation Summary ........................................................................................................................ 62
iv. Partnerships and Collaborations............................................................................................................... 63
v. Career Pathways ....................................................................................................................................... 69
vi. Outreach and Dissemination .................................................................................................................... 73
vii. Promising Practices: “Pathways for Success” ........................................................................................... 73
viii. Challenges and Sustainability .............................................................................................................. 74
E. STATE: MINNESOTA; FOCUS AREA: ADULTS WITH DISABILITIES .............................................................................. 77
i. Stated Goals and Objectives ..................................................................................................................... 77
ii. DEI Service Delivery Strategies ................................................................................................................. 77
iii. Implementation Summary ........................................................................................................................ 81
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
2
iv. Partnerships and Collaborations............................................................................................................... 82
v. Career Pathways ....................................................................................................................................... 86
vi. Outreach and Dissemination .................................................................................................................... 88
vii. Promising Practices: “Peer Mentoring” .................................................................................................... 89
viii. Challenges and Sustainability .............................................................................................................. 90
F. STATE: SOUTH DAKOTA; FOCUS AREA: ADULTS WITH DISABILITIES .......................................................................... 92
i. Stated Goals and Objectives ..................................................................................................................... 92
ii. DEI Service Delivery Strategies ................................................................................................................. 93
iii. Implementation Summary ........................................................................................................................ 94
iv. Partnerships and Collaborations............................................................................................................... 96
v. Career Pathways ....................................................................................................................................... 98
vi. Outreach and Dissemination .................................................................................................................... 99
vii. Promising Practices: “Employment Tours” ............................................................................................. 100
viii. Challenges and Sustainability ............................................................................................................ 101
VII. GRANT IMPLEMENTATION ROUND 6 ................................................................................................... 102
A. STATE: ALASKA; FOCUS AREA: YOUTH WITH DISABILITIES .................................................................................... 102
i. Stated Goals and Objectives ................................................................................................................... 102
ii. DEI Service Delivery Strategies ............................................................................................................... 103
iii. Implementation Summary ...................................................................................................................... 105
iv. Partnerships and Collaborations............................................................................................................. 105
v. Career Pathways ..................................................................................................................................... 107
vi. Outreach and Dissemination .................................................................................................................. 108
vii. Promising Practices 1: “Training DRC 1s” ............................................................................................... 108
viii. Promising Practices 2: “Coordinated DEI and DVR Approaches to Serving Youth with Disabilities with
Criminal Justice Backgrounds” ......................................................................................................................... 109
ix. Challenges and Sustainability ................................................................................................................. 110
B. STATE: GEORGIA; FOCUS AREA: YOUTH WITH DISABILITIES .................................................................................. 112
i. Stated Goals and Objectives ................................................................................................................... 112
ii. DEI Service Delivery Strategies ............................................................................................................... 113
iii. Implementation Summary ...................................................................................................................... 114
iv. Partnerships and Collaborations............................................................................................................. 115
v. Career Pathways ..................................................................................................................................... 117
vi. Outreach and Dissemination .................................................................................................................. 117
vii. Promising Practices “Pathways for Success” .......................................................................................... 118
viii. Challenges and Sustainability ............................................................................................................ 118
C. STATE: HAWAII; FOCUS AREA: INDIVIDUALS WITH SIGNIFICANT DISABILITIES ........................................................... 120
i. Stated Goals and Objectives ................................................................................................................... 120
ii. DEI Service Delivery Strategies ............................................................................................................... 120
iii. Implementation Summary ...................................................................................................................... 121
iv. Partnerships and Collaborations............................................................................................................. 124
v. Career Pathways ..................................................................................................................................... 126
vi. Outreach and Dissemination .................................................................................................................. 126
vii. Promising Practices................................................................................................................................. 127
viii. Challenges and Sustainability ............................................................................................................ 127
VIII. QUASI-EXPERIMENTAL METHODOLOGY ............................................... ERROR! BOOKMARK NOT DEFINED.
A. MATCHED COMPARISON GROUP ANALYSIS WHEN COMPARISON GROUPS EXIST ..................................................... 167
B. MATCHING USING TREATMENT AND COMPARISON GROUP INDIVIDUALS’ CHARACTERISTICS ...................................... 168
C. SHORT INTERRUPTED TIME SERIES (SITS) ANALYSIS WHEN NO COMPARISON GROUPS EXIST .................................... 169
D. MEASURING THE IMPACT OF THE CAREER PATHWAYS COMPONENT ...................................................................... 172
E. TREATMENT AND COMPARISON GROUP INDIVIDUALS IN THE CAREER PATHWAYS ANALYSIS ....................................... 173
IX. IMPACT ANALYSIS..................................................................................................................................... 173
A. DESCRIPTIVE STATISTICS ................................................................................................................................ 173
B. EMPLOYMENT/RETENTION/EARNINGS OUTCOMES ............................................................................................ 175
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
3
C. SURVEY TO COLLECT INFORMATION ON ACTIVITIES OF DAILY LIVING AND DISABILITY TYPE IN COMPARISON GROUP SITES 175
D. IMPACT ESTIMATES FOR MATCHED COMPARISON GROUP ANALYSIS ...................................................................... 177
E. OVERALL IMPACT OF DEI, WITH MATCHING ..................................................................................................... 179
F. IMPACT ESTIMATES FOR SHORT INTERRUPTED TIME SERIES (SITS) ANALYSIS .......................................................... 182
G. IMPACT OF CAREER PATHWAYS ...................................................................................................................... 187
H. IMPACT OF CAREER PATHWAYS, WITH MATCHING ............................................................................................. 190
X. FINDINGS AND DISCUSSION ...................................................................................................................... 193
A. POTENTIAL AREAS FOR FUTURE RESEARCH ....................................................................................................... 194
B. DEI EVALUATION LOGIC MODEL .................................................................................................................... 195
C. SUSTAINABILITY FINDINGS ............................................................................................................................. 213
APPENDIX 1: AMERICAN JOB CENTER SURVEY .......................................................................................................... 218
APPENDIX 2: DEI ROUND 5 AND ROUND 6 WDAS ..................................................................................................... 219
APPENDIX 3: ROUND 5 TREATMENT AND COMPARISON SITES FOR QED ANALYSIS................................................. 221
APPENDIX 4: ROUND 6 TREATMENT AND COMPARISON SITES FOR QED ANALYSIS................................................. 222
APPENDIX 4: ROUND 6 TREATMENT AND COMPARISON SITES FOR QED ANALYSIS................................................. 222
EXHIBIT 1: COMPLETE LIST OF SERVICE DELIVERY STRATEGIES ................................................................................... 11
EXHIBIT 2: UTILIZATION OF SERVICE DELIVERY STRATEGIES ....................................................................................... 12
EXHIBIT 3: ODEP-SUGGESTED DEI ROUND 5 AND ROUND 6 SERVICE DELIVERY STRATEGIES .................................... 21
EXHIBIT 4: PRE- AND POST-INTERVENTION TREND LINES TO MEASURE THE IMPACT OF AN INTERVENTION.….……171
EXHIBIT 5: DEI COMPARISON GROUP QED RAW SAMPLE CHARACTERISTICS-UNMATCHED………………………………….174
EXHIBIT 6: DEI COMPARISON GROUP QED RAW SAMPLE OUTCOMES-UMATCHED………………………………………………175
EXHIBIT 7: SURVEY CHARACTERISTICS OF INDIVIDUALS WITHIN COMPARISON SITES…………………………………………..176
EXHIBIT 8: DEI COMPARISON GROUP QED IMPACT RESULTS AND CONTROL VARIABLE COEFFICIENTS —
UMATCHED……………………………………………………………………………………………………………………………………………………….…178
EXHIBIT 9: DEI COMPARISON GROUP QED RAW SAMPLE CHARACTERISTICS — Matched ……………………………………180
EXHIBIT 10: DEI COMPARISON GROUP QED IMPACT RESULTS AND CONTROL VARIABLE COEFFICIENTS
-MATCHED……………………………………………………………………………………………………………………………………………………………181
EXHIBIT 11: SITS SAMPLE CHARACTERISTICS (INDIVIDUALS IN TREATMENT WDAs PRE- AND POST DEI) …….….……184
EXHIBIT 12: DEI SITS IMPACT RESULTS AND CONTROL VARIABLE COEFFICIENTS………………………………………………....186
EXHIBIT 13: CAREER PATHWAYS COMPARISON GROUP QED RAW SAMPLLE CHARACTERISTICS — UNMATCHED…188
EXHIBIT 14: CAREER PATHWAYS COMPARISON GROUP QED IMPACT RESULTS AND CONTROL VARIABLE
COEFFICIENTS-UNMATCHED…………………………………………………………………………………………………………………………………189
EXHIBIT 15: CAREER PATHWAYS COMPARISON GROUP QED RAW SAMPLE CHARACTERSTICS
— MATCHED…………………………………………………………………………………………………………………………………………………………191
EXHIBIT 16: CAREER PATHWAYS COMPARISON GROUP QED IMPACT RESULTS AND CONTROL VARIABLE
COEFFICIENTS — MATCHED………………………………………………………………………………………………………………………………….192
EXHIBIT 17: QUALITIVE RESEARCH QUESTIONS…………………………………………………………………………………………………….194
EXHIBIT 18: DEI LOGIC MODEL………………………………………………………………………………………………………………………………197
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
4
ACRONYMS
(ABE) Adult Basic Education
(ADA) Americans with Disabilities Act
(AJC) American Job Center
(ARC) Active Resource Coordination
(AT) Assistive Technology
(BLN) Business Leadership Network
(CE) Customized Employment
(CNA) Certified Nursing Assistant
(CP) Career Pathways
(CWIC) Community Work Incentives Coordinator
(DCS) MassHire Department of Career Services
(DEA) Disability Employment Accelerator
(DEI) Disability Employment Initiative
(DLR) South Dakota Department of Labor and
Regulation
(DOR) California Department of Rehabilitation
(DRC) Disability Resource Coordinator
(DVR) Department of Vocational Rehabilitation
(EN) Employment Network
(ETA) Employment and Training Administration
(GED) General Education Diploma
(GWDB) Governor’s Workforce Development
Board
(ICI) Institute for Community Inclusion
(ILP) Individual Learning Plan
(IRT) Integrated Resource Team
(IT) Information Technology
(JSWDs) Jobseekers with Disabilities
(MCB) Massachusetts Commission for the Blind
(MCDHH) Massachusetts Commission for the
Deaf and Hard of Hearing
(MOU) Memorandum of Understanding
(MRC) Massachusetts Rehabilitation Commission
(NDEAM) National Disability Employment
Awareness Month
(NDI) National Disability Institute
(ODEP) Office of Disability Employment Policy
(OJT) On-the-Job Training
(PASS) Plan for Achieving Self-Support
(PIRL) Participant Individual Record Layout
(QED) Quasi-Experimental Design
(SCCS) Systems Change Coding Scheme
(SDS) Service Delivery Strategy
(SITS) Short Interrupted Time Series
(SNAP) Supplemental Nutrition Assistance
Program
(SSA) United States Social Security
Administration
(SSDI) Social Security Disability Insurance
(SSI) Supplemental Security Income
(TANF) Temporary Assistance for Needy
Families
(TORQ) Transferable Occupation Relationship
Quotient
(T/TA) Training and Technical Assistance
(TTW) Ticket to Work
(USDOL) United States Department of Labor
(VR) Vocational Rehabilitation
(WBL) Work-Based Learning
(WDA) Workforce Development Area
(WDD) Hawaii Workforce Development Division
(WIA) Workforce Investment Act
(WIOA) Workforce Innovation and Opportunity
Act
(WIPA) Work Incentives Planning and Assistance
(WWL) Work Without Limits
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
5
I. Executive Summary
1. The Disability Employment Initiative (DEI) Round 5 and Round 6 evaluation was
conducted by Social Dynamics, LLC, and Abt Associates. It was designed to report to the
United States Department of Labor’s (USDOL) Office of Disability Employment Policy and
Employment and Training Administration on the implementation, outcomes, and impact of
the DEI. Research questions focused on employment outcomes and the impact of the project
on jobseekers with disabilities (JSWDs). Other research questions centered on service
delivery strategies (SDSs) and grant requirements that were embedded within the workforce
development systems of participating Workforce Development Areas (WDAs).
1
2. Grantees used a wide array of SDSs across treatment sites and exhibited considerable
variability in the way the DEI and grant-funded leadership positions were
implemented. A primary focus on the goal of employment for JSWDs was persistent
throughout. Unlike some earlier rounds, project implementation and engagement of JSWDs
was exceedingly precise, with few challenges, particularly in states that had prior DEI
experience. Staff turnover, retention, and contamination were not factors that, for most
grantees, affected the implementation or operation of the DEI. Those that experienced these
challenges were able to overcome them relatively quickly. At the same time, the level of
engagement of JSWDs appeared to be robust. With respect to certain prior rounds, we
characterized the DEI as a “living laboratory of practical insight about the way JSWDs can
be served through the AJCs and their partners.”
2
DEI Round 5 and Round 6 functioned with
less variability than in prior rounds with clearly defined roles and a distribution of tasks. In
most Round 5 and Round 6 WDAs, DEI State Leads and Disability Resource Coordinators
(DRCs) knew who to report to, who made key decisions, and who provided guidelines
leading to project effectiveness.
3. The DEI was designed to improve the capacity of AJCs to increase employment
opportunities for people with disabilities. AJCs are primarily financed by the federal
government through funding streams that provide support services to veterans, youth,
dislocated workers, school dropouts, individuals in poverty, and individuals with disabilities.
DEI State Leads and DRCs worked in 12 (Round 5 and Round 6) states to build capacity for
the implementation of SDSs and training for staff and JSWDs and to foster opportunities for
partnerships and collaborations with WIOA providers and community-based agencies to
resolve barriers that inhibited access to needed services. DEI Round 5 and Round 6 grantees
administered SDSs with proficiency, augmented by a plethora of services and supports,
including enrollment in WIOA and Ticket to Work (TTW).
4. The DEI included two grant-funded leadership positions. Each grantee included a DEI
State Lead and one or more DRCs for each treatment WDA. These positions provided
executive leadership and expertise in workforce development, program implementation,
1
Throughout the text of this report, we refer to both WDAs and American Job Centers (AJCs). A single WDA may
include one or more AJCs that provide core, intensive, and/or Workforce Innovation and Opportunity Act (WIOA)
services.
2
Cori DiBiase, comment, March 9, 2020.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
6
SDSs, partnerships and collaborations, benefits counseling, TTW and Employment Network
(EN) program management, organizational development, and project sustainability.
5. The DEI State Lead provided executive leadership. DEI State Leads were responsible for
monitoring finances, developing state and local partnerships, and providing workforce
development and program implementation support. They also monitored ENs, conferred with
WDAs, managed the grant, and corresponded with the Federal Project Officers. DEI State
Leads also provided support for the implementation of DEI projects and DRCs.
6. DRCs were tasked with the implementation of SDSs in addition to partnership-
building, benefits counseling, TTW and EN program management, organizational
development, and project sustainability. DRCs who completed Community Work
Incentives Coordinator (CWIC) training provided benefits counseling to JSWDs. Other DRC
tasks supporting JSWDs included employment preparation, job placement support, soft skills
training, job coaching, leadership for Integrated Resource Teams and support services for
JSWDs as they enrolled in community colleges to improve their basic skills in math, reading,
or English prior to accepting an offer of employment. DRCs who had prior case management
experiences were often recognized for their ability to understand the needs of JSWDs and
respond effectively to those needs.
7. The DEI Round 5 and Round 6 included 12 grantees.
Round 5 2014–2019
California (n=8,141)
Illinois* (n=3,120 of which 2,560 were youth)
Kansas (n=3,214)
Massachusetts**
Minnesota (n=5,440)
South Dakota (n=694)
Round 6 2015–2019
Alaska (n=310)
Georgia (n=84)
Hawaii**
Iowa (n=8,936)
New York (n=3,888)
Washington**
*Youth grantee/**Grantee did not provide WIA data
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
7
Round 5
California Employment Development Department
# of prior DEI grants: 1= R2
Workforce Services Division Illinois Department of Commerce and Economic Opportunity
# of prior DEI grants: 2= R1, R4
Kansas Department of Commerce
# of prior DEI grants: 1= R1`
Massachusetts Executive Office of Labor and Workforce Development, Department of Career Services
# of prior DEI grants: 1= R3
Minnesota Department of Employment and Economic Development, Division of Workforce
Development
# of prior DEI grants: 1= R3
South Dakota Department of Labor and Regulation, Workforce Training
# of prior DEI grants: 1= R2
Round 6
State of Alaska, Department of Labor and Workforce Development
# of prior DEI grants: 2= R1, R4
Georgia Department of Economic Development, Workforce Division
# of prior DEI grants: 0
State of Hawaii Department of Labor and Industrial Relations
# of prior DEI grants: 1= R2
Iowa Workforce Development
# of prior DEI grants: 1= R3
New York State Department of Labor
# of prior DEI grants: 1= R4
Washington State Employment Security Department
# of prior DEI grants: 1= R2
8. DEI provided coordination of integrated training and support services. DEI training and
support services provided adult and youth JSWDs with integrated workforce development
systems designed to improve training and employment outcomes. Individuals in a treatment
WDA who self-disclosed a disability were eligible to enroll in the DEI. The program did not
require individuals to provide documentation regarding the type or severity of their disability.
DEI participants received one-on-one support from a DRC. DEI participants from 12
grantees self-disclosed a disability but occasionally did not directly discuss their disability
with a DRC.
9. SDSs facilitated JSWD outreach and engagement and employer Partnerships and
Collaborations. Integrated Resource Teams, Blending and Braiding Resources, and
Customized Employment were implemented with selected features of SDSs, including
Active Resource Coordination, Partnerships and Collaborations, Asset Development, Work-
Based Learning, Apprenticeships, Job Shadowing, and Career Pathways. While there were
differences in the fidelity of SDSs, it was apparent that training such as person-centeredness
and systems integration resulted in improvements in service availability and implementation.
We conclude that while grantees may not have implemented each of their SDSs with fidelity,
the net effect of their introduction appears to have improved Partnerships and Collaborations
with employers and stakeholders and access to work incentives such as Trial Work
Period, Impairment-Related Work Expenses, Plan to Achieve Self-Support, and Partnership
Plus.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
8
10. DEI Round 5 and Round 6 grantees implemented six requirements:
i. Selection of either a youth or adult focus.
ii. Physical and programmatic accessibility of participating AJCs.
iii. Implementation of at least two DEI SDSs: Active Resource Coordination; Asset
Development; Blending and Braiding Resources; Benefits Planning; Customized
Employment; Guideposts for Success; Integrated Resource Teams; Partnerships and
Collaborations; Work-Based Learning Opportunities; and Entrepreneurship.
iv. Completion of a sustainability plan for after the grant period.
v. Availability of CWICs for United States Social Security Administration (SSA)
beneficiaries to provide benefits services.
vi. Implementation of TTW and access to work incentives webinars.
11. SSA Work Incentives Planning and Assistance providers certified CWICs who
provided benefits counseling to beneficiaries seeking employment. Many DRCs were
CWICs who made decisions about the impact of earnings on health care and benefits. Some
SSA beneficiaries enrolled in TTW to limit the effects of working on their continuing
disability review,
3
which can affect health insurance coverage.
12. We identified promising practices among Round 5 and Round 6 grantees. For example,
California’s Traveling DRC provided 1-day trainings for AJC Employment Specialists “to
bring best practices and perspectives discovered through California’s DEI and Disability
Employment Accelerator to improve service delivery to individuals with disabilities in each
treatment WDA.”
4
Illinois DRCs had a strong partnership with Vocational Rehabilitation that
included a dedicated staff member who made referrals to Career Pathways services, enrolled
individuals in WIOA services, and provided oversight of individuals who enrolled in
community colleges. DRCs worked with Vocational Rehabilitation to make referrals to
employment and Career Pathways training while apprenticeships and internships allowed
youth to learn about employment and what it entails (e.g., receiving remuneration, working
in a group environment, adhering to the requirements of a supervisor).
13. While there were no measureable impacts of DEI on wage and employment outcomes,
there were consistently positive impacts of career pathways programs on the same
outcomes. The DEI Round 5 and Round 6 impact evaluation included three distinct research
designs, with the aim of measuring the impact of DEI on employment and wage outcomes, as
well as the career pathways intervention on the same outcomes. Randomization was not used
in the evaluation, but instead, quasi-experimental methods were used to determine the impact
of DEI on outcomes. The evaluation team used administrative that were collected on people
with disabilities who were entering DEI-funded AJCs and comparison group AJCs that were
not funded as part of DEI, and used a matching procedure to help determine the impact of
receiving services at a treatment AJC. The level of comparison was at the WDA level, with
outcomes measured at the individual customer level. The results indicated that although there
3
U.S. Social Security Administration. (n.d.). Understanding Supplemental Security Income continuing disability
reviews — 2019 edition. Retrieved from https://www.ssa.gov/ssi/text-cdrs-ussi.htm
4
IL DRC comment, March 9, 2020.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
9
were no sustained impacts of DEI on outcomes, there were consistently positive impacts of
being in a career pathways program on the same outcomes.
II. Introduction
The Office of Disability Employment Policy (ODEP) of the United States Department of
Labor (USDOL) was created in 2001. Its mission is to provide national leadership by developing
and influencing disability employment-related policies and practices affecting an increase in the
employment of people with disabilities. Since then, ODEP has designed numerous programs and
policies that support people with disabilities and the workforce development system by
promoting Career Pathways training and employment, the Campaign for Disability
Employment, and the Workforce Recruitment Program, which connects employers with highly
motivated college students and recent graduates with disabilities. ODEP, in partnership with the
Employment and Training Administration (ETA), has also implemented which focuses on Stay-
at-Work/Return-to-Work strategies. Other projects include the SSI [Supplemental Security
Income] Youth Recipient and Employment Transition Formative Research
Project, which identifies promising programs and policies for youth SSI recipients,
and Evaluating the Accessibility of American Job Centers for People with Disabilities.
A. Disability and Labor Force Engagement
The median income of families living in the United States with at least one member with
a disability is 72 percent of the national median, while median incomes for families with at least
one member with a mental or physical disability are 66 percent and 68 percent, respectively, of
the national median.
5
6
All things being equal, age and disability are the most important
predictors of labor force participation leading to employment, as older workers are more likely to
leave the labor market than younger workers due to the onset of disability or social and economic
factors.
7
These factors include chronic health conditions and the changing social aspects of
income, education, gender, race/ethnicity, and variability in the labor market.
8
9
Labor force engagement tends to decline after the onset of a disability due to chronic
conditions.
10
Chronic conditions have been linked to reductions in household income and
increases in the likelihood of poverty, limitations in human capital, and job skill development.
11
For example, people with cerebral palsy, spinal cord injury, or rheumatoid arthritis have much
higher labor force participant rates than those with visual impairments or intellectual disabilities.
Working-age people with sensory, physical, and/or cognitive disabilities
12
are also at a
5
Minkler, Fuller-Thomson, & Guralnik, 2006.
6
Disability Statistics and Demographics Rehabilitation Research and Training Center. University of New
Hampshire.
7
Mitchell, Adkins, & Kemp, 2006; Rigg, 2005.
8
Mitchell et al., 2006.
9
Crimmins, Reynolds, & Saito, 1999.
10
Mitchell et al., 2006; Young et al., 2002.
11
Loprest & Maag, 2009.
12
Houtenville, Erickson, & Lee, 2012.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
10
significant disadvantage in terms of their employment and earnings when compared with
working-age individuals without a disability. While age is the most important predictor of labor
force participation, there are a variety of health, social, and economic factors involved in
workforce participation, including long-term chronic health conditions, work history, family
income, educational attainment, gender, race/ethnicity, and the vagaries of the labor market.
13
14
III. Description of the Program
The workforce development systems in the United States changed in 2015 with the
transition from the Workforce Investment Act (WIA) to the Workforce Innovation and
Opportunity Act (WIOA). WIOA replaced WIA, which used a hierarchical service delivery
system that required jobseekers to enroll in core services, employment placement, and intensive
services that included skills assessments and enrollment in college. Jobseekers with disabilities
(JSWDs) were required to start from core and intensive services in order to allocate resources
into three categories: disadvantaged adults, dislocated workers, and youth. In 2015, federal
resources were provided to USDOL agencies to WDAs to support the new WIOA programs. In
most WDAs, there is one comprehensive AJC and one or more satellite centers. WIOA provides
three levels of services for adults and dislocated workers: basic, individualized, and follow-up
services that are no longer needed because they can be accessed through WDAs; what were
intensive and training services are now “individualized services” that can be coordinated through
Partnerships and Collaborations.
USDOL’s ETA and ODEP sought to resolve chronic unemployment and limited
opportunities for people with disabilities. (WIOA) increased the capacity of WDAs to serve
jobseekers whether or not they have a disability. Understanding that there is a need for further
investment in JSWDs, the Disability Employment Initiative (DEI) was implemented to increase
the capacity of WDAs
15
and to facilitate Partnerships and Collaborations that improve access to
AJC services for people with disabilities.
16
Funding for DEI Round 5 and Round 6 was provided in 2017 and 2018, respectively,
when ETA and ODEP released a Solicitation for Grant Applications that made provisions for
$14,837,784.70 for Round 5 and $14,911,243.00 for Round 6 to support 12 cooperative
agreements through the Consolidated Appropriations Act of 2010.
17
One-half of the DEI funds
went to ETA for the training and technical assistance (T/TA) of DEI grantees. The other half
went to ODEP for the DEI evaluation.
As with earlier rounds, Round 5 and Round 6 were required to provide services to either
adults or youth participants. The age range for adult JSWDs was 21 or older, while the age range
13
Crimmins, Reynolds, & Saito, 1999.
14
Mitchell, Adkins, & Kemp, 2006; Rigg, 2005.
15
Information about WDAs and their services.
16
For detailed information on promising practices and challenges, go to page 37 for Round 5 and page 103 for
Round 6 grantee narratives. For detailed information on the implementation of DEI SDSs, go to page 15.
17
Pub. L. No. 111-117
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
11
for youth was 14–21. All but one Round 5 and Round 6 grantee did not have a prior DEI grant as
the Georgia Department of Economic Development, Workforce Division, was new to the DEI in
Round 6. Nonetheless, all grantees were positioned to maintain Ticket to Work (TTW) activities,
as many collected resources that provided access to new service delivery strategies (SDSs) and
training prior to project implementation. DEI State Leads, Disability Resource Coordinators
(DRCs), and AJC Employment Specialists helped grantees acclimate to the DEI and recruit
Ticket holders, collaborate with Vocational Rehabilitation (VR), and coordinate Ticket services
under Partnership Plus arrangements.
18
19
Grantees also had access to the following SDSs in
Exhibit 1 to help JSWDs obtain credentials required for in-demand occupations.
Exhibit 1: Complete List of Service Delivery Strategies
1. Active Resource Coordination (Selected Components)
2. Apprenticeships/Job Shadowing (Selected Components)
3. Asset Development Strategies (Selected Components)
4. Career Pathways
5. Customized Employment (Selected Components)
6. Guideposts for Success (Not selected)
7. Entrepreneurship/Self-Employment (Selected Components)
8. Integrated Resource Teams (Selected Components)
9. Leveraging Resources and Services/Blending and Braiding Resources (Selected
Components)
10. Work-Based Learning Opportunities (Selected Components)
20
Most grantees used components of SDSs to provide support for JSWDs. For example,
Active Resource Coordination, Integrated Resource Teams, Work-Based Learning,
Apprenticeships, and Career Pathways were commonplace as they facilitated communications
across WDAs and leveraged resources for transportation, training, employment opportunities,
and college enrollment that supported employment goals. Some SDSs were selected in their
entirety. Others were used as components of SDSs. For example, in the boxes in Exhibit 2 below,
SDSs 1–4 were more likely to be selected in their entirety versus SDSs 5–8, which were used to
augment specific components of an SDS.
18
Goodley, D. (2010). Disabilities studies: An interdisciplinary introduction. London: Sage Publications.
19
As the statutory-level Employment Network (EN), VR agencies could enter into Partnership Plus agreements with
DEI grantees, allowing VR and AJCs to receive outcome payments through case closures. DEI grantees could also
receive milestone payments for follow-up job retention services.
20
King, C. T., Choi, J., & Cerna Rios, A. (2014). Improving services for persons with disabilities under the
Workforce Investment Act and related programs: Challenges, opportunities, and a way forward. Retrieved from
http://raymarshallcenter.org/files/2014/11/Improving-Services-for-Persons-with-Disabilities-under-the-Workforce-
Investment-Act-and-Related-Programs_APPAM.pdf
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
12
Exhibit 2: Utilization of SDSs
1. Active Resource Coordination
2. Integrated Resource Teams
3. Blending and Braiding Resources
4. Career Pathways
5. Customized Employment
6. Guideposts for Success
7. Entrepreneurship
8. Leveraging Resources and Services
Of the 10 SDSs listed in Exhibit 1, we identified eight that used components of other
SDSs (Active Resource Coordination, Apprenticeships/Job Shadowing, Asset Development
Strategies, Career Pathways, Customized Employment, Integrated Resource Teams, Leveraging
Resources and Services, and Work-Based Learning) despite grantees not having formally
selected them for their projects. By scaling down the number of SDSs, grantees implemented
SDS components that allowed them to introduce and scale-up a broader array of support services
for JSWDs. In addition, using facile and familiar approaches to SDS implementation played a
role in accommodating the frequency of in-person meetings between DRCs and JSWDs through
Integrated Resource Teams (augmenting them when necessary with telephone and/or email
meetings).
Each SDS was used to “individualize the employment relationship” by identifying the
strengths and interests of each JSWD and utilizing a menu of customized assessments and team
planning that, in optimal situations, involved AJC Employment Specialists and VR counselors.
Furthermore, Round 5 and Round 6 grantees focused on systems change and improving access to
services for JSWDs among multiple partners, including AJCs, and augmenting their systems
change strategies by accessing resources from DEI and T/TA providers. Specific components of
Customized Employment and Work-Based Learning, including on-the-job training (OJT) and
internships, were available to help JSWDs obtain credentials required for in-demand
occupations. SDSs were implemented concurrently and often in partnership with other entities
such as USDOL Jobs for Veterans State Grants staff and local industries and employees. DRCs
and related partners were also involved in systems change, aiding WDA staff and providing
services for JSWDs designed to improve staff skills.
Due to their experience with prior DEI projects, Round 5 and Round 6 grantees did not
have many of the same difficulties of earlier grantees, such as the implementation of TTW.
While Alaska (Round 6) had some challenges throughout their grant period with SSA program
requirements, other grantees mentioned resource limitations and limited access to Community
Work Incentives Coordinator (CWIC) services, despite a less lengthy SSA suitability
determination process as compared to DEI Round 1 through Round 4 projects, grantees were
able to address these issues, and overcome them. These issues had been resolved due in large
part to the considerable experience of ODEP-ETA, DEI State Leads and DRCs, and an approach
to systems change that integrated employment and training into local communities with the
assistance of DRCs and experienced WDA personnel. DEI knowledge of ways to address
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
13
systemic barriers was also enhanced by DOL’s WorkforceGPS webinars and DEI peer-peer
learning exchanges.
While it is well-known that some JSWDs had trepidations about working due to a fear of
loss of SSI/Social Security Disability Insurance (SSDI) benefits, they generally reported that in
many instances CWICs allayed their concerns. CWICs, many of whom were DRCs, provided
counseling and outreach to beneficiaries who were eligible for work incentive programs. Free
CWIC services ensured that individuals on SSI or SSDI could transition to employment with the
help of SSA work incentives. The creation of ENs also facilitated systems change as DEI State
Leads, DRCs, and SSA staff worked together to implement ENs and create expectations and
working relationships that would leverage TTW resources to support AJC services for JSWDs
after the grants ended.
IV. Evaluation Design
The goal of the DEI was to improve the employment outcomes of JSWDs through the
application of specialized SDSs and the facilitation of systems change. The purpose of the DEI
impact evaluation was to measure and assess the outcomes and impact of DEI Round 5 and
Round 6. The main research questions focused on employment, employment retention, and
wages of DEI participants, while other questions focused on how specific SDSs impacted
participant outcomes. The source of information for the impact analysis came in the form of
WIA administrative data, which had been collected on individuals who received any form of
services at an AJC.
To provide context on the individuals with disabilities who received services outside of
DEI-funded AJCs, a survey on employment, activities of daily living, and disability type was
analyzed and reported in this evaluation. The source of information from this analysis came in
the form of a novel survey that was administered via web and phone to individuals in comparison
group AJCs who elected to answer questions in exchange for a $15 gift card.
The main impact analysis determined the effect of the DEI intervention on outcomes
using two quasi-experimental approaches. In addition, a tertiary analysis determined the effect of
Career Pathways programs on outcomes using only individuals who received services in the
treatment group. To provide context for these findings, the study examined descriptive
characteristics of all the individuals in the WIA data sample as well as findings on activities of
daily living and disability type by individuals in only the comparison group. The overall findings
suggest that although DEI as a program did not have statistically significant impacts on
outcomes, Career Pathways enrollment did have impacts on outcomes even after accounting for
selection bias.
We designed the primary impact analysis to measure differences in program outcomes
using a quasi-experimental design (QED) that compared treatment group to the comparison
group WDAs at the individual per-person level. The primary analysis included only state
grantees that had both treatment and comparison WDAs. A secondary analysis using a short
interrupted time series (SITS) design included treatment WDAs in order to accommodate those
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
14
states that only had a single WDA or had comparison WDAs that may have also received
services funded through DEI.
21
The SITS analysis provided a baseline impact measure on how
outcomes changed before and after the appearance of interventions for each round of grantees.
22
A tertiary analysis used WIA data from only the treatment group individuals to determine
whether Career Pathways programs had an impact on outcomes by delineating two groups:
individuals who enrolled in Career Pathways programs and those who did not. By using a QED
that compared these two groups, we determined if the participation in Career Pathways programs
impacted individual-level employment, employment retention, and earnings and outcomes.
V. Program Implementation
Fundamental to DEI practices were the ways that DRCs mobilized stakeholders and
resources to build strong, supportive services around the jobseeker in ways that respected their
self-direction and choice. Consequently, DEI State Leads and DRCs often found themselves
engaging in systems change through Partnerships and Collaborations and services on a daily
basis. Partnership development also included helping state, local, and community-based agencies
build capacity by training their staff in DEI SDSs. While some DEI grantees focused more on
particular job and career development strategies (e.g., Career Pathways, Customized
Employment, Self-Employment, Asset Development), they all involved some measure of
coordinated, leveraged team support leading to employment through the use of SDSs. Each of
the SDSs utilized by DEI grantees are described below in greater detail.
A. Implementation of DEI Service Delivery Strategies
As mentioned earlier, grantees used components of SDSs to operationalize Active
Resource Coordination and Integrated Resource Teams (IRTs), Work-Based Learning and
Apprenticeships, Career Pathways, youth in transition with disabilities, IRTs that facilitated
communications across WDAs, and leveraging resources for transportation, training,
employment opportunities, and college classes that supported employment goals.
21
California and Illinois were deemed to have comparison group WDAs that may have received services or benefits
from DEI funding and were excluded from the QED analysis. South Dakota (Round 5) and Alaska (Round 6) each
had only one WDA and were also excluded from the QED analysis. Massachusetts (Round 5), Hawaii (Round 6),
and Washington State (Round 6) did not make their data available for analysis.
22
It should be noted that quasi-experimental approaches cannot fully eliminate selection bias, and that proper
identification of the intervention effect is more difficult to determine than a randomized trial at the individual level.
Identification is particularly difficult to accomplish at the WDA level, when we know that some comparison group
WDAs were offering services that resembled the ones in some treatment group WDAs. Although the matching
strategy helps to make sure that individuals across treatment and comparison WDAs are similar to one another, the
authors acknowledge the difficulty in completely isolating the effect of DEI on outcomes. The authors have taken
states out of the impact analysis completely when there is evidence that there is an abundance of contamination in
the comparison group (e.g., California for the matched comparison group QED).
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
15
The figures and text below illustrate how various SDSs optimally flowed from the point
of outreach and engagement to employment and supportive services. Improvements in support
services were achieved when services were delivered with substantiated principles and practices.
Despite the fact that over the course of the DEI Round 5 and Round 6 grants DRCs and partners
had access to extensive training on SDSs, implementing them with fidelity was inconsistent.
While we found dispersion regarding the fidelity of SDSs, it was also apparent that
training focusing on person-centeredness and systems integration aspects inherent to all DEI
SDSs at the very least resulted in improvements in attitudes, awareness, and service availability
for JSWDs being served at AJCs compared to other comparison WDAs.
“Generally, we conclude that even though many grantees did not implement each SDS with fidelity, the
overall net effect of their introduction and use appears to have improved their partner agencies’
employment services by providing access to SDSs and learning how they may contribute to training and
employment. The following provides descriptions of each SDS implemented by DEI round and state.”
Active Resource Coordination (ARC)
Round 6 (R6)
Alaska – AK
Hawaii – HI
Iowa – IA
New York – NY
Washington – WA
Kansas: ARC occurred regularly statewide. DRCs frequently engaged other staff from
the AJC and other partners with resources because “we want everyone to get all the resources we
have.” We also heard that without DEI, ascertaining this information and resources would likely
be “pushed back to the participant.” One partner agency indicated that “more regular meetings”
could improve ARCs, alluding to the fact that sometimes it was a difficult task to get all
necessary partners to the table to better understand the resources they could collectively bring to
bear for a particular individual and ARC. It was described by a DRC as “finding the best
resources to meet individuals’ needs.”
South Dakota: ARC was practiced in Rapid City before enrollment as the first step of
forming an IRT, as is appropriate to the definition. It involved discussing eligibility and
employment goals. There was also statewide training on Active Resource Coordination.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
16
Asset Development
Round 5 (R5)
Round 6 (R6)
Not selected by any
R5 grantees
Washington – WA
Hawaii – HI
Generally, when queried about Asset Development, most respondents appeared to equate
that service with Benefits Planning. However, even though they are aligned, each is a distinct
SDS. We obtained data by interviewing DEI leaders, DRCs, and partners and stakeholders to
determine how extensively a full range of Asset Development services was being provided, or if
grantees indicating that they focused on Asset Development were only providing benefits and
work incentives planning, even if only basic personal budgeting. In this regard, the correlation
between Asset Development and Benefits Planning is strong because DEI customers who
received SSI/SSDI and/or veterans’ benefits or other forms of public assistance needed to know
how to manage their earnings from work and how they could spend and save money without
jeopardizing their benefits.
Although Alaska, Hawaii, Washington, and South Dakota selected Asset Development,
of these states, Alaska, Washington, and South Dakota made significant progress in creating and
strengthening Asset Development partnerships.
Alaska: DRCs often worked at an individual level in terms of Asset Development by
giving one-to-one advice on benefits and budgeting. One WDA indicated that a section on
budgeting was included as part of the basic training program. Alaska required all DRCs to go
through CWIC training for Benefits Planning. Frequently, some aspects of financial literacy were
provided, as one DRC from Alaska covered budgeting after her clients got full-time jobs and
advised them to be cautious of scams. She said, “Just be aware, watch your social media. Then
budgeting and saving. I do address that.” And another DRC integrated some aspects of financial
planning into her Benefits Planning work with customers by referring them to marriage and
family therapy and resources from the Volunteer Income Tax Assistance service.
New York: All DRCs were trained to answer questions related to Asset Development
and become familiar with the needs of their AJCs in this regard. Some DRCs attended a monthly
CA$H Coalition meeting and promoted both Volunteer Income Tax Assistance at the Albany
AJC.
“I’m a member of the CA$H Coalition. It’s part of United Way here. It’s creating assets, savings,
and hope. It’s a group that meets monthly and includes an SSA rep, which is fabulous. We get
together and do outreach, attend some political kick-off type things, on behalf of our customers.”
— DRC
South Dakota: South Dakota integrated Asset Development strategies that promoted
financial literacy, credit counseling, and tax assistance. There was also evidence of helping
JSWDs with public assistance related to housing, child care, and transportation. A number of
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
17
examples substantiate how the DEI was networking with financial literacy resources on behalf of
its customers. One WDA hosted a Wells Fargo Financial Literacy class in conjunction with a Job
Search Assistance class and My Free Taxes information that was shared at the South Dakota
Youth Council meeting. The WDA also hosted two Credit When Credit is Due classes, taught by
a local partner of Consumer Credit Counseling of the Black Hills. We also found some evidence
that local AJCs offered financial literacy classes to participants.
Washington: Washington understood the importance of integrating Asset Development
and financial literacy into its services strategy. From the outset, they developed their partnership
with the Northwest Area Foundation to provide financial literacy and income stability training
and support to participants. While our data substantiates that a number of DEI grantees included
Asset Development and financial literacy along with Social Security Benefits Planning,
Washington took a statewide approach to the integration of these services by leveraging partner
resources for ongoing access to expertise and training and provided a substantial number of
JSWDs with financial wellness training sessions over the life of the grant.
Blending and Braiding Resources
Round 5 (R5)
Round 6 (R6)
California – CA
Alaska – AK
Illinois – IL
Georgia – GA
Kansas – KS
Hawaii – HI
Minnesota – MN
Iowa – IA
South Dakota – SD
New York – NY
Washington – WA
Blending and Braiding Resources was frequently observed across Round 5 and 6 DEI
grantees. It is a process that is integral to ARC and IRTs and is an essential component of
systems change. Examples of Blending and Braiding Resources are provided below.
Alaska: Alaska embedded Blending and Braiding Resources quite effectively due to its
strong partnerships and the use of IRTs to support youth with disabilities in exploring various
career paths and educational opportunities. One DRC gave an example of a youth customer who
needed additional help with housing and transportation. According to the DRC, “I worked with a
bunch of different agencies on that. We were able to get him into Job Corps. … And they got him
in, right at the end of May, which was really nice because that’s when his housing was over and
then in Alaska, to do that temporary housing would have been so expensive.” The collaboration
between DEI and Department of Vocational Rehabilitation’s (DVR) Pre-Employment Transition
Services program resulted in co-funded summer youth employment programs, institutes for
youth in the criminal justice system, and shared financial support for training, transportation, and
other youth employment-related services.
California: California Blended and Braided Resources both at the state and WDA levels.
At the state level, the grantee combined resources from DEI and the Disability Employment
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
18
Accelerator (DEA). While working with multiple DEI projects, the DEI State Lead described
how it gave the state an advantage to blend resources. The availability of such resources helped
at the AJC level — in California, the state provides 50 percent of the OJT. One of the DRCs in
the state noted a few success stories of the clients going through OJT and on to permanent
employment. Furthermore, the strong partnership contributed to a more efficient blending and
braiding approach — the regional centers they worked with provided respite care for individuals
with disabilities and mental health, as well as training for ABLE accounts. As a result, the DRC
learned that establishing and maintaining partnerships was a key to blending and leveraging
resources. The DRC prepared a Disability Resource Guide, which listed the partner agencies
they worked with, contact information, and types of services they provided.
Kansas: Kansas Round 5 used Blending and Braiding Resources. One DRC stated that
he was “putting a greater emphasis on making sure that individuals are enrolled appropriately
for system-wide support. When I took over, there were many individuals that we were servicing
through WIOA that had not been co-enrolled in DEI.” Kansas also had strong collaborations
with partner agencies and DRCs who utilized partnerships in Blending and Braiding Resources.
DRCs often “pick up support services so Blending and Braiding of funding, Active Resource
Coordination, if you're talking about finding the best resources to meet the individuals’ needs,
that’s occurring on a regular basis.” In another Kansas WDA, co-enrollment was a strong factor
in Blending and Braiding Resources. According to one DRC, “We always co-enroll with DEI.
We also co-enroll with our adult program, our youth program, whatever they fit into. We try and
fit as many co-enrollments as we can, because it serves people better.”
Illinois: One AJC had good success with Blending and Braiding Resources. This AJC ran
into an issue serving youth, as their WIOA Youth program at the AJC could serve very limited
numbers of in-school youth; the WDA resolved this problem by having access to DEI funds in
the high school used to serve DEI-eligible students, while those who were out of school could be
served at the AJC using WIOA funds.
Massachusetts: The DEI grantee partnered with the Massachusetts Rehabilitation
Commission and the Department of Developmental Services to leverage resources in order to
provide extended services to JSWDs. These partners also helped pay for transportation and
provided job coaching for individuals once they started work. AJCs also leveraged DVR
resources to pay for transportation, but Blending and Braiding of other sources of funding was
limited. The grantee viewed Blending and Braiding Resources as a best practice, as one
respondent noted that it was very useful in getting a client everything they needed, such as
clothing or a driver’s license, because different agencies could pay for different services.
South Dakota: According to the State Lead, “We have tons of Blending and Braiding
examples. Someone might provide clothing for a client who needs them in order to work, or some
other needed resources. An example was a man that needed special clothing in an extra-large
size and that meant having them tailored. We split the bill with another agency.” They also split
costs for training and transportation. Blending and Braiding was deemed essential to successful
IRT implementation in South Dakota, and they indicated that it was very likely to continue after
the grant ended because agency partnerships had been so well established.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
19
Washington: DRCs focused on Blending and Braiding Resources. The DRC used WIOA
or Basic Food Employment and Training services for school enrollment and tuition. These Basic
Food funds paid for tuition, books, tools, test fees, etc. However, there seemed to be room for
more active Blending and Braiding at this site because a customer noted that they still had
trouble with their housing. As a DRC said, “The big challenge would be housing. That’s the
whole issue. DEI is not helping with housing at all.” In another WDA, the process for Blending
and Braiding seemed to be more stable. The DRC tried to utilize WIOA funds as much as
possible. Additionally, when a customer’s needs were not covered by WIOA, the DRC tried to
find additional resources for support. For instance, a customer noted that he received interview
clothes because those were not funded by WIOA, while the other customers were directed to
Goodwill for additional support.
Career Pathways (CP)
Round 6
Alaska – AK
Hawaii – HI
Iowa – IA
New York – NY
Washington – WA
Illinois: Illinois made CP a major part of its service delivery strategy. One AJC
developed close partnerships with its local community colleges and worked together to serve
JSWDs. The AJC and community colleges referred to each other, and the colleges worked to
find the best CP for each JSWD and offered job readiness courses and internships. The
community colleges also did outreach to high school students and attempted to get them involved
in employment and training. In addition, a local high school system made exposure to different
career options a central part of its curriculum. Students could select a career-focused track that
included an internship in a related career through coursework in high school and at a local
community college.
Kansas: Kansas leveraged its DEI funds to support longer participation in credentialing
and training programs that were more conducive to stackable credentials and career ladders.
According to the DEI State Lead, the DEI was instrumental in formalizing relationships with
postsecondary institutions and enhancing accommodation policies at local schools. Those who
participated acquired and maintained employment for at least 6 months at a higher average
hourly wage than projected.
Minnesota: A particularly robust employer engagement strategy was implemented in
three treatment WDAs focused on partnering with employers to provide CP-focused
credentialing, education, and apprenticeships in the career sectors of manufacturing; health care,
including certified nursing assistant (CNA), cardiopulmonary resuscitation, automated external
defibrillator, home health aides, personal care assistance, and medical technology; office and
administrative technology; precision sheet metal; retail management, customer service, and sales;
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
20
welding; food “ServSafe;” and commercial drivers. In addition, the involvement of JSWDs in
CP-focused training and employment was strengthened by opportunities to participate in peer
mentor meetings where jobseekers could both obtain and provide peer support to help offset any
challenges they encountered. Minnesota’s CP featured a close integration between acquiring the
credentials needed in a pathway and preparation for securing a job.
New York: WDAs developed health care pathways, including a certified nursing
certificate, while other participants worked toward a licensed practical nurse credential.
Manufacturing was another pathway that New York identified. DRCs engaged as advisors and
facilitators in crafting CP approaches and braiding resources with DVR, community colleges,
and community-based agencies. DRCs also leveraged resources from Pell grants.
South Dakota: South Dakota employment infrastructure was buttressed by its proximity
to the region’s technical school and leading adult education/supportive services provider. It also
took the lead in producing and disseminating the Building Pathways toolkit and CP visuals to
guide JSWDs with documentation on Job Shadowing for staff, clients, and employers. Statewide
trainings incorporated best practices around securing accommodations for individuals with
disabilities to complete a General Education Diploma (GED). Together, South Dakota, the Board
of Regents, and the South Dakota Department of Education built a Career Pathways development
web tool to be used by multiple partners.
“The Career Navigator goes out to the college and meets with the students we have and talks
with the instructor and student in terms of how they’re doing. And then when they start their job
search they meet with Career Navigators and do enhanced résumé writing to really tailor-make
it to that specific job/skill set.” — AJC Staff
Round 5 and Round 6 grantees appeared to scale-down the number of SDSs they used
relative to earlier rounds. In prior grants, grantees were required to select two SDSs. Logistical
challenges, however, may have led to more facile and familiar approaches to using SDSs. It is
striking that two Round 5 and Round 6 DEI grantees (Georgia and Hawaii) selected Customized
Employment but did not implement it in its entirety. Guideposts for Success, IRTs, Work-Based
Learning Opportunities, and Self-Employment were other previously favored SDSs. Components
of each of these SDSs were used by Round 5 and Round 6 on a less regular, informal basis by
individualizing the employment relationship through job carving; better understanding the
interests, strengths, and needs of each JSWD; and the use of Customized Employment’s
customization and negotiation strategies.
Social Dynamics researchers identified DRCs who
implemented IRTs, albeit in less coordinated ways than we saw in earlier rounds. DRCs
speculated that IRTs would continue to be implemented with less formality by Round 5 and
Round 6 grantees, as fewer incorporated the coordination of multiple services and leveraged
resources arranged by the DRC in support of the jobseeker (Exhibit 3).
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
21
Exhibit 3: ODEP-Suggested DEI Round 5 and Round 6 Service Delivery Strategies
Customized Employment (CE)
Round 5 (R5)
Round 6 (R6)
Not selected by any R5
grantees
Georgia – GA
Hawaii – HI
We obtained data that indicated an uncertain level of understanding and use of CE among
DEI grantees. However, to get a more complete picture of how CE elements were implemented
by DEI grantees even where not specifically chosen, we scanned interview and progress report
data and found that some of those grantees were in fact reporting the use of CE practices and
labeling person-centered employment planning as “Customized Employment.”
We found no instances of model fidelity that included CE implementation from
Discovery through to customized jobs. However, we did find adoption of some CE elements,
particularly Discovery, by a number of DEI grantees. Customized Support Teams are an integral
part of the CE model, and IRTs appear to offer a close approximation of CE team development.
Where IRTs occurred, it is reasonable to assume that they aligned with the principles, if not all
the practices associated with Customized Support Teams. DEI grantees also mentioned that
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
22
Discovery and CE aligned well with WIOA Title I and DVR services as enhancements to
person-centered assessments.
23
We view the CE-related elements that DEI grantees were implementing as a promising
practice because of the potential ways that they can be embedded within and reimbursed by DVR
and WIOA services systems. As staff used these methods, they appeared to reduce the need for
JSWDs to “impulse shop” for jobs.
Georgia and Hawaii stated in their grant applications that they intended to implement CE.
Other states were found to have implemented some strategies that aligned with or were based
upon CE. These are included below.
Georgia: The DEI State Lead and DRCs were trained in career mapping, a person-driven
assessment process that shares many elements with CE and Discovery.
“We do career mapping with everyone, which is a Discovery tool. We do that usually before we
enroll in Title 1. It’s before a DVR referral. The DRCs are good at that. After that, you do a
resource planning meeting where you identify needs.” — DRC
Georgia selected and adapted elements of CE. DRCs reported that there was improved
flexibility in providing services to customers based upon their needs and preferences, although it
appeared to be driven by an impulse to provide good customer service, not guided by use of a
formal CE methodology. One exception was due to an employer’s interest in adapting the
conditions of employment in order to hire and retain people with disabilities. However, these
actions were not guided by the formal components of CE, such as carving, negotiation, or job
creation, but bore a greater similarity to the use of flex work, which many employers offer.
“Under flexible work arrangements, employees can change the place where their work is done —
for example, by working from home or from a mobile location — or the hours when their work is
done, with different work start and end times, job sharing, or flexible or compressed workweek
schedules.” DEI DRC
Hawaii: Hawaii intended to focus on CE as one of its SDSs. Its T/TA partner, the
University of Hawaii, Center on Disability Services, provided numerous orientations to CE
principles and practices for DRCs. However, significant DRC turnover, due in part to a change in
the WIOA contractor; the part-time allocation of DRCs up until the contractor changed and
increased new DRCs’ time on the project; and changes in DEI leadership all contributed to
uneven CE knowledge translation and significantly curtailed its use. Consequently, little
evidence was found that Hawaii implemented CE as it proposed to do.
Kansas: Although Kansas did not select CE as a service implementation priority, the
state had received prior training on its implementation. CE appeared to be implemented with
mixed results. One DRC reported that they had success in adopting the CE Guided Group
Discovery technique and embedding it into their participant assessment and career development
23
Workforce Innovation and Opportunity Act information.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
23
process. However, some DRCs had trepidation about the match between Discovery and what
AJCs were required to provide because of the time it took and the potential effect that had on
achieving their performance outcomes.
“Discovery is kind of hard for us in the workforce center. Really, Discovery is not a concept
that’s going to be that easily useful for us.” — DRC
One DRC described a customized job that she had created, but the JSWD did not perform
well in their position. Instead of letting him go, the employer created a position for him at
another location in the business that was a better fit. This is an example of job customization
without labeling the process as CE.
“They were able to manipulate an active, open position and change the job description to fit his
ability and his needs.” — DRC
Minnesota: Minnesota also did not select CE as one of its focus areas. Minnesota
received CE training in the past, and some aspects of CE were implemented for some
individuals, although not across all DEI treatment WDAs.
“There has been pretty intensive and very helpful training on CE at all three sites over the years,
but CE is being implemented in components, piecemeal … with some efforts at job carving and
job sharing, Discovery, on-the-job training, Job Shadowing, … and flexible scheduling.” —
DRC
California: Although not focused on CE within their Round 5 DEI grant, some elements
associated with CE were used to strengthen a person-centered approach to employment
development.
“All three sites are doing a great job of highly individualized placement, really paying attention
to where the individual’s interest and abilities lie to finding them work that will set them on a
good retention and career path, but not so much carving out or customizing employment. The
DRC has done a lot with the Discovery assessment. That interview style of asking ‘why does this
interest you’ is important.” — DRC
New York: Although CE was not formally implemented, the methodology for working
with customers suggested a more comprehensive, informal assessment process that was
conceptually aligned with CE principles. The DRC would not call it “Discovery,” but talked at
length about getting to know each person more holistically rather than working through formal
assessments and standardized intake forms that Discovery is intended to improve upon.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
24
Guideposts for Success
Round 5 (R5)
Round 6 (R6)
Illinois – IL
Alaska – AK
Georgia – GA
We obtained data that indicated the level of understanding and use of Guideposts for
Success among Round 5 and Round 6 DEI youth-serving states. Only Illinois selected a focus on
youth with disabilities in Round 5, and only Alaska and Georgia focused upon this population
among Round 6 sites, although Hawaii shifted its Round 6 focus to an emphasis on youth in
2017. The following examples are representative of Guideposts implementation.
Alaska: Guideposts for Success appeared to be an important component of the Alaska
DEI when developing comprehensive service plans for youth. DEI leadership described how
Guideposts were central to the development of employment goals: “When our DRC1s and
DRC2s are working with our youth and they are creating their plans, they’re going to the
Guideposts. So those are their goals. They are creating their goals from their Guideposts, from
the six strategies.” DRC
Illinois: Although the grantee did not report using Guideposts for Success as an SDS, a
DEI-funded partner agency did make some use of the Guideposts. This partner received training
from a TA provider on implementing the Guideposts and used them as a guide when producing
materials to help serve students with disabilities. This partner reported that they viewed the
Guideposts as a “road map,” a useful tool that they wanted to continue to use after the grant
ended to help produce materials for students with and without disabilities.
Integrated Resource Teams (IRTs)
Round 5 (R5)
Round 6 (R6)
California – CA
Alaska – AK
Illinois – IL
Georgia – GA
Kansas – KS
Hawaii – HI
Massachusetts – MA
Iowa – IA
Minnesota – MN
New York – NY
South Dakota – SD
Washington – WA
All Round 5 and Round 6 states were required to implement IRTs. Consequently, IRTs,
along with Career Pathways, were among the most commonly identified and understood DEI
SDSs. We found IRTs to be among the most prevalent DEI SDSs, and they were described by
almost all grantees as being among the most useful SDSs in creating comprehensive, customer-
centered employment plans. DRCs were often able to describe the function and intent of an IRT
and how they were implementing them without specifically labeling the process as an IRT.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
25
Alaska: DRCs in Alaska viewed the IRT as a key SDS to leverage resources to ensure
that individuals entered a career path that was the right fit with all necessary supports available
for success. IRTs could involve multiple players. In one WDA, an IRT was comprised of the
Office of Public Advocacy, Juneau Alliance for Mental Health Inc., DVR, and Job-X. The DRC
reported that as a result of a strong IRT, a customer was accepted into the Carpentry Union, and
the IRT was still working on trying to obtain more Work-Based Learning experiences to build
credentials for job entry.
California: Based on strong partnerships developed by each local area, the DRCs were
more readily able to hold IRTs focused on the customer that engaged multiple stakeholders in
ways that they had not done before. DEI staff at all pilot sites stated that they used IRTs often
and found them useful.
“One of the best things about IRTs is that, if done correctly, which all three sites are [doing],
they are really coming together around one plan. And that’s the difference between consecutive
versus sequential services. Prior to the IRT, you’d end up with a jobseeker who might be working
with multiple agencies, but each agency could have a slightly different employment plan, which
pulls the jobseeker in a variety of different directions. With the IRT, having one plan really
creates a focus and it gets the individual, at least I’ve seen, employed more quickly because
everyone’s on the same page.” — DRC
Another feature of IRTs is relevant to youth JSWDs due to its value in engaging their
parents:
“Bringing the parent on to the IRT and having them see the service providers working together
at the direction of their adult child has really been helpful to negate that fear that the parents
have. That’s been a great benefit with the IRT.” — DRC
Illinois: Respondents at one AJC pilot site indicated that they made use of IRTs
consistently with almost all their customers, and both the AJC and its major partners reported the
IRTs as being a useful tool to get customers their needed resources. However, staff at another
site reported not finding IRTs necessary for all customers, although they were useful in engaging
and working with VR.
Kansas: All WDAs actively used IRTs. One DRC mentioned that scheduling and
interacting with partner agencies on behalf of the customer was a daily task. The IRT often
involved partner agencies, VR, business services, and WIOA case managers. Kansas emphasized
the importance of sitting down at the table with WIOA partners to “figure out what’s going on.
Maybe you have an employer I didn’t think about, maybe you’ve got an idea that I didn’t think
about, and vice versa.” At the other location, the DRC was also using IRTs frequently to bring
multiple perspectives focusing on the needs of the customers.
Massachusetts: DRCs managed and coordinated IRTs by building partnerships and
bringing resources to the table. One WDA seemed to have a relatively strong capacity to manage
and coordinate IRTs by leveraging various partnerships necessary to providing a holistic
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
26
approach to serving customers. Although a DRC at one pilot site discussed how they formed
IRTs through leveraging resources, they noted that formal IRTs were rare. While the essential
IRT functions were in place in Massachusetts, the IRT as implemented did not adhere to all
components of the practice.
New York: A DRC in New York noted that they used IRTs on regular basis, especially
when the client needed extensive assistance. In one of the pilots, the DRC noted that the
employment counselors were often present during the IRT with the client. She also noted that the
AJC tried to train staff to spot the individuals who might need an IRT at an early stage to prevent
the clients from going through unemployment for a prolonged period.
South Dakota: South Dakota implemented IRTs and described them as core to their
services on behalf of JSWDs. But, in addition to individual, customer-centered IRTs, they also
developed “agency IRTs,” which were held with the WIOA core programs, United Sioux Tribes
who operated a Native American program, and a local transit agency and can be considered a
promising practice. “We used IRTs in all three sites … we also embraced this strategy and did
trainings across the state with core and other agencies. The IRT model is going well as a
management technique. We had a DVR supervisor in Rapid City ask, ‘Why don’t we do this for
agencies and collaborate with community groups, too?’” While IRTs can take time and be
intensive based upon a person’s circumstances, the DEI State Lead reported that because AJC
staff were achieving better employment outcomes with IRTs than they had in the past, they
believed that it was worth the effort.
Washington: Both WDAs in Washington offered that the IRT was a crucial part of their
DEI. Similar to data obtained from other states, IRT-like functions were being implemented
without labeling them as IRTs. As one DRC stated, “we don’t use the term but we do that all the
time because it’s an interdisciplinary team. We work with providers and a community rehab
provider for job placement, case managers, families, etc.” Also, customers noted in the focus
groups that the DRC was helpful in finding the right services and connecting them to the right
personnel using a process that included identifying their needs, employment history, and
experience, and eventually forming an IRT with partner agencies that provided their particularly
appropriate services.
Work-Based Learning (WBL) Opportunities
Round 5 (R5)
Round 6 (R6)
California – CA
Alaska – AK
Illinois – IL
Georgia – GA
Kansas – KS
Hawaii – HI
Minnesota – MN
Iowa – IA
South Dakota – SD
New York – NY
Washington – WA
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
27
Work-Based Learning was most often integrated with other DEI SDSs, including the
experiential aspects of goal-setting within CE, Discovery, and/or CP. The states below offer a
glimpse into how they integrated WBL into their overall menu of services.
Alaska: The partnership between DEI, VR, and other provider agency partners to
implement summer employment opportunities for youth was an example of integrating WBL
with career exploration, goal-setting, and soft skills development. With DEI, DVR, and other
braided resources, foster care youth from around the state were flown into a “transition to
employment” experience in Anchorage, where they focused on job goal development, workplace
environment soft skills, actual work experiences, and life-skills training that provided real-
time/real-life employment experiences for youth with disabilities.
Massachusetts: One individual worked at a manufacturing company, but she recently
started working in an internship at a local hospital that was planning to offer her a full-time job at
the end of the internship. The local VR agency also partnered with a DRC to offer OJT to DEI
clients as a component of assisting JSWDs to obtain and advance in their careers.
Self-Employment
Round 5 (R5) Round 6 (R6)
Not selected by any R5
grantees
Not selected by any R6
grantees
Self-Employment was not selected as an area of focus by any Round 5 or 6 grantees.
However, it did tend to emerge as a topic of discussion during interviews, particularly in states
where DEI grantees served rural areas and where a relative dearth of jobs existed, but DRCs
were not specific about how they assisted prospective entrepreneurs with disabilities. Overall,
other than Alaska, little evidence supports the use of Self-Employment as a DEI SDS by Round
5 and Round 6 grantees.
Alaska: DEI staff and partners regarded Self-Employment as a viable employment
strategy for JSWDs in areas where few opportunities for wage employment existed. DEI
leadership and their partners, particularly DVR, helped JSWDs access Self-Employment training
and business development resources by leveraging support among their partners. In Anchorage,
they developed the Alaska Business Week magazine and dozens of employers participated as
mentors. Youth were brought in from across the state for a weeklong business development
camp and they began from the ground up to start a business.
“Self-Employment? We actually do have it because there is a population of students who live
at home and support their family. That is something unique about our region. We have
business and marketing Entrepreneurship. We bring in artists, and students learn from
resident experts, and we blend that training with business and marketing so they can market
their arts or carving.” — DEI Partner
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
28
B. Findings from the Social Dynamics Systems Change Coding Rubric
Social Dynamics used the same systems change analysis for the DEI Round 1 through
Round 4 and Round 5 and Round 6 systems change analyses. Social Dynamics examined how
each grantee used SDSs, the level of DEI stakeholder engagement, and the extent to which that
engagement linked JSWDs to employment-related services and skills. Qualitative data analyses
were central to the Round 5 and Round 6 evaluation, as they were in previous rounds. The
implementation evaluation provided information on the roles, responsibilities, and relationships
of grant-funded positions (e.g., DEI State Lead, DRC) and AJC and WIOA-mandated partners
that participated. We conducted site visits and follow-up telephone interviews on an annual
basis. We also prepared site visit agendas, taking into consideration reading materials produced
by each grantee. Conference calls were held with stakeholders to learn their perspective on
progress made. To ensure confidentiality, we did not attribute observations and comments to
specific individuals nor did we reference their names, titles, or organizational affiliations in this
report.
The evaluation team also trained DEI stakeholders from all 12 grantees on the
requirements of the evaluation, with the principle challenge being the training and retraining of
DEI State Leads and DRCs on the project’s reporting requirements and evaluation. Social
Dynamics created state-specific “binders” for each grantee and WDA. The binders included fact
sheets outlining basic grant information, including participating WDAs, grant type (adult/youth),
selection of SDSs, names, addresses, and maps of all DEI sites; confidentiality and informed
consent protocols; state annual WIOA reports; WDA newsletters; a glossary of federal and state-
specific terms and acronyms; and site visit questions and probes.
All site visits began with an orientation for WDA staff. We conducted interviews
annually with 65 individual respondents across DEI Round 5 and Round 6, including DEI State
Leads, DRCs, AJC managers, business services staff, Local Veterans’ Employment
Representatives, Disabled Veterans’ Outreach Program staff, youth workforce programs and
services staff, and WIOA-mandated partners. We conducted focus groups with JSWDs in all 12
WDAs. We used a purposive sampling methodology to select respondents while on site at each
WDA. Site visits were conducted by teams of three trained researchers.
24
The research team
relied on the opinions of “primary respondents,” which included DEI State Leads and DRCs who
were responsible for the implementation of the DEI in each WDA.
We used qualitative analysis to analyze the data from interviews and focus groups. It
incorporates “triangulation,” which involves the use of multiple data sources to produce a
thorough understanding of project implementation. This component of the evaluation focused on
modifications in the way the DEI State Leads and DRCs functioned, how they developed their
skill sets, how SDSs were implemented, and how their use evolved over time. WIOA-mandated
and non-mandated partners, communications with employers, and collection of information on
the operation of TTW were other key topics of inquiry.
24
Researchers also collected “artifacts” such as grantee quarterly reports and written descriptions of procedures or
activities observed at WDAs.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
29
C. Developing Domains and Indicators for Systems Change
DEI systems change represents an adjustment in the way WDAs coordinate and allocate
resources and sustain promising practices after the grant period. Within the context of the DEI,
systems change is a corollary of numerous federal, state, and local initiatives that address the
needs of individuals who are members of disadvantaged and/or disenfranchised groups. Systems
change is needed when improvements in the social, civic, and/or economic circumstances across
WDAs are made more difficult because of a combination of prevailing attitudes, knowledge,
skills, and/or resources that inhibit systems change.
To better understand the dynamics of systems change, Social Dynamics developed the
Systems Change Coding Scheme (SCCS), which was revised in 2015. The SCCS measures the
implementation of DEI requirements, including the implementation of SDSs and the capacity of
each WDA to offer JSWDs employment and related services, including CP training; remedial
classes that relate to the requirements of each Career Pathway were also available to JSWDs.
Definitions of what constitutes systems change vary. However, for our purposes, we
referenced ODEP’s Criteria for Performance Excellence in Employment First State Systems
Change & Provider Transformation.
25
Though the focus of the Criteria is more related to the
developmental disability and VR systems, its relevance in addressing changes to complex
systems for individuals with disabilities correlates with the goal of the DEI. The facilitators of
cross-systems change include the development or refinement of promising practices,
advancements in the legal/policy landscape as they relate to accessibility, accommodations for
JSWDs, and strategies that maximize efficiencies through goal alignment, resource coordination,
and improvements in program performance.
The SCCS is a conceptual framework designed by Social Dynamics. It includes eight
domains and indicators that operationalize systems change as it relates to individual WDAs. The
coding methodology is enhanced by information from interviews and focus groups, where
nuances can be observed that lend greater insight into the challenges inherent in any type of
innovative, large-scale initiative. For example, although an AJC might have had assistive
technology (AT) equipment, observing that it was not easily accessible, or that staff who knew
how to use it were not present, or that staff had limited availability or knowledge of the
technology could corroborate why a DRC may have had difficulties providing universally
accessible services to JSWDs. Therefore, the objective is to achieve reliability and validity in
analyzing the impact of systems change by analyzing data from interviews and observations of
WDA operations using inter-rater reliability with three raters.
The SCCS coding methodology includes a four-point Program Implementation Rubric on
the y-axis and a Program Maturation Rubric on the x-axis. Reliability is achieved when three
coders analyze the same data and produce quantifiably consistent results. To achieve reliable
25
U.S. Department of Labor, Office of Disability Employment Policy & LEAD Center. (n.d.). Criteria for
performance excellence in Employment First state systems change & provider transformation. Retrieved from
http://www.leadcenter.org/system/files/resource/downloadable_version/Employment_First_Technical_Brief__3_0.
pdf
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
30
coding, we use mutually exclusive and clearly defined coding categories. To achieve validity, we
combine discrete items into broader, clearly defined constructs. Our coding methodology focuses
on developing a reliable and valid assessment of implementation and maturation or change. This
process includes decision rules, a codebook, and an internal review to confirm that the data
collection protocol, field experience, and observations are reliable.
System Change Indicators
The Program Implementation Rubric is a four-level ordinal scale used to determine the extent to
which grantees monitor and implement the requirements of the DEI grant (y-axis).
1
2
3
4
No evidence that this
indicator is being
met
Some evidence that an
effort is underway to
implement this
indicator
Evidence that
implementation of this
indicator is partially in
place
Evidence that this
indicator has been
fully implemented
The Program Maturation Rubric is a four-level ordinal scale designed to determine the extent to
which DEI grant activities achieves sustainability of DEI practices (x-axis):
Start-Up (1)
Implementation (2)
Operational (3)
Sustainable (4)
An element in the
earliest planning
phases, not yet
formally
implemented
An element that has been
initially implemented, but
not yet formalized beyond
a trial or experimental
phase
An element that has
been consistently
implemented, often with
involvement of staff
beyond the DRC
An element that will/
has persisted in the
operations of the AJC/
WDA/state beyond the
grant period
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
31
System Change
Domains
Definitions Key Areas of Measurement
1. Capacity to
achieve
integrated
supported
employment
for JSWDs
Development of functional IRTs and
resources and workforce development
systems that are inclusive and change
perceptions, attitudes, and understanding
of the issues related to disability and
employment. In addition, improvements
in access to different types of resources
and SDSs, designated staff with expertise
supporting JSWDs, policies that facilitate
access to services and employment and
access to WIOA services, an experienced
DEI State Lead and DRCs tasked with
managing/coordinating services.
DEI State Lead involvement
in systems change
DRC involvement in systems
change & jobseeker support
EN/TTW activity
Knowledge of SDSs
A plan for SDS
implementation
2. Coordination
& integration
of services
Coordination of employment services for
JSWDs. Integrated workforce
development systems that provide SDSs
and related support services. Partnerships
and Collaborations that facilitate cross-
agency training, interagency partnerships,
shared resources, employer cooperation
and engagement, and innovative
approaches to Blending and Braiding
Resources.
Partnerships & Collaboration
Blending & Braiding
Resources
IRTs
Shared resources
Employer outreach
Asset Development training
Benefits Planning
3. Customer
choice
Customization of products and services to
each JSWD as they make their own
decisions about training and the
employment process. Customer
involvement is part of the design of
products and services, the use of financial
assistance (e.g., SSA TTW, Medicaid,
Medicare, and VR services), and targeted
training that focuses on the individual’s
requirements and needs.
Customer choice
Services supported by system
Existing subsidies/benefits
used efficiently
Training availability
Financial literacy assistance
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
32
System Change
Domains
Definitions Key Areas of Measurement
4. Employer
support &
employer
partnerships
Employers support the recruitment and
hiring of JSWDs. WDA provides support
for employers in forums to discuss their
hiring needs and job candidate pool,
development of position announcements,
pay scales for employment opportunities,
and apprenticeship opportunities and
other forms of training such as OJT and
Career Pathways.
Facilitate recruitment & hiring
of JSWDs
Opportunities to discuss
hiring needs
Recruitment of SSA
beneficiaries
Support in developing
position announcements
Opportunities for enrollment
in CP, apprenticeships, OJT,
& other supportive
employment opportunities
5. Use of
enhancements
to existing
SDSs
Identifying, developing, and/or adapting
innovative practices and approaches to the
use of IRTs, Customized Employment,
Self-Employment, Guideposts for
Success, Asset Development, and
Partnerships and Collaborations.
IRTs, CE, Self-Employment,
Guideposts for Success,
Asset Development, &
Partnerships &
Collaborations with a DRC
or Employment Specialist
Using SDSs to facilitate the
employment process
6. Dissemination
of effective
practices &
outreach to
disability &
employer
communities
Identifying, developing, and/or adapting
practices to the use of IRTs, Customized
Employment, Self-Employment,
Guideposts for Success, Asset
Development, and Partnerships and
Collaborations.
Knowledge dissemination and transfer of
best practices to employers and WDA
partners through webinars and other
formalized methods of communication to
JSWDs and employers.
IRTs, Integrated Resources,
CE, Self-Employment,
Guideposts for Success,
Asset Development,
Partnerships &
Collaborations are available
to JSWDs with a DRC or
Employment Specialist
using other SDSs to
facilitate the employment
process
Communication strategies
such as webinars, issue
briefs, & in-person forums
targeting key audiences:
adults with disabilities,
youth with disabilities,
federal & state agency
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
33
System Change
Domains
Definitions Key Areas of Measurement
partners, & support service
providers
7. Universal
design for
learning
Provide multiple means of representation.
Offer ways of customizing how
information is used. Make learning more
helpful with multiple representations of
course content.
Provide multiple means for
representation, development,
& dissemination of effective
practices & options for self-
regulation
8. Sustainability
Sustainability achieved through system
members developing access to alternative
sources of funding through interagency
partnerships, grants, and legislation.
Ensuring that TTW ENs and DEI SDSs
continue after the grant period. Policy
development and policy change that leads
to the sustainability of DEI strategies and
activities.
Evidence of plans to sustain
DEI strategies & activities:
formal agreements,
Memoranda of
Understanding (MOUs),
identified sources of
funding, new grants, &
legislation
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
34
Program implementation and program maturation are linked to the systems change
indicators. They provide information on the start-up, implementation, operation, and
sustainability of each DEI grantee.
Program Implementation and Program Maturation
Program Maturation
Rubric
x-axis
Program Implementation Rubric
y-axis
Not
Implemented
(1)
Exploring
(2)
Partially
Implemented
(3)
Fully
Implemented
(4)
No evidence
that this
indicator is
being met
Some evidence
that an effort is
underway to
implement this
indicator
Evidence that
implementation
of this indicator is
partially in place
Evidence that this
indicator has been
fully implemented
(1) Start-Up
• Grant-funded positions (e.g., DEI State Lead & DRCs) are in place
• DEI State Leads are trained to administer the grant & oversee its
implementation
• DRCs are trained to monitor the implementation of the program,
provide case management support for DEI participants, & engage in
systems change activities throughout the grant period
• SDSs are selected
• Active outreach to WDAs begins
• Information on the program is distributed to stakeholders & JSWDs
• Coordination & integration of services
•
Participant customer choice
(2) Implementation
• Grantees engage in a strategic process to define goals & objectives
• SDSs are implemented with fidelity
• DEI requirements are implemented with fidelity
• DEI participants receive training, counseling, &/or job placement
support
• Capacity to achieve positive employment outcomes
• Employer engagement
• Employer support & partnerships
• Dissemination of effective practices & outreach to the disability
community
(3) Operational
• DRCs collaborate with WDA staff to implement all program
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
35
requirements & have a clearly defined agenda that engages the
employer community, JSWDs, & WDA personnel & stakeholders
• WDA recruits SSA beneficiaries through TTW
• WDA TTW milestone payments are received
• Implementation fidelity data is used to determine why program
outcomes are being/not being met
• Coordination & integration of services, customer choice, &
employer support
• Dissemination of effective practices & outreach to the disability
communities
• Employer & JSWD outreach
•
Development of new or enhancement of existing practices
(4) Sustainable
• WDA personnel & local area agencies & support services have
developed partnerships & collaborations that improve access to
employment & training services for JSWDs
• WDA personnel have created impactful relationships that have
increased access to key supports services & employment
• WDA has a realistic sustainability plan in place
• WDA has resolved challenges that hinder progress to implementation &
sustainability
• Promising practices are sustained after the grant period
• WDA outcome payments are received
• Employer partnerships & development of new or enhanced
strategies
• Employer outreach to JSWDs
• SDSs continue after the grant period
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
36
VI. Grant Implementation Round 5
A. State: California; Focus Area: Adults with Disabilities
Stated Goals and Objectives i.
The state’s Workforce Services Branch in the Labor and Workforce Development
Agency of the Employment Development Department administered the Round 5 DEI grant to
achieve systemic change and expand the capacity of WDAs to serve customers with disabilities.
This included significantly increasing training opportunities for JSWDs as well as for staff and
employers and continuing efforts toward serving adults with disabilities with universal AJC
accessibility across multiple dimensions. California planned to achieve the following objectives:
• Expand the capacity of AJCs to use core and WIOA services as a part of IRTs that serve
people with disabilities;
• Continue to develop a system of state-centric T/TA to build capacity and expand the use
of DEI service delivery strategies;
• Increase WDA participation in TTW and Partnership Plus;
• Provide access to Career Pathways for individuals with disabilities; and
• Continue to demonstrate that AJCs can partially fund disability programs if they
complete the suitability determination application and become an EN.
California Round 5 included three WDAs that demonstrated a high capacity to serve
JSWDs due in part to their experience with the Disability Program Navigator project prior to the
DEI and the continuity of staff at the state and WDA levels. Prior to joining the DEI, DRCs often
previously worked in related occupations, including job coaching, mental health counseling, and
other occupations supporting individuals with disabilities. Some DRCs also had specific areas of
expertise, including working with veterans and ex-offenders with disabilities.
Each WDA designated one or more DRCs to provide direct services and case
management. California reported that they saw a lot of back injuries and carpal tunnel syndrome
among JSWDs as well as “a fair amount of individuals on the autism spectrum. Other JSWDs
had problems with workplace appropriateness and mental health issues.”
An important issue for the grantee was self-disclosure, as DRCs “were required to
explain to JSWDs that disclosing a disability is about finding opportunities and
accommodations” and not about restricting access to services for individuals. DRCs created a
worksheet to help JSWDs make decisions about whether to disclose a disability or not.
State leadership created a position that served under the DEI State Lead that we referred
to as a “State-Level Supervisory DRC.”
26
This individual had responsibilities beyond the
administrative aspects of the DEI State Lead role and engaged in providing WDAs with support
26
Social Dynamics created the term “State-Level Supervisory DRC” to distinguish between the DEI State Lead and
the former’s subordinate leadership position.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
37
for program implementation, strategic planning, and training of DRCs on a quarterly basis. Two
further key components of California’s Round 2 grant to support DEI staff that continued into
Round 5 were the peer advisory team and the Traveling DRC. The peer advisory team planned
and conducted “mentoring meetings” with DRCs and partners to discuss the implementation of
SDSs and other DEI requirements. The Traveling DRC provided oversight for each treatment
WDA, offering “interactive trainings and practical information based on successful models
currently operating statewide.” One large WDA grantee created two DRC positions, one for
direct service and one to coordinate and build capacity. In the smaller WDAs, a single DRC
performed both of these roles.
DEI Service Delivery Strategies ii.
IRTs and Asset Development were the two SDSs of emphasis for California. IRTs
strengthened partnerships and leveraged resources among a variety of partners. Asset
Development focused on financial literacy combined with benefits counseling that facilitated
JSWDs to assess the impact of full-time work on their benefits. California also provided a
Workforce Development Manager who oversaw Community Services Block Grants and helped
strengthen integrated services for an array of practical challenges to employment such as eviction
notices, transportation, housing, shelter, and child care services. JSWDs were also able to access
emergency services, OJT and opportunities, AARP employment and related services, and Back
to Work 50+. The latter links individuals over 50 with information, support, training, and
employer access needed to regain employment and advance in an occupation.
DEI Round 5 led to an increase in the use of IRTs; all JSWDS participating in the DEI
also participated in an IRT. IRTs were implemented in a number of different ways, from group
meetings to telephone calls with WIOA-mandated partners. All JSWDs were co-enrolled in
WIOA and VR, and blending and braiding were also common practices, as was providing
resources for access to transportation, child care, and housing. DRCs also provided training in
résumé writing and job interviewing and information about TTW. Some JSWDs also received
job coaching and case management services both before and after employment was secured.
TTW was a priority for California and reportedly implemented with much success in
DEI, especially in the two largest WDAs. DRCs focused on workforce development issues and
collaborated with the Department of Rehabilitation (DOR) to support Partnership Plus and access
to training and employment using TTW. The strengthened relationship between the AJCs and
DOR helped the DEI gain access to SSA beneficiaries and collaborate to serve JSWDs enrolled
in TTW. Through a partnership with DOR, a “menu” was made available so that each JSWD
could select SDSs that would be most advantageous to them. The collocation of DOR and DEI in
most instances within AJCs and the establishment of all three DEI WDAs as ENs strengthened
the project’s support for TTW. Nonetheless, state agency personnel described challenges in
working with SSA that may have affected enrollment, utilization of workforce incentives, and
the fidelity of their implementation, including difficulties in accessing information and support.
Another challenge was the reluctance of some JSWDs to seek full-time employment due
to fears that they would lose their benefits, including SSI/SSDI, health, Medicare, Medicaid, and
Temporary Assistance for Needy Families (TANF). To address trepidations about working,
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
38
JSWDs were provided pre-employment workshops. These were reported to get JSWDs into “job
mode: getting up, coming to the AJC, and getting their brains running again.” They were also
used as a tool to figure out whether JSWDs were ready for work: “If JSWDs cannot come every
day for a workshop or they find that overwhelming, they need to talk about why it was
overwhelming.”
“There are not that many differences between SSA beneficiaries and JSWDs who are not eligible
for TTW. It is always about being out of work for a long time and wanting to get back to work.”
— DRC
DRCs either became CWICs or consulted with CWICs to provide information on
Benefits Planning to clarify how employment income may affect a jobseeker’s benefits.
California instituted an online Work Incentive Coordinator certification, Disability 101, that
covered the basics of how returning to work would affect benefits. JSWDs were also encouraged
to seek full-time employment that included benefits. WDAs used Ticket revenue to fund staff
training, staff wages, and job coaches. Job coaches were available to individuals who needed
short-term, emergency resources to earn a certificate that could lead to training and long-term
employment. AJCs also prepared participant maps that focused on goal development and access
to technology to help individuals who would ordinarily “walk away feeling embarrassed or
illiterate.”
“Individuals who have been out of work for a long time are hesitant about losing benefits. There
is a long list to get child care benefits and there is fear that they will go back to work and lose
their benefits before being able to afford to be independent. There was a lot of concern from
customers about losing health benefits. In their mind, they are at risk of losing them. They know
their overall existence is dependent on benefits. Therefore, the WDAs encourage customers to
look at full-time jobs with benefits. They may need to start in an entry-level position, but we want
to get them comfortable working full-time.” — DRC
WDAs also engaged in the identification of job candidates, development of résumés,
matching jobseeker skills to employer needs, and practice interviewing. Helping to strengthen
JSWDs’ support systems was also an important part of California’s Round 5 grant. JSWDs were
encouraged to engage with family, friends, and professionals who might support their job search
activities and training and employment aspirations. JSWDs were encouraged to communicate
with individuals who were already employed to help them access community resources. One
DRC mentioned that it “takes a village” to ensure that JSWDs have the resources, skills, and
motivation to become employed. This approach was integrated into their CP services (see more
below in the section on Career Pathways).
Implementation Summary iii.
The California DEI sponsored and organized quarterly meetings for T/TA, capacity-
building, and mentoring of Employment Specialists. The grant was overseen by the DEI State
Lead and DRCs. The quarterly meetings included WIOA partners, subject matter experts, DEI
staff, and WDAs that were prior DEI grantees. Topics included DEI SDSs, Partnerships and
Collaborations, organizational development, monitoring TTW implementation, and general
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
39
upgrading of the skills and knowledge of DRCs and Employment Specialists. The DEI State
Lead also coordinated regional meetings for skills development through group trainings. The
DRCs were full-time on DEI and spent their time on both building capacity of AJC Employment
Specialists to serve JSWDs and providing case management and direct services to jobseekers. In
the largest WDA, there were two DRCs, one for capacity-building and one for direct services.
To enhance their knowledge, DRCs received T/TA from the National Disability Institute
(NDI) and the Traveling DRCs. Trainings targeted areas of local need and included topics such
as DEI program implementation, disability etiquette, and JSWD counseling for Employment
Specialists. DOR provided a staff member who traveled to each WDA to provide support to
DOR customers. Other trainings were provided by Americans with Disabilities Act (ADA) staff
on a yearly basis covering employee rights, self-disclosure, hiring individuals with disabilities,
access to job accommodations, and enrollment in SSA and related programs.
JSWDs felt that there should be increased access to the Limited Examination and
Appointment Program for individuals who are members of AARP, as there was reportedly “a
significant amount of discrimination against people who have disabilities.” The program
facilitates the recruitment and hiring of individuals with disabilities. It also provides alternative
ways to demonstrate qualifications for employment rather than through aptitude tests. A
workforce development manager for the Limited Examination and Appointment Program
oversaw Community Services Block Grants that integrated services into the AJC to deal with
challenging situations such as eviction notices, transportation to a shelter, and access to child
care.
During the implementation of Round 5, each participating WDA was enhanced with
upgraded physical and programmatic accessibility such as automated doors, access to
transportation, disability-friendly cubicles, parking locations, and AT. These enhancements in
the accessibility of AJCs and access to external resources were designed to increase participation
by JSWDs in WIOA Title 1 programs during Round 5.
California’s Round 5 DEI focused on multidimensional, universal AJC accessibility to
serve JSWDs with a collaborative approach that included the state’s Workforce Services Branch
in the Employment Development Department. To achieve systemic change and increases in the
availability of training for in-demand occupations, California expanded the capacity of WDAs to
provide JSWDs opportunities for education, access to services, soft skills training, CP
instruction, and collaborations and partnerships with employers and the WDAs. California also
developed T/TA for DRCs and Employment Specialists and expanded the use of DEI SDSs such
as IRTs and Asset Development, TTW, Partnership Plus, T/TA and skill-based interest
assessments, and documentation of employment experience and eligibility for SSA, Medicare,
Medicaid, and TANF support services.
Partnerships and Collaborations iv.
Integration and collaboration across WDAs implementing the DEI included state-level
interagency cooperation, particularly between the Labor and Workforce Development Agency
and DOR. These partnerships enabled consistent and productive messaging, resource leveraging,
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
40
and access to SSA beneficiaries and non-beneficiaries. California also created new partnerships
with local agencies. Agencies partnered with DEI by participating in T/TA, statewide
conferences, and webinars and assisting in career development and job placements for JSWDs.
Other collaborations included those developed with the Department of Developmental
Services and TANF, California Committee on Employment of People with Disabilities,
Veterans’ Employment and Training Service, California Health Incentives Improvement Project,
and community-based organizations and nonprofit service providers. Across the state, more than
two dozen agencies were identified as partners of the DEI.
The DEI State Lead partnered with DOR to generate informational material encouraging
businesses in developing alternative methods for JSWDs to apply for jobs since the California
Labor and Workforce Development Agency had concerns about the use of online job application
processes for JSWDs. Career Pathways road maps developed in conjunction with employers
served to both visualize employment journeys and motivate JSWDs to access employment and
training opportunities. Continuous supports from DRCs, AJC colleagues, and job coaches, such
as motivational interviewing, were used to “encourage aspirational discovery, self-regulation,
attainable goal-setting, and persistence.” WDAs tended to be customer-centric, meaning that the
employment process was individualized and driven by each individual “that walks in the door.”
Participation in the IRT process also was a source of client empowerment and choice. In addition
to disability-friendly modifications to AJCs, community-based partners helped implement self-
paced and multi-modal assessments, including use of AT.
California sought to educate employers about reasonable accommodations and the use of
Customized Employment and training through various means. DEI worked with employers to
develop strategies for employer engagement, which included collaborating with economic
development agencies and local Chambers of Commerce and educating employers on topics such
as tax credits, accommodations, and events such as job fairs and employer forums to increase
employer awareness and employment of people with disabilities. California also engaged with
the employer community through OJT sponsored by the Labor and Workforce Development
Agency and supported by several employers that played a key role in the DEI Employer
Engagement Community of Practice and through establishing partnerships with WIOA-
mandated partners. Employer partnerships included entertainment companies that hired people
with disabilities.
Employer engagement and training was also provided through the state’s Medicaid
Infrastructure Grant at San Diego State University and online employer human resources
trainings. California has an “employer services” department that reached out to employers to
provide training in OJT, CP, CE, and/or Entrepreneurship (Self-Employment). Through the
DEA, California connected with senior living facilities to hire people with disabilities into staff
positions. The Northern California Business Advisory Council included service providers and
employers that hired people with disabilities and regularly advocated hiring JSWDs by
upgrading their qualifications to meet the needs of employers and overcoming the fear that many
JSWDs have about losing SSA benefits. They also provided access to accommodations to
address employer concerns about hiring JSWDs and potentially insufficient accommodations to
support long-term placements beyond temporary work experiences and OJT.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
41
Career Pathways v.
California was one of the early implementers of Career Pathways training; it has
extensive experience with providing similar services for more than 15 years. Their services
include “industry-linked programs” and supports for JSWDs to help them gain access to in-
demand occupations that could lead to economic self-sufficiency. California’s long-standing
effort to enroll individuals in CP training included both educational and employment-related
training in partnership with local businesses and across a wide range of industries. California CP
training was provided at no charge to DEI participants, as were work experiences, OJT, and
internships. However, JSWDs and DRCs commented that individuals often waived CP training
in order to more quickly enter employment. According to one DRC, JSWDs tended to achieve
high employment placement rates by quickly accessing a job, rather than taking the time needed
to go through vocational training and OJT, even though better training could mean that the job is
more likely to be sustained.
“Training is a harder sell because people do not see the benefit to putting income off.” — DRC
Although JSWDs were interested in immediate employment with only limited training,
there was some basic education that was needed to get them on the path to employment. With
OJT, “we have the placement immediately, the client is getting a check, and AJCs can fill a
training gap during that period. Employers are committed because it will support their
workforce development needs. Through OJT, DEI participants have a $2-per-hour greater wage
than clients in general employment.”
Outreach and Dissemination vi.
California conducted outreach, recruitment, and referrals for individuals with disabilities
through its partner agencies, including DOR, the California Department of Developmental
Services, veterans’ agencies, TANF, and a wide range of community-based agencies. California
also conducted outreach and recruitment in support of “targeted populations” such as individuals
with complex employment situations. The state created linkages within WDAs to interact with
local agencies that provided support services and coordinated outreach efforts, including referrals
to and from partner agencies, disseminating through radio and newspaper ads, sending “email
blasts,” posting on social media/websites, and presenting to local community-based agencies,
religious organizations, schools, and libraries. California increased the number of JSWDs
entering CP programs and implementing Individual Employment Plans, OJT, and CE
opportunities. Quarterly meetings were a critical component in training and disseminating
information that all current and former DEI grantees (and some other non-DEI WDAs) were
funded to attend by the DEA state program. Many agencies and service providers were also
invited to participate in these quarterly meetings. The Traveling DRC also played an important
role in disseminating effective practices across the state.
Promising Practices: “Pathways for Success” vii.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
42
The Traveling DRC provided 1-day trainings for AJC Employment Specialists in
separate modules “to bring best practices and perspectives discovered through California’s DEI
and Disability Employment Accelerator (DEA) to improve service delivery to individuals with
disabilities in each treatment WDA.”
“Traveling DRC trainings are intended for various levels of AJC staff, management, and
relevant collocated partners who have a stake in improving services to individuals with
disabilities in WIOA-funded Job Centers. We would love to see some front desk staff, case
managers, and anyone else who might interact with individuals with disabilities, but [we] are
interested also in having supervisors, managers, and the like who can put action to some of the
discussions that will occur and understand what partnership and collaboration looks like on
paper.” — Traveling DRC
Round 5 Traveling DRC modules included the following:
• One System for All and All for One — Serving People with Disabilities — All of Us Can,
But Do We?
• To Politeness and Beyond! — Disability Awareness, Sensitivity and Etiquette
• Here an Acronym, There an Acronym — What is the DEI, DEA and IRT, and what do
they have to do with PWD and WIOA?
• We Need a Driver — The Disability Resource Coordinator (DRC) Defined
• Workforce, WIOA and Compliance Oh my! — Coordinated and Complementary
Employment and Training Services
• Creating an Environment of Trust and Building Rapport — Disclosure 101
• If We Build It…They Will Come — Examining Customer Flow and Improving Service
Delivery
• Creating an Organization Nest Egg — Become an Employment Network (EN) and Build
Discretionary Funds
Rationale for Its Implementation
The Traveling DRC, in combination with the peer advisory team and its extant partners,
addressed the need for consistent and relevant T/TA across a wide range of topics and across
partners that would continue to provide services to JSWDs after the grant period. This position
was advantageous to the adoption and implementation of promising practices for DRCs and
Employment Specialists to implement training with fidelity.
Why the Practice Could Be Considered Promising
The Traveling DRC provided T/TA for DRCs and Employment Specialists in an effort to
build staff expertise and capacity to implement disability and employment-focused services. It
also provided opportunities for engagement with other agencies, such as DOR, in the use of DEI
services. The Traveling DRC, when combined with a peer advisory team and its partners, had
opportunities to share information and collaborate statewide. In essence, the Traveling DRC
differed from a DRC who remained stationed at an AJC within a particular WDA in that it
allowed DRC services to be available across a number of WDAs.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
43
Challenges and Sustainability viii.
As the DEI State Lead noted, “transportation is a huge problem in such areas as
Sacramento.” Most WDAs and the populations they served increased markedly, yet there was
little change in the state’s transportation system. For JSWDs who have limited access to
transportation, it may be difficult to travel to work that is outside of their metropolitan area as
light rail has not been expanded to some WDAs and employer locations.
“A lot of thought about the job goes into how to get there. Employers ask for a driver’s license,
even if the job does not require driving. Many people with disabilities do not have a driver’s
license, which [requiring a license] performs a de facto screening-out function.” — DRC
Challenges to the implementation and sustainability of the DEI are limited due to the
state’s access to additional resources that fund DEA through state resources. DEA focuses on
partnerships between the AJCs and employers that, per the DEI State Lead, “accelerate
employment and reemployment strategies for individuals with disabilities.” The DEA is similar
to the DEI as it focuses on connections with employers and engaging them in the development of
job training and employment opportunities.
The DEI team believed that the conclusion of DEI funding would lead to a significant
funding void and thus would mean limited resources for assessments, training, supports, and
dissemination. Nevertheless, there were many signs that disability awareness and promising
practices would be sustained as DEI State Leads and DRCs were intentional about integrating
DEI practices and policies into WIOA processes to forge system change. A major asset is the
DEA program, which funds all current and past DEI grantees to attend the state quarterly
meeting for peer learning, sharing effective practices, staff mentoring, capacity-building, and
sustainability.
The fruitful collaboration spurred by DEI, particularly with the California Labor and
Workforce Development Agency, appears to be continuing. Consequently, there will be “no
right door or wrong door” for JSWDs to enroll in WIOA services. The use of IRTs should also
endure due to state leadership’s involvement in “cementing integration of this service delivery
strategy.” DRCs will continue at their respective AJCs at DEI’s end through DEA. In addition to
state funds to sustain DRCs and DEI practices, stakeholders mentioned tapping into Title I,
which provides WIOA services, and Title III, which authorizes Employment Service. TANF and
Supplemental Nutrition Assistance Program (SNAP) allocations are also being used by
California to provide access to employment and training of JSWDs. A prominent partner also
suggested that statewide philanthropy can help sustain Career Pathways programming. By
having DEA and TTW, DEI respondents said that DRCs “have spread the roles out so our
services are easier to sustain over time because there is overlap with other programs and
because we have DEA and Ticket income, our services should be easy to sustain.”
Case management services also should continue as a key function of the grant as many
individuals benefitted from collaborations spurred by DRCs and WDA leaders. Many California
DRCs also became CWICs and were trained by ADA staff on employee rights, self-disclosure,
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
44
hiring, job accommodations, and enrollment in SSA and related programs. During the grant
period, WDAs also upgraded their physical and programmatic accessibility while state
leadership, in collaboration with the DEI State Lead and State-Level Supervisory DRC,
monitored access to WIOA services and programs. Given access to state resources, including the
DEA program, the state’s long-time experience in and commitment to supporting individuals
with disabilities, and the state’s focus on workforce development, it is likely that DEI/DEA
services will continue to serve JSWDs. California’s DEI addressed issues across micro
(individual), meso (services), and macro systems change concurrently.
B. State: Illinois; Focus Area: Adults with Disabilities
Stated Goals and Objectives i.
The Illinois Round 5 DEI proposed to achieve the following:
• Create systems change within existing Career Pathways systems to increase participation
of individuals with disabilities in the information technology (IT) sector;
• Increase awareness and involvement of JSWDs in CP and related employment and
training programs;
• Achieve the following individual outcomes: post-placement employment through CP
completion, benefits counseling, job coaching, AT, and support services (e.g.,
transportation, child care, housing, etc.);
• Allow DRCs to function primarily as case managers, mentors, and program navigators
who provide both employment and “life” coaching;
• Ensure options for access to information technology CP and related fields as well as
training in life skills such as budgeting and soft skills; and
• Allow DEI State Leads and DRCs to enroll JSWDs in information technology Career
Pathways programs through the DEI and extant training programs.
DEI Service Delivery Strategies ii.
Illinois DRCs developed IRTs and engaged in Blending and Braiding Resources to
provide SSA beneficiaries and other individuals with disabilities access to WIOA and/or SSA
services. IRTs were used to ensure that high school students enrolled in postsecondary education
and/or employment and CP and to facilitate opportunities for social support through bridge
programs that aided families in need of supportive services for their children with disabilities.
For example, helping a family access medical, educational, social, or other services could assist a
youth in achieving their employment goals, and coordinating multiple community-based services
could address the needs of an entire family and help them better support their child’s
employment plans.
DRCs also facilitated the transition process from school to training and eventual
employment in addition to providing outreach and marketing opportunities and ensuring students
had opportunities to participate in Career Pathways and receive appropriate supports and
services. Some DRCs engaged in ARC, which sometimes led to an IRT. ARC engaged
jobseekers early in the intake process, which included an application and documentation of a
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
45
jobseeker’s age and employment and training history, to discuss their key interests and concerns.
It was also used to identify specific activities or opportunities that may assist individuals with
disabilities to deal with challenges such as employment, socialization, and Asset Development.
Then, an assessment process was used to determine each individual’s skills and goals and if there
was a match between training, an internship, or OJT that may lead to employment. When
implementing ARC, DRCs used “scripts” for each youth to target specific activities designed to
assist individuals with multiple challenges to employment by increasing access to services and
supports, including DEI SDSs. While some DRCs were less familiar with benefits counseling
services, the grantee did have access to these services through DRCs who completed their CWIC
certification.
Illinois DRCs coordinated the employment and training process for each JSWD and were
the contact persons who provided case management support and access to SDSs and IRTs. IRTs
were beneficial to JSWDs because they brought together WIOA-mandated partners as well as
other DEI SDSs, including financial literacy training, WBL Opportunities, CE, Asset
Development training, and social support through both DRCs and Employment Specialists.
Other SDSs included Blending and Braiding Resources, Partnerships and Collaborations,
flexible opportunities for T/TA, and systems change. The Computing Technology Industry
Association was a key partner and “leader in the national IT industry” for providers of industry
certifications. It provided employment opportunities for individuals with and without disabilities
and offered a set of recently expanded Career Pathways. Youth with disabilities 18 years of age
and older were enrolled in SSI through the DEI. DRCs let them know early in the engagement
process about their eligibility and how working could be possible without affecting their
beneficiary status. During Round 1, DRCs had very limited understanding of TTW. Illinois
Round 5 was prepared for TTW implementation, although most youth were not old enough to
enroll in the program.
Participants’ assessment results may assist in determining the relevance of certain
activities to employment, such as socialization with peers or opportunities to meet with
employers, résumé development, mock interviews with employers, WBL, Job Shadowing, CE,
and OJT. WBL and OJT were used to acclimatize jobseekers to the work environment and
provide “real-world” wage-earning employment. For youth with disabilities, the DEI
implemented Guideposts for Success together with WBL.
Implementation Summary iii.
The key components of the Illinois youth project included job training opportunities and
job placement support through “sectorial partnerships”
27
in support of CP-oriented employment.
Sectoral partnerships were comprised of the DEI and AJCs, local employers, and the state’s
education community, including high schools, community colleges, and 4-year colleges. Illinois
27
National Skills Coalition. (n.d.). Sector partnerships. Retrieved from https://www.nationalskillscoalition.org/state-
policy/sector-partnerships
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
46
emphasized technology-related Career Pathways designed to help youth with disabilities obtain
in-demand skills and credentials designed to help them obtain a “living wage” employment.
28
While DRCs focused on systems change and universally accessible AJC services, the
state implemented professional development plans for each agency partner, including WDAs and
the Division of Rehabilitation Services, and improved IT focused on “self-paced, accessible
online learning and classroom-based opportunities.” Illinois also developed an employer
outreach strategy that included regional partnerships designed to facilitate enrollment in IT-based
occupations from high school through college. DEI SDSs included Individual Learning Plans
(ILPs), Individual Education Plans, IT-related WBL Opportunities, Job Shadowing, and CE.
These were augmented by ARC, OJT, internships, and CP tailored to youth with disabilities.
Other SDSs include Asset Development training, IRTs, Blending and Braiding Resources,
Benefits Planning, and Partnerships and Collaborations with WIOA-mandated and non-mandated
partners.
Partnerships and Collaborations iv.
Illinois Career Pathways included partnerships with the Illinois Pathway Initiative
Council, the Division of Rehabilitation Services, and local and regional partners, including the
Computing Technology Industry Association, local high schools, community colleges, nonprofit
agencies, and participating WDAs. The Department of Commerce and Economic
Opportunity also worked with the Division of Rehabilitation Services on “Have Dreams,” which
was the Illinois Task Force on Employment and Economic Opportunity for Persons with
Disablities. The xpanding information technology Career Pathways project was another
important component of the project The latter had representation from several state agencies
that coordinated statewide services for individuals with disabilities that included “employer-
driven pathways” in DuPage, suburban Cook County, and two Project SEARCH Transition-to-
Work Programs. The latter were led by local employers that provided a workplace that included
classroom instruction on soft skills and technical skills, career exploration, and worksite training.
DRCs appeared to be involved as partners with the high school transition staff “so by the
time they graduate, they’ll be comfortable. There’s a seamless transition from school to training
or whatever. We only have relationships with some of the high schools here, but the county is
big, so there are a lot of potential clients. We also focus on the community colleges because they
have a lot of JSWDs, so we go and see if we can help them out too.”
DEI employment outcomes focused on job retention and wage outcomes by providing
follow-along job counseling and support by engaging partner organizations. JSWDs were
connected to VR services, TTW services, and DEI supports such as case management, AT, and
transportation and child care assistance as needed. DRCs reported that they functioned primarily
as case managers, mentors, and program navigators who provided both employment and “life
skills.”
28
Partnership for Working Families. (n.d.). Policy & tools: Living wage. Retrieved from
https://www.forworkingfamilies.org/resources/policy-tools-living-wage
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
47
Illinois implemented TTW and was certified by SSA as an EN after completing its Round
1 grant. As with most early grantees, Illinois Round 1 struggled with EN implementation. By
Round 5, their EN was operational and provided opportunities for DRCs to enroll individuals in
Benefits Planning workshops; several became CWICs who supported the implementation of
TTW and collected milestone and outcome payments in Round 5. Illinois also worked closely
with AJC staff and WIOA-mandated partners and CP program administrators.
Employer Partnership v.
DEI youth with disabilities shared that they appreciated help with training and
employment opportunities. Youth were eager to meet and communicate about work and life with
employers, DRCs, and other people who were interested in their situations. Several mentioned
that developing a network of “friends” was an important part of employment as it provided an
outlet for socialization, building a reputation, and sharing information with colleagues. As one
youth beneficiary said, “I like helping people, anything involved with getting customers napkins,
sauces, etc. What I don’t like about my job is that there’s not enough good communication
between staff and managers. They’ll come in and work a shift and ask us to stay longer so we
end up doing a double shift. And sometimes you tell one manager one thing, but they don’t
communicate with each other. So sometimes you have managers all scheduling you at different
times. But that’s what it is. It’s sometimes a challenge to do everything.”
“I was having trouble rounding-up the shopping carts and they noticed I was having trouble, so
they worked with me. The movie theatre; I wouldn’t recommend them. I ended up losing that job
because I got sick and they weren’t willing to be flexible because I was sick during the first 90
days, which is a rule they never told me about. The training was very sparse; they never
explained how clocking in and clocking out worked, and I once had to leave work in an
ambulance, which I think made them nervous.” — Youth Beneficiary
“I worked at an ice cream shop. It was a good experience, but the pay was below minimum
wage. I had to leave to get a different job at White Castle, which wasn’t a good experience. They
would schedule us and then send us home regularly. But I worked until I graduated high school,
then left and now I work at McDonald’s, which is OK.” — Youth Beneficiary
Youth received VR services for accommodations (e.g., extended time on high school
tests and exams) as well as WIOA and Career Pathways training through community-based
agencies located in each WDA; several Illinois youth were noted to have received extended time
on reading and mathematics, automotive training and repair, and interface with local employers
to provide opportunities for Apprenticeships and Work-Based Learning Opportunities. Illinois
youth were dually enrolled in employer-sponsored vocational and academic training, which
provided individuals with a new perspective.
“I can get help and be supported through all of these programs. I’m enrolled and pursuing a
career in early childhood education. But, there’re a lot of different routes. I like that it helps you
get skills that you need. I also like how there’s multiple paths to the program, like there’s
training and education programs. Some of us are in a program doing IT.” — Youth Beneficiary
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
48
Youth also had access to funds through WIOA that supported training needs. As one
youth noted, “the DEI helped me pay for my training. It gave me a new perspective, that I can
get help and support through all of these programs.” Many students pursued certifications in
early childhood education, computer and information systems, software development,
automotive repair, positions as teaching assistants at child care facilities, and supports such as
training in public speaking and community activism.
“I started out in the 3-week workshop. I was originally going to take an internship, but I had an
issue. Not enough income. So, I changed at the last minute and am now going to college. I think
what the DEI is doing is great because it helps JSWDs get a job that pays well and helps me
understand finances. Going in to the workshops to get ready for college was great. I also liked
that they showed a variety of career options that didn’t involve academics.” — Youth
Beneficiary
“I had a job before this and another one before that, but I left them both. I worked at a movie
theater making sure everything was clean, helping people get to the theater. I also worked at a
union job at a grocery store, and I bagged groceries, but then I switched departments because of
my health issues. My latest job was to make sure everything on the shelves looked nice and was
in the right place. They were really accommodating with my disability too. I had to be in and out
of the hospital, and I appreciate that they worked with me so I could do that.” — Youth
Beneficiary
The grantee’s employer outreach efforts included representation from several state
agencies that coordinated statewide services for individuals with disabilities. Additionally,
DuPage and suburban Cook County engaged school systems and community colleges with
“Have Dreams,” a project that provided services to individuals with autism. The project
maintained relationships with employers, the Chamber of Commerce, and economic
development agencies to provide work opportunities for autistic youth and adults. As a
coordinated network, Have Dreams also partnered with local AJCs to identify and recruit
employers that could provide mentoring support services, including WBL Opportunities. The
state’s WDAs also help to identify “champion” employers that served as mentors in identifying
employment supports for information technology Career Pathways “partnering employers.”
The grantee also implemented “Illinois Pathways,” which were statewide public-private
learning exchanges designed to support local and regional partnerships in each WDA. Local
WDA partnership efforts included those with employers, targeted business sectors, unions,
higher education agencies, and community-based agencies. When taken together, employer-
focused outreach combined with individualized employment and training services for JSWDs
with partners like the Computing Technology Industry Association helped employers connect
with DEI participants who had acquired locally in-demand skills.
Career Pathways vi.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
49
Career Pathways included services for youth in transition designed to develop skills in
information and related sectors.
29
Available occupational sectors included health care,
manufacturing, agriculture, transportation, logistics, and IT. Through various public-private
partnerships, the Computing Technology Industry Association offered an IT skill development
and employment system that facilitated employment soon after the completion of job training.
The state’s WDAs maintained task forces designed to increase opportunities for individuals with
disabilities through “employer champions” that provided access to “full-inclusion employment.”
Recently, the Illinois Department of Commerce and Economic Opportunity conducted a
statewide study to identify “key practices among schools, colleges, universities, and workforce
partners in job-driven, full-inclusion strategies.” Illinois also expanded its partnerships to
include VR and the Department of Education with the expectation that individuals with
disabilities who enrolled in Career Pathways would be equipped with soft skills, including
reading, writing, workplace etiquette, and access to “living wage employment.” There was also
an IT “boot camp” that helped jobseekers become familiar with basic computer use and
technology.
Outreach and Dissemination vii.
Youth with disabilities were referred to DEI from WIOA Youth services, teaching staff
from local high schools, community-based agencies, and DRCs. Targeted outreach to youth was
delivered by DRCs in schools where they conducted presentations about DEI and AJC services
and supports. Learning forums that included information about career opportunities such as
Apprenticeships, WBL Opportunities, and internships also helped eligible youth learn more
about DEI services. DRCs used separate “scripts” when presenting to youth, covering topics
related to job prospects, access to support services, ARC, and IRTs. Outreach was also done to
engage youth in the process of selecting a college or technical pathway. For example, if a youth
was interested in IT, they could enroll in a technology-related boot camp to familiarize
themselves with the IT field prior to receiving formal training.
DRCs also engaged in outreach to youth with disabilities through presentations about
DEI services, including college enrollment and CP opportunities. In addition, DRCs did outreach
to enroll youth in programs that provided an array of support services, from opportunities for
socialization to physical or psychological support.
29
Illinois expanded the capacity of its workNet Centers (i.e., AJCs) to improve the employment outcomes of
individuals with disabilities age 16–24. Many WIOA youth do not attend school regularly. They also may hold a
high school diploma and are low income and/or basic skills deficient. Youth may also be involved in the juvenile
or adult justice systems and categorized as homeless, runaway, eligible for foster care (Section 477) and/or in a
designated placement (e.g., foster care, home care, juvenile program placement, etc.), pregnant/parenting, having a
disability, and/or low income.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
50
“I was referred to the DEI from WIOA Youth services. They told me about it and that they could
help me get accommodations and things I need. I like the DEI because it helps me with stuff that
I have a disability with. They’ve helped me get extra time on tests. And I’ve been taking classes
and doing training to get ready for work. I’m also taking automotive classes and reading and
math.” — Youth Beneficiary
Promising Practices: “Pathways for Success” viii.
Illinois DRCs had a strong partnership with VR that included a dedicated VR staff
member who made referrals to Career Pathways services, enrolled individuals in WIOA services,
and provided oversight of individuals who enrolled in community college.
Rationale for Its Implementation
DRCs worked closely with VR to make referrals to employment and training
opportunities, including Career Pathways. In addition to the implementation of CP, Illinois
provided WBL Opportunities, including Apprenticeships and internships, that allowed youth to
learn about employment and what it entailed (e.g., receiving remuneration, working in a group
environment, adhering to the requirements of the leader or “boss”). The advent of CP allowed
JSWDs to prepare themselves for the work environment, sample different kinds of occupations,
and develop interests that could lead to long-term employment.
Why the Practice Could Be Considered Promising
DRCs organized job fairs in which JSWDs learned about new opportunities for
employment and WBL Opportunities, Apprenticeships, internships, and Benefits Planning, all of
which were key components of the DEI that lead to self-efficacy and transition to full- or part-
time employment. “The benefit is that youth get to go to school and get any assistance that they
are eligible for from DEI or SSA, which is an important step toward developing the confidence to
be self-sufficient and maintain employment and independence.”
Challenges and Sustainability ix.
Illinois leadership initially expressed concern about the structure of the DEI grant and its
implementation. The grantee’s original idea was to develop partnerships across the state’s WDAs
and provide leadership and support to AJCs and JSWDs. However, the grantor wanted the DRCs
to coordinate outreach in each WDA to inform the AJCs and JSWDs of the kinds of services the
DEI offered and to focus on recruitment of JSWDs and integrating the AJCs so that all services
were available to both individuals with and without disabilities. The goal was a seamless,
integrated system for serving all jobseekers.
“The grant has great intentions, but all the things that require of it take away from
serving JSWDs. Like meeting with all of the different agencies for the same thing, needing to
report information in a very specific way, made it less productive than it could have been.”
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
51
Initially, DRCs appeared to have limited knowledge of TTW, EN implementation, milestones,
outcome payments, and SSA work incentives. Although DRCs initially reported a limited
understanding of TTW, beneficiary recruitment, and the overall structure of TTW, DRCs
operated with limited involvement and training from the WDAs. This situation was improved
with the addition of a DEI liaison who focused on TTW and EN and was the initial point of
contact for individuals with disabilities. Eventually, DEI leadership successfully implemented
TTW and recruited beneficiaries.
While TTW was seen as a way to sustain the DEI, many youth did not have an assigned
Ticket because they were under the age of 18. DRCs commented that they had limited familiarity
with TTW, with one saying that they “know it has a goal of employment rather than receiving
benefits. I don’t know how it factors in with us, but I think we may pursue it. But I don’t have
much knowledge of it.” DRCs connected with the NDI TA center to discuss TTW and providing
access to community-based service providers through a statewide EN, including milestone and
outcome payments. Illinois also provided TA regarding the implementation of their EN. The
relationships that were created with VR since the implementation of Round 5 grant will continue
to be available after the grant period, and local colleges, such as the College of DuPage, will
continue to provide academic supports. However, it does not appear that all DEI WDAs will
have access to DRCs to implement TTW.
“We’d have to work at TTW to become more familiar with it and sustain DEI in Illinois. We’d
need to involve partners. It may get territorial because people don’t want to give up their
involvement in TTW. However, the relationships we’ve created through the DEI will still be
around because everyone’s happy to have all those resources. The relationship with the College
of DuPage will be there too because we have common goals.” — DRC
C. State: Kansas; Focus Area: Adults with Disabilities
Stated Goals and Objectives i.
Kansas Round 5 was designed to build upon the foundation of the state’s Round 1 DEI
grant by continuing efforts toward universal accessibility. A primary objective of Round 5 was to
“increase the percentage of JSWDs served by the public workforce development system with
training services more typically available to other jobseekers” beyond WIOA. Improving
accessibility included an emphasis on cultural change within the workforce system, with an end
goal of augmenting access to Career Pathways and certification for JSWDs.
The Kansas Round 5 DEI proposed to achieve the following:
• Improve the postsecondary education and training outcomes for adult customers with
disabilities by:
o Increasing the percentage of adult customers with disabilities who are referred to
postsecondary education and training for existing successful Career Pathways;
o Increasing the percentage of adult customers who receive community and technical
college credentials along successful CP; and
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
52
o Increasing the percentage of adult customers who receive paid work experience
(internships, job sampling, etc.).
• Improve the employment outcomes of adult customers with disabilities who obtain and
retain employment through a continuum of services to support education, training, and
employment success by:
o Convening individuals representing project partners, including community and
technical college staff, VR staff, community-based organization staff, WIOA staff,
and individuals from industry, to promote collaboration among multiple service
providers toward relevant skill development of customers with disabilities;
o Developing leadership training to community staff and improving and enhancing
assessment, accommodation, and coaching of students with disabilities;
o Through DRCs, training WIOA staff to establish and support OJT and other paid
work-based training opportunities;
o Through the DEI Technical Assistance Coordinator, providing a written report to the
State Lead detailing a review of quarterly performance data, individual customer
records, and case management documentation to ensure customers are receiving
relevant, timely, and effective services;
o Providing real-time employment and training data on JSWDs; and
o Increasing the percentage of KANSASWORKS customers who have disabilities who
are placed in OJT.
DEI Service Delivery Strategies ii.
Active Resource Coordination — or “finding the best resources to meet individuals’
needs” — occurred regularly across treatment WDAs. ARC was the first step in the Integrated
Resource Team. ARCs identified targeted activities/programs designed to assist customers with
multiple challenges to employment by increasing access to services. A participant’s assessment
results assisted in determining such targeted activities/programs.
DRCs frequently engaged staff from other service delivery partners such as VR and
community-based agencies because “we want everyone to get all the resources we have.”
Without DEI, ascertaining this information and additional resources would likely be “pushed
back to the participant” without support from DEI and WDA personnel. One WDA partner
indicated that “more regular meetings” could have improved ARC by providing opportunities for
JSWDs to select particular programs, activities, and Career Pathways to support their specific
employment needs.
IRTs were used frequently across the state. They included representatives from VR and
the state’s mental health system, where appropriate. Kansas reported that 160 IRTs were formed
during the grant period. However, IRTs were often conducted informally due to time and
resource constraints. In one WDA, a formalized meeting occurred twice because necessary
parties were “slammed” due to other commitments. However, when they occurred, those
meetings “worked seamlessly” and included DRCs, VR staff, WIOA case managers, job coaches,
and business services. While Kansas’s final quarterly report indicated IRTs were being initiated
by partnering organizations on behalf of shared customers, interviewed partners in one WDA
reported no involvement in IRTs.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
53
Blending and Braiding Resources happened consistently, including WIOA co-enrollment,
which helped fund certifications and support services. Combining WIOA, DEI, and VR funds
also allowed for extended training. One staff member asserted that blending WIOA and DEI
enabled youth programming to last 8 to 10 weeks to explore careers. Pell Grants also helped with
school-to-work transitions. As a DRC testified, funding cuts elevated the importance of Blending
and Braiding Resources.
While Kansas selected Self-Employment/Entrepreneurship, it was “not a focus” of the
grant due to it being “difficult … to get up and running.” The AJC purportedly lacked business
development experience (e.g., about insurance), combined with high start-up costs, that would
make the implementation of Self-Employment a challenge for the grantee and JSWDs. AJCs did
refer clients interested in starting their own business to community colleges and organizations
that supported business development. One DRC “learned a little bit” about helping clients with
a Plan for Achieving Self-Support (PASS). DRCs also encouraged JSWDs seeking to explore
Self-Employment to consider “supplementary” income, such as from Etsy or farmers’ markets.
In terms of Asset Development, one AJC facilitated access to budgeting assistance, but it
did not offer any classes focused on Asset Development, nor did it proactively promote ABLE
accounts. A DRC asserted that the AJC prioritized immediate employment and training
opportunities rather than long-term asset building. Another WDA focused on Asset Development
that encompassed PASS plans and Individual Development Accounts. The third WDA did not do
any Asset Development with JSWDs according to a DRC. A related service, Benefits Planning,
was provided frequently to JSWDs.
There was evidence of TTW activity in treatment WDAs, as it was administered by the
state with some resources distributed to WDAs. One WDA was still researching the process of
becoming an EN, while another mainly referred TTW-eligible clients to available ENs. DRCs in
one WDA critiqued TTW for being “convoluted” in several ways: the ambiguous relationship
between employment and benefits and which clients were suitable; the delay and uncertainty in
returning revenue to local areas; and it being framed as a vehicle for immediate employment.
There was also uncertainty about AJC capacity to handle TTW in one WDA. Still, according to
its final quarterly report, Kansas exceeded its TTW expectations with 32 beneficiaries with
Ticket assignments and $79,307 in revenue.
The state’s Department of Commerce trained partners on CP programs. Significant funds
were directed to OJT training, internships, and apprenticeships. Project SEARCH was
implemented and DEI funds were made available for training from the Cerebral Palsy Research
Foundation and School of Adaptive Computer Training.
Implementation Summary iii.
Kansas’s Round 5 grant was administered by its Department of Commerce, which
oversaw both the workforce system and state economic development initiatives. State leadership
had significant and relevant experience, including with the Disability Program Navigator and a
DEI Round 1 grant. A unique feature of Kansas’s service infrastructure included managed care
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
54
organizations and three state agencies directly involved in Medicaid policy, service delivery, and
oversight. Round 5 included three treatment WDAs. One WDA had multiple concurrent DRCs
although DRC turnover was common across sites.
To accomplish its goals, Kansas proposed creating cross-agency partnerships through the
Kansas Department of Commerce, which was the linchpin of multi-agency work groups that
streamlined referrals, employer outreach, and data-sharing, including the Governor’s Technical
Education Authority, the Kansas Commission on Disability Concerns, and the Kansas
Employment First Oversight Commission.
Kansas engaged businesses in in-demand sectors, including engineering, manufacturing,
IT, health care, energy generation and distribution, and construction. Education and training
programs were provided, including Career Pathways with multiple levels of education so
students could make progress at their own speed. In addition, their education strategy included
flexible entry points and short-term credentialing that included remedial intervention and
responded to labor market information where employment opportunities existed.
Kansas measured systems change and performance with postsecondary education
partners to report DEI participants’ progress, including grades, attendance, and accommodations.
The Department of Commerce promoted CP programs, including through Family Employment
Awareness Training. DRCs facilitated IRTs, including staff from managed care organizations,
colleges, parole offices, and community-based organizations, and they encouraged staff to
engage JSWDs with high-quality intensive, supportive, and wraparound services, including
Benefits Planning, financial literacy, and career coaching along with universal accessibility
through alternative assessments and trainings (e.g., hybrid learning and cohort model), ILPs, and
service coordination.
Kansas implemented most features of its grant application, though there was not much
evidence of a monthly reporting mechanism for IRTs or ILPs. A DRC asserted that her WDA
was “the only area” in Kansas to implement CE. An OJT opportunity went well and an employer
wanted to keep the client, but for another position. A new job description was created to “fit his
ability and needs.” However, a partner in this WDA desired more CE. Another WDA used group
Discovery as a tool for Career Pathways.
Partnerships and Collaborations iv.
Coupled with WIOA, DEI strengthened the state workforce development system’s
collaboration with VR and fostered many new partnerships throughout treatment WDAs. As
well, DRCs served on Commerce steering committees in their respective service areas. However,
in one WDA, there was some “pushback” from providers leery that DEI was encroaching on
their funding.
Heightened collaboration fomented productive cross-training — especially between the
core partners, KANSASWORKS and VR — which enlightened staff about additional funding
sources and services for JSWDs. Cross-training was promoted by state policy frameworks,
including integrated service delivery and functional management, which included management
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
55
that maintained authority over organizational units in an agency. In the case of the DEI, it was an
approach to selecting WIOA and Workforce Innovation Fund services.
DEI also helped codify linkages with postsecondary and career and technical schools.
Schools became more aware of JSWDs and AJC services. WDAs also cultivated synergy with
Project SEARCH through DEI, enhancing information-sharing and assistance to mutual clients.
DEI strengthened bonds with Behavioral Health Services agencies, local nonprofits, and
managed care organizations, which led to more AJC programming, more efficient job placement,
and easier access to partners and supports for JSWDs. One DRC solidified bonds with reentry
programs.
IRTs and staff meetings increased partnerships and referrals. A WDA director
appreciated workforce representation at board meetings and praised core partner meetings for
facilitating “comprehensive” service delivery, “smooth” referrals, and enhanced relationships
with job developers and college counselors. Nonetheless, one AJC partner suggested there could
be more cooperation to increase mutual awareness.
DRCs were the linchpin of expanding partnerships. DRCs connected stakeholders with
“the right person in the system” and elucidated “how to use different assets from different
agencies.” In one WDA, multiple interviewees credited the DRC role with improving
KANSASWORKS’s relationship with VR. DRCs also improved rapport with referral sources.
However, ongoing turnover in the DRC role frustrated relationships with some partners.
State policies and leadership encouraged collaboration. For instance, the DEI State Lead
met with VR leadership to promote referrals to WDAs. A DRC valued state leadership as a
“resource” and for “connections.” Per the DEI State Lead, WDAs were much more involved due
to WIOA’s access to its services and supports. Co-enrollment — particularly for young adults
but also for VR and Title I clients — expedited integration of JSWDs, while state-based projects
to improve data-sharing and referrals were also underway toward the end of the grant period.
Cross-agency staff training maximized universal access, established eligibility, and
provided career services to JSWDs. Support services were provided by employers, community-
based agencies, and the DEI. Treatment WDAs made progress with the state in upgrading
physical and communications accessibility, including installation of automatic doors, refined
language for engaging JSWDs, and “plenty of resources” for AT. DEI also furnished resources
that provided clothing for employment opportunities, trainings, and services for a broader array
of employment pathways, including social work, child care, and taxidermy.
Treatment WDAs indicated that they needed more training and capacity-building in
benefits counseling. JSWDs generally had access to a benefits specialist, but AJCs discussed
staff becoming CWICs as well. In one WDA, DRCs expedited “constant” staff training,
including on co-enrollment. DRCs commented that “still more training in disability etiquette and
eligibility would be beneficial across the state. One of our WDAs could use more regular
trainings on serving JSWDs.”
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
56
Two of three WDAs demonstrated customer-centric qualities, such as providing
information on job accommodations as JSWDs sought employment and encouraging them to
articulate their employment goals. In one WDA, a participant focus group yielded mixed
feedback about customer choice. Positive testimony included characterizations of staff as “very
friendly” and supportive of individuals by securing internships in relevant fields. Jobseekers also
appreciated vast “networking resources.” However, there were multiple critiques from JSWDs in
one WDA, including a lack of clarity about wages upon placement and long-term opportunities
and a “one-size-fits-all model” approach to job development, including mandatory completion of
unnecessary classes. By contrast, the DRC from one WDA described efforts to streamline client
assessments at intake to make the process more customer-directed. She also reflected on how
DEI funding allowed clients to explore more career options. Further, she encouraged enrolling
clients in as many programs as possible, “so long as it is appropriate for the customer.”
Though a DRC stated she was “not even familiar” with universal design, treatment
WDAs exhibited several inclusive features. One AJC facilitated both in-person and online
coursework through partners, depending on a client’s preference. AT in treatment WDAs
included Microsoft Reader and other screen-reading applications, as well as adaptations like
putting programs on participants’ phones or adjusting computer layouts. Collaboration with local
community colleges expedited appropriate AT. Workshops were multi-modal, including
PowerPoint slides, written material, videos, and interactive “soft skills” activities.
Goal-setting was promoted through user-friendly job-seeking platforms like O*NET
OnLine and guidance on budgeting. Career exploration inventories and identification of
transferrable skills reflected efforts to promote autonomy and relevance in job searching. DRCs
also varied demands and resources based on a client’s background, interests, and skills.
DEI built upon a state initiative that incentivized employers hiring JSWDs. DRCs were
influential in rolling out the program. In one WDA, the AJC was proactive in engaging
employers, with one DRC seeing his primary role as “to reach out and educate employers.” This
AJC interacted with employers through a variety of vehicles, including direct outreach, planned
meetings, and job fairs. Business services also facilitated employer outreach, particularly for
young adults, and a partner assisted in this endeavor. A staff member affirmed that the DRC was
integral to understanding which employers were interested in hiring JSWDs. Outreach to
employers involved framing JSWDs as being “more dedicated” than jobseekers without
disabilities.
Employers confirmed this AJC’s proactive engagement and depicted AJC outreach as a
catalyst for relationship-building with JSWDs. The AJC also helped employers understand
JSWD needs and skills and provided assistance with any issues. WDAs conducted outreach to
businesses; while one DRC discussed considering an employer as “a partner,” there was little
evidence of employer involvement in AJC strategic planning.
DRCs identified several challenges to employer engagement. Securing buy-in to hire
JSWDs, including through provision of OJT funds, was challenging due to uncertainty about
liability, the practicality and cost of accommodations, and disability etiquette. DRCs suggested
small businesses may be more receptive to outreach than larger corporations due to a greater
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
57
likelihood of engaging an individual comfortable with relating their own familiarity with
someone with a disability. They also indicated employers were more open to hiring younger
JSWDs. An employer identified the obstacle of needing specialized and technical skills from
JSWDs. Some employers were also reluctant to get involved with OJT opportunities. Employers
showed preference for paid work experiences or direct placement. Partners bemoaned confusion
about establishing OJTs despite a multitude of potential worksite connections and called for
additional information and training in this facet.
Career Pathways v.
DEI financing supported longer training programs than WIOA and was more conducive
to stackable credentials and career ladders. In one WDA, DRCs were conversant in career
assessment options, including WorkKeys, the O*NET Interest Profiler, and TABE for youth.
This AJC also harnessed labor market information and job databases to provide JSWDs with
pragmatic insight. According to the DEI State Lead, Round 5 was instrumental to formalizing
relationships with postsecondary institutions and to enhancing accommodation policies at these
schools. These improvements were crucial to supporting WBL and work experiences
implemented statewide, including for youth and epitomized by Project SEARCH internships.
DEI facilitated 42 Project SEARCH internships in Kansas.
According to Kansas’s final quarterly report, 152 DEI participants completed classroom
training leading to certification, and 138 individuals participated in paid work-based training.
About half acquired Microsoft Office Specialist certifications; customer service training was also
popular. Per Kansas’s DEI State Lead and DRCs, enrollees acquired and maintained employment
for at least 6 months at a higher rate than anticipated and at a higher average hourly wage than
projected.
Outreach and Dissemination vi.
DEI state leadership helped form the Transformers Coalition. The Coalition was designed
to improve access to and knowledge about transition services for youth with disabilities,
including through town halls across the state. Treatment WDAs engaged in multifaceted
outreach. For instance, DRCs and AJCs reached out to local nonprofits and relevant county
agencies to discuss available services and employment opportunities such as WBL and OJT, as
well as the potential for collaboration and resource leveraging to assist JSWDs. An “accessibility
committee” was another venue for outreach, and DRCs promoted DEI through other committees.
In one WDA, DEI was “discussed at a board level” to reach employers. Events like job fairs for
JSWDs and the Midwest Ability Summit also enabled outreach. DRCs were proactive in
engaging school districts and county governments, distributing flyers with information about
DEI.
To broadcast best practices, the Department of Commerce produced video interviews
exhibiting the promise of OJT. Interviews included DEI participants and employers from Kansas.
The state’s CP efforts through DEI were also amplified by NDI in a webinar, highlighting
significant CP enrollment. Local dissemination revolved around case management for JSWDs
and encouraging staff to be “creative” in serving this group. A partner cited the Project SEARCH
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
58
committee as another channel for information-sharing. Cross-training also allowed for
dissemination beyond AJCs. In addition to productive individual relationships with employers,
one AJC shared information with employers via business services and the Society for Human
Resource Management.
Promising Practices: “Pathways for Success” vii.
One DRC developed a “Pathways for Success” curriculum in partnership with mental
health providers. The curriculum emerged from IRTs and was targeted to jobseekers with
multiple barriers to employment, including mental health challenges. Pathways for Success
integrated elements of Discovery, CE, career exploration, résumé writing, soft skills
development, mock interviews, benefits counseling, and Asset Development. Participants met
biweekly with workforce and mental health staff. Several clients also developed PASS plans.
Rationale for Its Implementation
Pathways for Success precipitated other agencies and workforce regions to pursue this
model. An intellectual and developmental disabilities provider adopted the model and formed
two cohorts. Two DRCs also partnered to form a regional Pathways to Success group that
continued to meet monthly.
Why the Practice Could Be Considered Promising
The curriculum addressed a significant gap in services for JSWDs, particularly the many
who had mental health challenges and multiple barriers to employment. Without tackling each
obstacle with comprehensive services and supports, the employment journey would likely be
daunting.
Challenges and Sustainability viii.
Like many DEI grantees, the most trenchant challenge for Kansas was turnover in the
DRC role. This prevented optimal training and collaboration. Capacity gaps also remained in
benefits counseling, AT utilization, and serving JSWDs with multiple barriers. Capacity
constraints were also evident in one WDA that lacked resources to regularly assemble IRTs. In
addition, there was uncertainty about disability disclosure and serving individuals with severe or
multiple disabilities, though there was progress in deciphering specific disability information.
Multiple DRCs spoke about the challenge of navigating “helicopter” parents of youth with
disabilities to ensure youth voiced their input. Another staff member stated, “Our challenge is to
understand their needs” and to “channel [programming] to their needs.” Meeting one-on-one
with clients helped in this regard. There was no evidence of JSWD involvement in strategic
planning for the design of products and services for JSWDs.
Several features of treatment WDAs bode well for sustaining DEI practices. Multiple
DRCs had significant case management backgrounds, so they understood well the delicate
balance between service provision and systems-level training and coordination. In one WDA,
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
59
DRCs “[drew] a line with case management” but were “pulled back in several times” to provide
support and answer questions: “there’s always that mixing and molding.” A State Lead professed
that Round 5 DRCs did “not allow themselves to become case managers. They really are
resources for the case managers and other folks in the system.” According to a quarterly report,
case managers became more aware of other agencies and independently connected with partners
to create IRTs for JSWDs. This was corroborated in a post-grant sustainability call.
One DRC described her role as coordinating case management between agencies. DRCs
also spent much time fostering partnerships and conducting outreach to agencies; some explicitly
articulated this as their primary function. The DRC role encouraged sustainability by facilitating
awareness, training, and relationships. Moreover, former DRCs were integrated into AJCs —
two were still called DRCs — and were accelerating adoption of IRTs. AJC and some DEI staff
were also retained. Nonetheless, much turnover in the DRC role undercut long-term embedding
of DEI practices, as confirmed by the DEI State Lead in a post-grant call.
In terms of financing to undergird functions of a DRC after DEI, there was mixed
feedback. A staff member in one WDA suggested it would be “very challenging” to sustain the
DRC role, a reality that would also hinder collaboration. Many strengthened partnerships could
endure, such as with VR, but others could “suffer” without “the glue” of a DRC. The absence of
a DRC would also jeopardize AJC proficiency in engaging JSWDs “in an appropriate way.” On
the other hand, a State Lead communicated plans to fund DRCs as an “Other Shared Cost”
through partners in accordance with WIOA regulations. Reentry funds and data showcasing a
DRC’s impact were other potential avenues for sponsoring DRCs.
Beyond the DRC position, an interviewee from one WDA predicted that outreach and
training focused on JSWDs would be sustained by KANSASWORKS. The DEI State Lead
offered that JSWD engagement with Workforce continued to expand. The state’s final quarterly
report suggested training on DEI practices would be afforded to WDAs that were not involved in
DEI. Cross-system collaboration also became a feature of Kansas’s workforce development
system. Moreover, IRTs were replicated statewide, including by the state’s Service Guidance and
Support Teams, due to training through DEI. Per the State Lead, IRTs were the most lasting
legacy of DEI. OJTs continued to be promoted through television commercials and social media.
Post-grant, Kansas hired a consultant group to explore customer-centered design with employers
and customers. A State Lead was also charged with creating a Business Leadership Network
(BLN) and connecting with employers.
Blending and Braiding Resources will continue to be crucial, especially for a treatment
WDA with fewer resources and to ensure JSWDs receive both workforce and VR services.
However, a DRC submitted that local momentum to become an EN will subside without DEI,
restricting TTW revenue. After the grant ended, a State Lead did suggest that Kansas saw a
significant increase in Ticket holders. There were multiple potential additional avenues for
financing DEI practices. A State Lead indicated Pre-Employment Transition Services funding
would persist through an agreement with VR. A DRC mentioned the possibility of tapping into
the End Dependence Kansas grant and relying more on partners. Moreover, an AJC staff member
cited a state grant to fund IRTs. A partner intimated that Project SEARCH financing was solvent
through the county government. Finally, to translate DEI’s local impact into an outcome-based
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
60
regional and statewide sustainability strategy, Kansas also scrutinized local DEI data to better
understand customer performance and service utilization relevant to all five WIOA titles.
30
Kansas stands out in its alignment of federal, state, and DEI policy frameworks. This
synergy fostered the collaboration inherent to DEI, as exemplified by fruitful partnering, IRTs,
and Blending and Braiding Resources. These processes are likely to sustain well beyond the
grant’s end, as they had spread statewide. DEI contributed to significant JSWD engagement with
KANSASWORKS and partners, though capacity, resource, and training gaps linger. Project
SEARCH and Behavioral Health Services were unique partners, and Kansas leveraged these
relationships to cultivate pathways for clients with substantial hindrances. Ample employer
engagement led to opportunities for Work-Based Learning, including OJT. Still, OJT presented
complications that should be resolved. There were also implementation issues with TTW, and
the relationship between state and local TTW administration should be addressed.
D. State: Massachusetts; Focus Area: Adults with Disabilities
Stated Goals and Objectives i.
Massachusetts’s Round 5 “Pathways to Employment” project proposed to serve 165 adult
JSWDs, with a focus on bettering job placement rates by increasing access to community college
and Career Pathways in manufacturing, health care, and hospitality. “Better services” through
service coordination and universal design were seen as pivotal to this goal. The state laid out
seven objectives for the project:
1. Stimulate cohesion and collaboration among providers and agencies serving JSWDs;
2. Expand access to technical training and education in targeted sectors;
3. Increase the amount and diversity of employers hiring JSWDs;
4. Raise awareness among employers about the benefits of hiring JSWDs and about
available support services;
5. Provide training and support to JSWDs in navigation of career development, education
and training, and disability service systems;
6. Augment access to short-term subsidized work through WBL, OJT, and other direct work
opportunities; and
7. Support job placement and retention in unsubsidized employment.
Five additional “systemic changes” were presented as crucial:
1. Raise awareness among providers about current CP offerings and how to integrate their
services with existing initiatives;
2. Create trainings for employers, CP instructors, and AJC staff on supporting JSWDs;
3. Forge inclusivity and integration in CP programs, including by reflecting input and needs
of JSWDs;
30
WIOA includes five titles: Workforce Development Activities (Title I), Adult Education and Literacy (Title II),
Amendments to the Wagner-Peyser Act (Title III), Amendments to the Rehabilitation Act of 1973 (Title IV), and
General Provisions (Title V).
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
61
4. Augment support and ENs for students who complete CP; and
5. Engage employers through a “continuous communication loop” with educators and
trainers, “inclusion planning,” and job development and coaching to align their needs
with training and job placement strategies. Build employer bonds and translate employer
information to lead to “on-target” training and a greater likelihood of “pre-employment
exposure,” OJT, and placement. Several strategies were proposed to engage employers:
a. Business leaders were to present quarterly about needed skills;
b. Information sessions for at least 30 business leaders in manufacturing, health care,
and hospitality;
c. A biannual course for employers on recruitment and hiring, job accommodations, and
creating an inclusive environment resulting in an Inclusion Plan;
d. Consulting on job carving and development, accommodations, assessments, and
internships; and
e. Training on “natural” supports to provide long-term job coaching; five co-workers to
be trained to become natural support job coaches to assist in job stabilization.
DEI Service Delivery Strategies ii.
Massachusetts planned to achieve its goals and objectives through several service
delivery approaches. To supervise implementation, each treatment WDA established a DEI
committee that included partners and employers, such as the University of Massachusetts at
Boston, Institute for Community Inclusion (ICI). In addition to setting project guidelines, the
DEI committee promoted resource blending and braiding and gathering partners to participate in
an annual conference. The Department of Career Services (DCS) and the Massachusetts
Rehabilitation Commission (MRC) co-enrolled JSWDs; three other agencies or grants were
tapped for resource leveraging (e.g., Trade Adjustment Assistance Community College and
Career Training, Department of Mental Health, and Veterans’ Employment and Training
Service).
The WDAs featured DRCs, career counselors, and business services representatives, and
they also included mental health providers in IRTs. A business services representative asserted
that IRTs were essential to optimizing knowledge and resources for JSWDs, including about
accommodations. In another treatment WDA, IRTs included MRC, the Massachusetts
Commission for the Blind (MCB), and career development coaches from local community
colleges and nonprofit providers. IRTs were implemented more formally in the third treatment
WDA, which included wraparound services and training and employment opportunities.
Blending and Braiding Resources included partnering with VR, particularly for training
and work experiences. In one treatment WDA, DEI funding enabled VR clients to complete
trainings, primarily in access and registration. This AJC often referred JSWDs to VR to facilitate
transportation assistance, including cab vouchers. Toward the end of the grant, this WDA
overcame limited training funds and provided the same level of services to JSWDs by leveraging
WIOA Adult funds.
Through alternative assessments in concert with career exploration and benefits
counseling, the state expected to achieve “improved assessment, matching, and referrals to
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
62
appropriate programming with supports up-front.” Available assessment tools included
WorkKeys,
31
Transferable Occupation Relationship Quotient (TORQ)
32
software, and
Accuplacer, which is a school readiness program.
33
IRTs within each treatment WDA developed individualized service and career action
plans and integrated these plans within existing services (including TTW). Plans involved an
“eco-map” of existing systems and supports that outlined the role of each partner. In terms of
training, strategies featured “contextualized learning, compressed training awarding credit for
prior learning, dual enrollment, hybrid learning approaches, and job carving.” Community
college students also joined “learning community clusters” that offered small class sizes that
included team projects. DEI pathways included both certificates for entry-level employment and
stackable credentials.
Round 5 included a train-the-trainer approach to financial literacy through which all DEI
staff could promote Asset Development services designed to help an individual purchase a car or
home. Finally, Massachusetts hosted their second disability employment conference. DEI
stakeholders participated and employers advertised job openings. This conference helped with
replicating DEI practices throughout the state’s 16 WDAs.
Implementation Summary iii.
Massachusetts Round 5 built upon a CP infrastructure incubated by the community
college system in Round 3. Round 3 established the groundwork for an “expedited start-up
process” in Round 5. In addition, Round 5 benefited from statewide collaboration around CP that
germinated as a result of the state’s Transformation Agenda, a systems- and industry-based
approach supported by a $20 million USDOL grant. A state-level, cross-agency Coordinating
Committee oversaw the DEI grant. The committee was led by the Executive Office of Labor and
Workforce Development and the Executive Office of Health and Human Services and included
representatives from DCS, MRC, the Department of Mental Health, Veterans Services, the
Massachusetts Commission for the Deaf and Hard of Hearing (MCDHH), and Developmental
Services. This committee was a “permanent component for cross-agency coordination to support
… the needs of JSWDs.” Round 5 also included regional working committees to identify
resources and services for JSWDs. Multiple AJC staff members, the DEI State Lead and DRCs,
and employer partners were involved.
Notwithstanding much administrative restructuring and staff turnover, treatment WDAs
showcased the necessary leadership and capacity to facilitate positive employment outcomes for
JSWDs. Accessibility was a major focus that coincided with the state’s AJCs’ undergoing
certification processes. Compliance with WIOA Section 188 guidelines drove much of
accessibility planning, particularly in one treatment WDA. Local and national TA furnished by
31
ACT, Inc. (n.d.). WorkKeys assessments. Retrieved from https://www.act.org/content/act/en/products-and-
services/workkeys-for-employers/assessments.html
32
O*NET Resource Center. (n.d.). Transferable Occupation Relationship Quotient TORQ ™ from Workforce
Associates, Inc. Retrieved from https://www.onetcenter.org/paw/entry/125
33
College Board. (n.d.). What is ACCUPLACER? Retrieved from https://accuplacer.collegeboard.org/
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
63
ICI and NDI was instrumental in guiding treatment sites to alignment with Section 188,
including through cross-training. Massachusetts’s Office on Disability also conducted an ADA
assessment and reviewed AT access at participating AJCs.
To actualize accessibility, one treatment WDA focused on training, procedures,
accountability, facility upgrades, and communications. It completed three accessibility
assessments, scrutinized the 188 Disability Reference Guide, and developed an action plan to
remediate accessibility issues. The plan included two full-day, all-staff trainings, which were
conducted with assistance from ICI and focused on best practices for universal design and
services. Attendees included MRC, MCB, MCDHH, and Work Without Limits (WWL).
Representatives from NDI and the LEAD Center served as trainers.
In addition to AJC signage changes to improve communications accessibility, one WDA
enhanced its MassHire website to make it more inclusive, though it had insufficient funding to
completely transform the site as desired. The AJC’s lobby and wheelchair ramp were redesigned
to better accommodate people with a mobility impairment. In another treatment WDA, the AJC
was relocated during the grant period and subsequently assessed for accessibility by an ADA
coordinator. The AJC also conducted Section 188 training and received accessibility guidance
from NDI.
Partnerships and Collaborations iv.
Round 5 built upon collaboration and integration spurred by earlier DEI grants and access
to WIOA services. In addition to serving as state partners, ICI and WWL nurtured local
connections. ICI was integral to incubating partnerships in treatment WDAs, and WWL
connected treatment sites to other disability initiatives in the state, including Partners for Youth
with Disabilities’ Campus Career Connect program and the National Organization on
Disability’s Campus to Careers pilot. Round 5 bolstered bonds with MRC, which is co-located in
treatment AJCs at least monthly, including through co-enrollment and by treatment sites
attending monthly MRC meetings to encourage referrals. Treatment WDAs also partnered with
MCB, MCDHH, the Department of Developmental Services, the Department of Transitional
Assistance, and the Massachusetts Department of Elementary and Secondary Education’s special
education unit.
Collaboration was encouraged by monthly meetings coordinated by the State Lead, which
were either in-person or by teleconference and attended by DRCs, WDA leadership, ICI, WWL,
and NDI. Successful placements and upcoming events were discussed, and participants also
resolved difficult cases and other implementation challenges. One DRC described the meetings
as “great … I think you learn a lot from hearing what other sites do.” In between monthly
meetings, DEI leadership communicated regularly with DRCs and Employment Specialists
through T/TA workshops. Two treatment WDAs also partnered to facilitate internships.
At the local level, one treatment WDA established a Disability Employment Coalition
since there was no existing regional collaborative like in other treatment WDAs. The coalition
met regularly and grew throughout the grant, incorporating employers, schools, family assistance
providers, youth transition services, Adult Basic Education (ABE), Job Corps, Easter Seals, and
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
64
other social service agencies. The coalition helped match JSWDs with available trainings and
positions and expedited IRTs and referrals. In addition to the coalition, the DRC worked closely
with MRC, MCB, the Department of Developmental Services, and the local community college
and high schools to leverage resources, implement IRTs, co-enroll, facilitate trainings, and
recruit and place candidates.
In another treatment WDA, DEI contributed to “a much more collaborative approach”
and “a whole lot of communication.” The AJCs and a local provider that specialized in training
jobseekers with significant disabilities partnered on designing and implementing trainings and a
“referral pool” of JSWDs and employers. For example, one AJC trained the provider’s youth
clients for retail jobs. AJCs also united with veterans’ representatives and a mental health service
provider. A DRC remarked that communicating with referring agencies was a challenge.
The third treatment WDA made deliberate efforts to ensure DEI “was not a siloed
program.” Accordingly, about “90 to 95” percent of DEI clients were co-enrolled in WIOA,
Title I, or other programs. As conveyed by career counselors, however, a caveat to this strategy
was the stringent employment outcomes required by WIOA. This WDA also fostered “a really
strong relationship” with MRC, including through many referrals, alignment of trainings, and
“constant communication.” The DRC also presented at MRC staff meetings. Each treatment
WDA also conducted cross-agency training, as well as training of provider staff that included
information about JSWDs, opportunities for soft skills training, and information about TTW.
Transportation was cited as a “huge” barrier to physical accessibility. This dilemma was
mitigated by the local staffing agency instituting a new van route. Additionally, with the support
of DRCs, the WDA secured funding from a local bank to help with transportation. AJC staff also
attended a statewide transportation conference as part of ongoing efforts to tap into state
resources and identify transit options. In another treatment WDA, a DRC was compelled to refer
JSWDs to MRC because “we cannot provide the transportation that someone might need,”
whereas MRC could supply cab vouchers. Braiding funds enabled MRC to finance transportation
and the AJC to fund training. A lack of transportation access was compounded by many
vulnerable jobseekers having to secure a driver’s license or a vehicle; “they may be able to go to
work tomorrow if they had transportation.” The AJC also did not have accessible parking. To
address these issues, DRCs worked with WDA staff to engage MassRIDES and advocated via
the state’s transportation committee. One WDA offered travel training for JSWDs, which
provided road maps to frequently visited locations selected by JSWDs.
ICI and NDI helped with mental health and anxiety accommodations and clinical
supports to facilitate programmatic accessibility. ICI also evaluated the intake process at
treatment AJCs. Treatment WDAs made substantial efforts to serve all JSWDs, including those
with significant additional barriers (e.g., substance abuse and criminal history). One treatment
WDA featured a partner that specialized in serving JSWDs with intellectual and developmental
disabilities and who needed “more significant supports,” as well as serving referrals from MRC
for the “hardest-to-place” individuals. A DRC from this partner indicated that AJCs struggled to
serve and place jobseekers with intellectual and developmental disabilities and that such
jobseekers were less likely to seek full-time employment and/or Career Pathways training due to
the skill and training requirements. The DRC argued AJCs were not designed, staffed, or funded
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
65
to provide the necessary “level of individualization” or “intensive one-on-one services” to this
population. Still, DEI facilitated service coordination to engage AJC staff and JSWDs.
Another treatment WDA served clients with substantial barriers to employment,
including those who were deaf or hard of hearing, blind or visually impaired, and facing mental
health challenges. According to the DRC, “nobody [at the AJC] knew about MCDHH, [MCB],
or working with blind and deaf individuals” prior to DEI. MCDHH supplied interpreters and
signers, and MCB collaborated with WWL and employers to implement accommodations such
as JAWS screen readers. Also, MCB committed to providing additional resources for AT to the
AJC while MCDHH raised awareness about AT for hard-of-hearing individuals, and both
agencies offered TA on using AT. The DRCs also trained JSWDs to use JAWS to attend AJC
workshops. In addition, the AJC career seminar and customer flow were transformed to be
universally accessible, and the disability services planning chart was changed to reflect that of a
typical jobseeker.
Another treatment WDA similarly “mainstream[ed]” JSWDs through all employment
processes. The DRC from this WDA suggested that CP completion and placement rates for DEI
clients were high because their journeys virtually mimicked the employment processes of WIOA
Adult clients. To facilitate this integration, the DRC conducted staff trainings and served as a
“co-case manager” for JSWDs to model to staff how to secure accommodations and other
employment-related disability services so customers were “suitable for training.” Co-
management of cases also facilitated integration of DEI practices into WIOA services.
This site also tailored multiple workshops for persons with intellectual and learning
disabilities. In terms of accommodations, the DRC arranged for a large keyboard and
magnifiers/readers in every AJC room with a computer and secured captioned telephones. This
AJC also installed adjustable desks and tables and lowered bathroom mirrors. The AJC did not
procure JAWS, however, based on a recommendation from MCB. The DRC also expressed a
desire to have AT to assist people with learning disabilities like dyslexia.
Another treatment WDA featured partners that illustrated universal design enhancements,
as well as an educational partner that was convenient to public transportation and offered
inclusive, student-centered trainings. Its instruction was hands-on, and JSWDs were provided
much testing preparation and could audit classes. Job coaches also spent extra time with anxious
trainees, many of whom had been employed for many years. The third treatment WDA involved
trainers who supplied “extra information” and allowed JSWDs to observe classrooms to be more
comfortable. Low-vision individuals were sent workshop handouts so they could absorb the
information with JAWS prior to the workshop. With reference to DRC and staff capacity to serve
JSWDs, the competitive rebidding of AJC management and staff turnover led to a delay in
implementation and presented obstacles throughout. Two treatment WDAs transitioned to new
operators during Round 5, and the other WDA saw AJCs close, staff laid-off, and “essential
workshops” cut.
Another treatment AJC referred Ticket customers for assignment after intake. In the third
treatment WDA, there were not many Ticket-eligible customers among enrollees, and those who
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
66
were eligible already assigned their Tickets. Moreover, this WDA did not have sufficient staff
capacity to administer TTW, including due to turnover.
Locally, one DRC led whole-staff training “all the time,” such as by presenting about
accommodations at staff meetings. An AJC manager credited the DRC with improving disability
etiquette among staff. To accomplish this, the DRC was aided by a train-the-trainer session with
WWL. In another treatment WDA, a DRC joined the grant midway and received a bevy of
training from NDI, ICI, WWL, and the BLN to get up to speed. She also attended trainings on
the Help Wanted Online Index and TORQ systems, which facilitated advanced job search
techniques that prioritized labor market information. AJC staff were also trained on nascent
assistive technologies. The third treatment WDA arranged for full-day, all-staff trainings devoted
to engaging JSWDs. This WDA’s AJC held joint trainings delivered by ICI and another local
provider, including on cultural competence. Upon transitioning to a new operator, staff were
trained on serving JSWDs.
Training to buttress capacity-building was supported by the state’s rich TA infrastructure,
exemplified by ICI and WWL. Early in the grant period, ICI and WWL conducted trainings on
serving JSWDs, TTW, and employer engagement. ICI and WWL also provided targeted
assistance to sites on accommodations, employment placement and supports, self-disclosure,
motivational interviewing, career assessment and pathways, universal design, and disability
awareness. NDI webinars and tutelage were also pivotal for DRCs to learn about “the world of
workforce development.”
Treatment WDAs prioritized customer choice, but they also stressed pragmatic career
planning. One DRC submitted that “everything needs to be person-centered.” This AJC
completed interest and skills inventories with each client, and the DRC met with JSWDs
individually. If a client decided to undergo training, he or she typically took the TABE, which
was used diagnostically:
“It’s not the score you get that really matters … but you’ll know whether or not you could finish
a training, or that you’ll struggle ... so what can we do to help make your skills work for you so
that you can be able to go?” — AJC Staff
In another treatment WDA, the AJC utilized CE strategies to develop paid work
experiences that reflected each individual’s abilities and goals. The DRC also partnered with
WWL to help JSWDs envision opportunities “out of their comfort zone.” A JSWD confirmed he
“definitely” led his employment search with the AJC. The DRC “constantly” asked him which
opportunities he “could and wanted to pursue,” including his fields of preference and willingness
to travel. The DRC also regularly solicited his “feedback on whether we were going in the right
direction and reminded me I should be driving the search for my own employment.” He never
felt his “hand was being held.”
The third treatment WDA also centered on JSWD strengths, interests, and personality. At
the same time, plans had to be “goal-oriented,” prioritized, and geared toward addressing
barriers. An individual career plan was formulated based on detailed assessments, which
included identifying preferences in work setting (e.g., in a nursing home or hospital), type of
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
67
occupation, and work schedule. DRCs stressed client commitment and a realistic evaluation of
what outcomes clients could achieve; for instance, those who could not “comprehend a patient
care plan” were “not a good match” for the CNA profession. Another DRC from this WDA met
frequently with a business services representative to discuss whether client profiles would match
available job opportunities.
A DRC and community-based provider were proactive about accommodations during the
assessment phase: “We believe anybody can get a job. How they get there is different, and some
people need more significant supports.” JSWDs with minimal work experience were especially
encouraged to self-disclose “because it makes the job coaching when they first start working
much easier to negotiate.” As with most grantees, there was scant evidence of JSWD
involvement in strategic planning. Nevertheless, a WDA director and DEI leader divulged being
hard of hearing, “nearly blind,” and diagnosed with diabetes and a heart condition.
DRCs balanced case management with coordination and collaboration activities. Some
saw case management as their main responsibility, but they also made a concerted effort to
understand and forge systemic coordination. One DRC who joined the grant midway initially
saw it as “very overwhelming. … There wasn’t a full-time person so I just came in and learned
all of the pieces.” Over time, however, she cultivated partnerships and built the AJC’s capacity to
serve JSWDs. Capacity-building was expedited by all-day staff trainings on serving JSWDs and
other DRC functions, including via the “DRC I” training, some of which were open to staff
beyond the AJC. During our site visit, the DRC reported that many JSWDs were served and
placed without her intervention. At the same time, she advised that a DRC role should be
permanently funded at 75 percent full-time equivalent to sustain expertise, partnerships, and
accommodations forged during DEI. In a post-grant call, leadership from this WDA asserted that
cross-training and IRTs led to other staff internalizing how to leverage resources to serve
JSWDs. These leaders also approached DEI as a lever for systemic change, such as around
accessibility. In another treatment WDA, the DRC approached her role with an “exit strategy”
and strove to make “really sustainable changes.”
DEI and its employer partners provided child care, work uniforms, and a phone that
JSWDs used to communicate with employers. Employers supplied other employment support
services, including practice with alternative interviewing styles and strategies to help JSWDs be
“more at ease with selling themselves.” JSWDs were prepared for OJT scenarios, including
through coping strategies.
34
Employer engagement was integral to DEI implementation across treatment WDAs, with
WWL and business services as key linchpins. WWL facilitated several employer outreach
functions, helped identify employers hiring JSWDs, and arranged for JSWDs to tour employers
through events like Disability Mentoring Day. WWL’s annual career fair featured dozens of
employers and was well attended. Additionally, WWL invited two DRCs to a BLN meeting to
promote DEI and job candidates. JSWDs could also access WWL’s virtual employer bank and
assisted with placement and sensitivity training for employers. Business services staff earmarked
34
Centre for Studies on Human Stress. (n.d.). Coping strategies. Retrieved from https://humanstress.ca/stress/trick-
your-stress/steps-to-instant-stress-management/
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
68
and engaged industry sector employers throughout treatment WDAs and were also central to
navigating turnover within employer partners.
One treatment WDA initially invested much time in employer outreach since there was
limited engagement there prior to DEI and WWL’s BLN was less established there. This WDA
focused on engaging businesses in manufacturing, health care, hospitality, and banking. WDA
leaders also engaged a manufacturing staffing agency to explore JSWDs participating in on-site
trainings and paid work experiences. The agency’s director encouraged JSWDs to apply for the
many local openings, especially since new transportation options emerged to a local
manufacturing site. While the site typically only offered 12-hour shifts, it indicated it would
consider splitting shifts for the right candidates.
DRCs engaged employers in various ways, such as inviting employers to the AJC to
conduct mock interviews; attending career fairs, human resources events, and a WWL-sponsored
BLN event; taking company tours, including of a manufacturing facility; and meeting with
employers to deliver résumés and discuss staffing needs and candidates. A DRC also met with a
local economic development director to connect with employers. Additionally, she arranged for
MCB staff to meet employers and train them about accommodations and to visit and assess work
sites. The WDA chairwoman and business services also engaged employers, including through
presentations at career fairs and trainings.
One JSWD suggested that DEI funding for employment-related internships was helpful
in persuading employers to hire JSWDs. However, he also indicated he could benefit from more
assistance in reaching out to employers and leveraging his own connections; the DRC primarily
sent him online applications for openings. Despite these employer engagement efforts, the DRC
argued that employer bias remained a significant obstacle, particularly if an individual had a
visible disability or was deaf or blind: “I think it’s a huge challenge to get some employers to see
the person and not the disability.” Another hurdle was churn and restructuring within employer
partners; persistence and constant networking were critical to overcoming these transitions.
In another treatment WDA, the DRC worked “really strongly” with WWL to incubate
employer bonds. This synergy resulted in a robust partnership with a local hospital, which led to
many internships and placements. This AJC’s operations manager asserted that the DRC knew
local employers well, worked closely with job developers, and was significantly involved with
job placements. In addition, a job placement specialist from VR ran a regional employment
collaborative. The collaborative met monthly, providing a venue for JSWDs to network with
employers and employment service providers.
In the third treatment WDA, WWL and business services spearheaded employer
engagement. This partnership coordinated multiple job fairs for JSWDs, including for health and
human services and hospitality occupations, as well as one-on-one meetings with employers
about candidates. Like in another treatment WDA, DEI in concert with WWL facilitated a
relationship with a local medical facility that led to a fruitful Career Pathways pipeline. DEI also
helped cultivate links with many other employers, such as nursing homes and Sodexo, which
provides facilities management and food services for schools, universities, and hospitals.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
69
A provider and lead partner in this WDA interacted with employers in furnishing
“placement support” services, which included navigating disclosure, résumé development and
distribution, setting up interviews, accommodations, and job coaching. The provider encouraged
employers to allow its clients to attend “regular orientation” without accommodations, but it
also conducted “informational interviews” to help clients practice seeking accommodations.
These interviews were particularly useful for youth; the DRC argued youth may be less inclined
to engage in self-advocacy with employers since they increasingly grow up in inclusive and
accommodating settings: “That’s a big awakening. … They don’t identify as having a disability,
but some of them definitely will need extra supports at work. Most of the time when they fail it’s
because they didn’t want to self-disclose.”
Once a client began working, employers were urged to provide specific feedback on
challenges and clients were observed in action to customize job coaching. Stakeholders also
discussed how it was crucial to understand employers’ perspectives on accommodations,
including allaying fears about confidentiality and liability and gauging their openness to on-site
job coaching. One employer accommodated a trauma-affected JSWD by enabling her to work
near child care. This WDA also leveraged job coaching funding from statewide employment
services to furnish on-site employment supports for two autistic clients working at Sodexo. In
each treatment WDA, leading employers were central to long-term strategic planning for
pathways customized to JSWDs. One employer partner and BLN member was a “vocal
advocate” for businesses to collaborate with DEI. The state’s overall proficiency with engaging
employers was evidenced by its leading development of a business engagement Community of
Practice presentation for NDI in concert with WWL.
Career Pathways v.
Two treatment WDAs demonstrated elements of CE that helped individuals enroll in CP
training. A provider and lead partner in one WDA were very active as liaisons between JSWDs
and employers. This included customized job coaching based on employee needs, employer
preferences regarding work opportunities, and the workplace setting. Employer suggestions for
task reassignment were common, and the provider adapted its coaching to these preferences.
Another treatment WDA explored flexible scheduling to enable JSWDs to apply at a
manufacturer that used 12-hour shifts. This WDA also customized an internship for one
individual at a café so that he could work and play music at an “alternative” place with a “small
community feeling.”
In another treatment WDA, the DRC described cost-sharing as “the best way to
encompass all of the services that an individual may need.” She blended and braided funds with
MCB to prolong internships, as well as to assist with supports like clothing and obtaining a
driver’s license. The third treatment WDA leveraged resources from MRC and the Department
of Developmental Services. This enabled it to offer more individualized retention services,
including extending onboarding and providing transportation supports. The lead partner and
provider also engaged local colleges to seek funding for individualized supports that would
ensure that individuals had access to CP training. For example, a JSWD whose reading is not at
an eighth-grade level will likely not flourish in a patient access and registration class. To upgrade
basic skills, a DEI AJC offered a “Career Ready 101” online curriculum. The AJC was proactive
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
70
in identifying possible accommodations. Despite these tactics, treatment WDAs still faced
obstacles identifying JSWDs who could commit to and complete CP training. Challenges
included chronic health problems, such as terminal and mental illnesses, “ex-offender issues,”
and refugee statuses. ICI and WWL were consulted to address these hurdles and to help identify
suitable candidates. Disclosure was another challenge, especially for those with mental health
disabilities.
A DRC contrasted DEI with a prior system that funneled JSWDs to VR rather than
“trying to really figure out what the JSWD actually needs … Maybe they’ve been working with
VR and it’s not been a good process.” The AJC also had much more access to employers than
VR, could intervene early in the job search process, and was not hamstrung by the wait time or
inflexibility that characterizes VR: “You better go to VR knowing what you want … because
they’re not going to open up the menu of services and tell you, ‘You have this option or this
option.’” The DRC also distinguished the AJC from SSA’s customer service and suggested “it is
crucial to assist JSWDs so they can avoid SSA’s disincentives to work.”
JSWDs corroborated that the AJC and DRC balanced customer choice with realistic
options. One was transformed from feeling “lost” to learning about stackable credentials and
studying the local labor market and projected growth occupations to match with her skills,
background, and disability. She also stated that JSWDs “have to put some work into it too.”
Another individual relayed how the DRC built upon her desire to be a nurse by encouraging her
to pursue a medical internship, which opened her to “a new world” and “a dream come true.”
After her internship, she secured a full-time temporary job. The DRC considered the JSWD’s
aspirations and abilities. She was “always positive” and offered sound step-by-step advice. This
JSWD deemed the AJC as very helpful, including its career fairs, and referred her peers to the
AJC. A third participant attested that the DRC was “ready to help me with everything I need” and
helped her secure a medical internship.
With respect to Career Pathways, treatment WDAs focused on health care (e.g., CNA,
medical assistant, home health aide, sterile processing assistant, surgical technology, and central
processing), culinary arts, hospitality, and manufacturing (e.g., machinists). Hospitality trainings
were generally of shorter duration and led to quicker placements.
DRCs facilitated various customized assessments and utilized state-sponsored CP tools
and O*NET OnLine. AJC enrollees also had to complete the state’s mandatory Career Center
Seminar. One DRC conveyed how CP-specific assessments could shape a JSWD’s employment
journey. For instance, those with developmental disabilities could struggle to pass the entrance
examination for CNA trainings; following directions sequentially and being able to comprehend
a patient care plan are crucial to being a CNA.
For the lead partner and provider in this WDA, individualized assessment was essential to
identifying appropriate supports. Assessments occurred in a comfortable setting: “We have their
assessments, so they can do it untimed in a relaxed place.” Assessments gauged strengths and
barriers to employment, informed strategies to tackle barriers, and determined whether assistance
was needed “to get fully trained.” After a series of technical and readiness assessments,
individualized supports were developed. A “commitment assessment” helped determine how
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
71
equipped a jobseeker was to enter a pathway. Assessment also honed in on the client’s ideal
work situation:
“Are they going to want to be working in a nursing home or in a hospital? Do they want to work
in a home-based program as a home health aide? If they’re doing culinary arts, are they
interested in working in the cafeteria? Do you want to work in a restaurant? Do you want to
work Monday through Friday?” — DRC
This provider also delivered 30 hours of job readiness instruction before clients enrolled
in CP. Training focused on soft skills like time management, professional etiquette (e.g.,
documenting absences and workplace communication), socializing at work, and dress codes.
This training period also allowed clients to obtain “extra supports,” such as for transportation.
In another treatment WDA, the DRC used various assessments to “see where they are,
where they’re coming from, and where they want to be,” including the Massachusetts Career
Information System, Holland Self-Directed Search, the Work Readiness Assessment, and TORQ.
The DRC also arranged career development coaches to help “calm fears” about starting a course
or taking a test.
The third treatment AJC’s assessment process included a Career Directions workshop
that offered interest and skills inventories, as well as TABE testing. During pathway trainings,
DRCs helped request academic supports at community colleges, such as one-on-one tutoring and
group sessions to prepare for certification exams. One DRC also advocated for participants to
retake exams multiple times. Another DRC attended information sessions at community colleges
with JSWDs to help them feel more comfortable with college staff. This WDA also met with a
college’s coordinator of disability services, services that included accommodations (e.g., note-
taking) and services provided by the school’s “Visions”
35
program. This meeting led to other
connections between the AJC and the college, and some Visions attendees were trained through
DEI.
One treatment WDA nurtured a fruitful partnership with a local hospital, which led to
numerous successful internships, paid work experiences, and placements in patient access and
registration. According to multiple data points, “nearly anyone” who completed an internship
with this hospital quickly landed a job. Moreover, these positions were often full-time and long-
term. The DEI State Lead indicated that two DEI graduates of this training were offered full-time
jobs starting at $16.67 per hour; both were long unemployed and earned just above minimum
wage in their last jobs. One of the individuals was a TTW beneficiary.
One WDA did not have a prosperous relationship with the CVS internship program. DEI
funded retail and customer service internships through CVS. At least six participants graduated
from the program, and CVS provided positive feedback, yet none were hired. The DRC and a
partner opted to end the partnership with CVS. Nevertheless, this WDA exceeded grant
objectives according to the DEI State Lead, as they achieved success with multiple TTW clients.
35
Visions supports disadvantaged students to persist through graduation and helps with transferring to 4-year
college institutions. The program provides a variety of services, supports, and accommodations.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
72
One gained an associate’s degree, finished a nursing program, and was considering becoming a
registered nurse. She was in the interview process and withdrew from SSA and SNAP benefits.
In another treatment WDA, DRCs help JSWDs focus on “the end product” from the
beginning; “everything we do has to be employment-focused.” With DEI, this WDA built upon a
grant from the state’s Commonwealth Corporation to continue a partnership with a local
community college. The state grant focused on CNA, an allied health, and culinary arts
pathways. This WDA also coordinated a summer boot camp for youth interested in culinary arts
or CNA pathways. The camp focused on work readiness, such as résumé writing, networking,
interviewing, and learning styles.
DEI and WWL helped this WDA develop its CP infrastructure further during Round 5.
WWL facilitated a meeting with a local rehabilitation hospital, and the WDA began placing
CNAs there. “The relationship grew,” and the hospital created a customized CNA training for
JSWDs. With additional funding from a state grant, the hospital committed to hiring any JSWDs
who completed the credentialed training. A lead partner from this WDA hoped to convince the
hospital that it could recruit and train all the hospital’s prospective CNAs, which would cost
approximately $5,000 per participant.
The third treatment WDA was innovative in helping JSWDs pursue a variety of
pathways. The WDA trained DEI participants for the same fruitful hospital pathway as another
treatment area. This involved much collaboration with hospital managers, MRC, and a
community college. The college offered a customer and patient service certificate program,
which the WDA also suggested could be useful for banking pathways.
This WDA also cultivated a relationship with the volunteer coordinator at another local
hospital. This bond enabled the AJC to refer JSWDs for volunteer opportunities, which provided
access to internal openings before their broader advertisement and a reference from the
coordinator. In addition, this WDA aided JSWDs in pursuing careers in health care,
manufacturing, culinary arts, and IT. As well, JSWDs secured customized internships, such as at
a video game store and with the United Way.
MCB was a key partner in this WDA’s employment and placement strategies. Several
JSWDs were co-enrolled with MCB, and MCB provided additional internship funding for clients
who needed more work experience. MCB was also pivotal to ensuring that AT was compatible.
Regarding Work-Based Learning, one DRC described it as “everything to me” because it
could “make a huge difference” for the long-term unemployed, those who had to change careers,
and those with little experience. This DRC asserted that combining WBL with short-term
training was “really helpful.” However, this WDA did weather challenges with WBL due to
Boston city requirements related to the provision of stipends that were not expected to be
provided to JSWDs. Another DRC valued aspects of WBL, such as Job Shadowing and
employer tours, which provided a “hands-on feel” and insight into a typical work day.
Multiple DRCs commented on the value of “post-placement support” to promote CP and
job retention. One treatment provider arranged individualized supports for independent travel
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
73
and self-regulation. According to a DRC, self-regulation (e.g., being on time and working
without assistance) was a primary challenge for youth JSWDs. This DRC also indicated that a
“shortfall” of DEI was its lack of resources for ongoing support and retention. This complicated
serving those with the “most significant barriers to employment.”
Notwithstanding successful facilitation of diverse pathways, treatment WDAs endured
multiple obstacles related to CP. One WDA struggled to identify candidates who could complete
the CNA training. Additionally, this AJC engaged youth referrals who were trained but “lacked
the commitment to stay employed.” Conversely, an AJC director from another treatment WDA
suggested that most DEI opportunities were for lower-skilled individuals and did not align with
higher-skilled JSWDs in that area. Another challenge was finding appropriate trainings that were
certified in TrainingPro, the state’s training administration program; “Many effective trainings
were not certified, and the system’s interface is insanely tedious.”
Outreach and Dissemination vi.
In terms of outreach, DEI was kicked off with a press conference featuring congressional
representatives. One treatment AJC advertised DEI through email blasts from its database of
jobseekers — cultivated through relationships with counselors and recruiters — as well as
marketing materials and word of mouth from recent program graduates. A DRC here dubbed
himself “a marketing motor on wheels” and plugged DEI frequently. To recruit for its boot
camp, a provider and lead partner leaned on referrals from schools and local disability service
providers.
In another treatment WDA, the “lion’s share” of DEI referrals were AJC customers. In
addition to also receiving clients from MRC and MCB, the DRC conducted outreach through
various channels: engaging a coalition of social service agencies; attending career and veterans
resource fairs and youth council meetings; and visiting high schools and providers. Job-specific
recruitment posters and other dissemination materials were also distributed at Disability
Employment Coalition meetings. The third treatment WDA’s DRC was described by career
counselors as “very active in the community” as she advertised DEI services to JSWDs.
With respect to dissemination of best practices, treatment WDAs primarily shared
promising innovations and success stories through internal AJC trainings, monthly meetings
convened by the State Lead, and written documentation. Other dissemination vehicles included
videos, such as an on-the-job filming of a participant that was circulated statewide and shared
with USDOL. NDI and ICI also helped Massachusetts identify promising strategies to share
broadly via video and helped broadcast the state’s Access for All manual that was produced with
DEI funds.
Promising Practices: “Pathways for Success” vii.
DRCs and other DEI leadership presented about best practices and “strategies for
replication” at WWL conferences and other events. Topics included IRTs, career assessment
tools, enhanced benefits counseling, TTW, interagency collaboration, capacity-building, and
employment readiness for JSWDs (including those in substance abuse recovery). The positive
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
74
reception to this latter presentation signaled that the AJC could garner referrals for JSWDs in
recovery. Additionally, DEI leaders from one WDA, including the executive director of the
WDA’s board, presented at a National Association of Workforce Development Professionals
conference.
Rationale for Its Implementation
AJC staff, DRCs, and Employment Specialists needed information on promising
practices and core strategies for meeting the needs of JSWDs, such as Section 188 of WIOA, that
were integrated with the needs of youth with disabilities and Career Pathways developed through
cross-agency partnerships.
Why the Practice Could Be Considered Promising
DEI aided in transcending Section 188 requirements and in conceiving accessibility as a
long-term, “systems change” endeavor. WDA leaders shared practices in conference calls with
WDAs in other states. One DRC co-presented with MRC on panels. The state’s extraordinary
emphasis on dissemination was epitomized by its second DEI Best Practices Conference.
Challenges and Sustainability viii.
Massachusetts Round 5 presented multiple implementation challenges. Structural barriers
included dysfunction precipitated by two treatment WDAs transitioning to new AJC operators,
which led to staff turnover and layoffs as well as elimination of workshops. Insufficient
resources complicated provision of accommodations and long-term supports and optimization of
AT and accessibility. TTW did not supply predicted revenue to supplement a dearth of resources,
and treatment AJCs struggled to overcome the complexities and resource investments associated
with establishing ENs and facilitating Ticket activity.
Treatment WDAs also weathered difficulties matching JSWDs with planned pathways,
including due to the inflexibility of the state’s training administration program. Even with
additional funding and resources, individual barriers were often too significant for treatment
AJCs to facilitate completion and retention of Career Pathways training. Employer biases may
have limited access to employment in some areas, especially for those with a visible disability,
deafness, or blindness. Finally, though treatment WDAs were innovative in addressing
transportation challenges, physical access to services and jobs remained a quandary.
In one treatment site, a DRC retired with 9 months remaining in the grant period. This
WDA also experienced multiple instances of turnover in personnel responsible for grant
administration and implementation. A new DRC at grant’s end required training and had to
rebuild trust with clients. He also suggested that JSWDs were not able to participate in training
in part due to the operator’s transition to another job. In another treatment WDA, there was also
turnover in the DRC role and many layoffs. The DRC from the third treatment WDA was hired a
few months into the grant. This DRC experienced a change in supervisor and left the position in
the final grant year.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
75
Treatment WDAs navigated numerous roadblocks in incubating TTW capacity. ICI
supplied initial TA on EN creation, nurturing partnerships, Ticket recruitment and assignment,
and staff training, while WWL helped with benefits counseling and employer outreach. One site
was an established EN, while the other two were undertaking suitability clearance processes
during the grant period. Quarterly reporting toward the end of the first grant year described
access to a new TTW portal as “time-consuming and confusing … with little guidance from
Social Security;” not all portal features were accessible. There were also delays in two treatment
WDAs establishing an EN; at least one did not become an EN by grant’s end. In addition,
turnover in AJC operators and staff contributed to lower levels of Ticket activity than expected,
prompting intervention from ICI.
One DRC suggested that successful local TTW programs in Massachusetts were in
locations where there was not a strong relationship between VR and DCS. Her reasoning was
that collaboration with VR led to more referrals and ceding Ticket ownership to VR rather than
keeping it locally within the WDA. Most Tickets were processed centrally through the state. This
AJC helped some JSWDs engage with TTW and secure employment. Toward the grant’s end,
the DRC offered regular office hours for work incentive counseling. However, this DRC left the
grant early and the WDA’s TTW clients were transferred to other providers.
Regarding practices to encourage sustainability, DRC roles ranged from primarily case
management to a combination of case management and systems-level activities. One DRC
preferred “one-on-one” engagement and operated “like a one-stop shop … he sticks with the
person all the way through,” including recruitment, intake and assessment, training, placement,
and retention support.
“We’re thinking about how to make sure staff are better equipped at making sure people with
disabilities are given access here, that we are a welcoming place, that we are a place that is fully
available to this community.” — DRC
To facilitate this transformation, the DRC provided TA to staff and was available as a
resource for staff serving JSWDs to troubleshoot or assist with accommodations. She also co-
managed cases with other staff to model service delivery for JSWDs. And, despite much
attention to training and capacity-building, the DRC met one-on-one with JSWDs “all the time,”
especially those who needed “more intensive counseling.” She was also intimately involved with
job placements. According to the AJC operations manager and career counselors, staff were
much more knowledgeable about engaging JSWDs and accessing accommodations and partners
and not as reliant on MRC. However, counselors admitted it would be difficult to take on all
DRC responsibilities. The AJC director also mentioned efforts to “institutionalize” DEI
practices, including train-the-trainer trainings, but suggested there would be a “gap” without a
DRC. This DRC left the grant before it ended to join NDI, but her replacement testified that staff
continued to work with JSWDs. ICI furnished TA as the grant ended to expedite new staff
learning DEI practices, particularly in the treatment WDAs that experienced significant turnover.
With respect to financing to prolong DEI practices, one treatment WDA garnered a
2-year state grant to continue building its health care pathways. However, a DRC from this
WDA argued it was imperative to allocate funding for ongoing, long-term supports for JSWDs.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
76
The AJC director from this WDA also lamented dwindling federal and state funding for
personnel, though TTW could help with some of this void. A DRC from another WDA stated
that funding to sustain DRC practices was “in limbo;” TTW revenue did not meet expectations.
Another DEI stakeholder offered that TTW revenue could help support DRC functions or
support services, but not both. Blending and braiding will persevere, especially since MRC and
MCB are “MOU partners” with greater investment in DCS services.
Though continued funding is tenuous outside of one WDA, many DEI practices and
partnerships should sustain. Accessibility enhancements and capacity-building for serving
JSWDs in each treatment WDA will persist, as will the IRT model. Trainings on universal access
continued through the end of the grant, and one treatment WDA’s leadership reported that
trainings on serving JSWDs were continuing post-grant. Relationships with MRC, MCB, and
local colleges should endure and serve as a steady referral stream as multiple stakeholders
attested. One DRC added that the Department of Transitional Assistance and ABE being more
integrated into AJCs is another significant systems change. One WDA’s Disability Employment
Coalition was expected to continue meeting and networking:
“It’s more advantageous for everybody to share resources and hear what’s going on. That way
you can provide the most holistic experience for who you’re working with. So I don’t think that
will just end because disability isn’t going to end.” — DRC
A post-grant call confirmed that the coalition planned to continue meeting quarterly,
which will solidify bonds with employers. DEI also paved the way for internship pipelines with a
local hospital, which was previously averse to new internship programs. Local banks and
financial institutions and manufacturing staffing agencies are also emergent employer partners in
one treatment WDA as a result of DEI.
In one treatment WDA, DEI lessons were leveraged for another grant. Moreover, ICI’s
model documents, policies, videos, and other dissemination materials related to DEI should help
with institutionalizing promising practices. Finally, the DEI Best Practices Conference was cited
as very helpful to sustaining accessibility, universal design, and other DEI features. Materials
from the conference are electronically accessible.
Notwithstanding structural obstacles in each treatment WDA, Massachusetts
demonstrated innovation, systems change, and promising practices. DEI implementation greatly
benefitted from robust alignment between state and local leadership and among treatment
WDAs. ICI and WWL also provided pivotal TA, including with accessibility, employer
engagement, and TTW. Round 5 significantly bolstered collaboration with MRC, MCB, and
other state agencies and also forged partnerships with local colleges, hospitals, banks, and other
employers that led to placements and long-term pipelines. Treatment WDAs also exhibited
promising strategies with customized assessments and job supports, outreach and partnership
development, staff training for sustainability, dissemination, and employer engagement. In sum,
Massachusetts is well-positioned to continue nurturing bona fide Career Pathways for JSWDs.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
77
E. State: Minnesota; Focus Area: Adults with Disabilities
Stated Goals and Objectives i.
Minnesota’s Round 5 grant was designed to transform the state’s AJC and CP systems to
enhance accessibility, inclusivity, and competitive employment for JSWDs through embedding
“on-demand” supports and leadership, bolstering partnerships and leveraging resources, and
developing new service capabilities through innovation and professional development. DEI was
pivotal to piloting universal service models that could engage those not served by VR, including
through partnering with Title III and Title IV so AJCs could offer “more choices and no-wrong-
door” policy. Locally, these systems collaborated with employers, community-based agencies,
chambers of commerce, and other entities to build more inclusion in manufacturing, health care,
business, and technology. A “Career Pathway convener” in each treatment WDA aligned cross-
agency leadership and collaboration by building on the state’s Skills @ Work campaign, which
established regional industry action teams to engage sectorial employers and integrate CP with
stackable credentials. Ultimately, DEI helped to shift the CP infrastructure’s emphasis from
individual to system readiness to provide T/TA to each individual enrolled.
36
These strategies were designed to engender the following outcomes:
• Engage adults with disabilities in work-based and CP training approaches;
• Increase employment and wage outcomes of JSWDs:
o Increase the length of employment of JSWDs; and
o Increase 6-month earnings;
• Increase credential attainment;
• Facilitate CP program completion; and
• Engage employers to provide better services and outcomes for JSWDs through existing
Career Pathways systems.
DEI Service Delivery Strategies ii.
In treatment WDAs, JSWDs could access a range of trainings and pathways, including
education or work-based programs. Treatment WDAs also served jobseekers with diverse
disabilities, including those with traumatic brain injuries and paralysis. Correspondingly, each of
our JSWD site visit focus groups was heterogeneous. Data revealed that two WDAs intentionally
integrated JSWDs into workforce programs with the general AJC population.
WDAs regularly furnished AT to JSWDs. Each WDA had access to the state’s System of
Technology to Achieve Results, which features a lending library for individuals and employers
to try out assistive technologies, as well as professional development. One WDA facilitated AJC-
wide trainings by a state representative on the deaf and hard-of-hearing population, including
about AT resources. Also, treatment WDAs benefitted from trainings and resources provided by
PACER. In one WDA, AJC and partner staff attended a PACER livestreaming training. As with
36
Minnesota’s Round 5 project narrative.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
78
other states, universal design was not a primary focus of Minnesota. Universal design
considerations were integrated with Minnesota’s plan for AJCs to be more mobile and reliant on
other public spaces and partners for meeting venues.
Still, treatment WDAs demonstrated progress with universal design. Multiple learning
methods were common, including audio, visual, and tactile activities. One WDA assessed
learning styles to tailor instruction and utilized diverse assessment mechanisms. JSWDs from
treatment WDAs reflected upon how DRCs and other staff enabled customers to proceed at their
own pace and secured appropriate testing, curricula, learning accommodations, and AT. More
classroom breaks and sign language interpreters also exemplified universal design for learning
principles. In one WDA, JSWDs remarked about the flexibility of classroom instructors:
“I was just excited at how the instructors understood that most of the people in the room had
some type of issue and they sensed when we needed a break. It was okay to get up and walk
around. They helped you with adaptive equipment.” — JSWD
Another confirmed that instructors were “tuned in with the class:”
“If you had any issues, you felt so comfortable just going and talking to them — whether it’s the
fan that was too cold, or you were having a hard time seeing the board, or keeping up with the
materials, or whatever it was. It was just paced very well and you felt comfortable coming
forward and saying, ‘Hey, I'm having a hard time.’” — JSWD
One WDA relayed how it conducted CE “before we knew we were actually doing it.” It
facilitated workplace modifications and negotiated a wheelchair user’s job description to allow
for job sharing and to match his skill set. Moreover, VR in this WDA utilized Discovery and
received professional CE training. Another WDA customized a work experience for an enrollee
with “fairly severe” disabilities. The DRC also consistently engaged participants and employers
to customize hours and training for JSWDs.
Despite some initial staff and partner resistance regarding the use of IRTs — with some
divergence from fidelity — it became an integral SDS across sites. Co-location of many AJC
partners created IRTs that incubated “pretty easily.” Per a final progress report for the quarter
ending March 2018, Minnesota established 103 IRTs during Round 5, more than double its goal
of 44. Quarterly reports also noted increased usage of ILPs in conjunction with IRTs; final
reporting stated that DEI generated 97 ILPs, dwarfing the goal of 21.
In one treatment WDA, the DRC outlined the process of their IRT, which included a VR
counselor and ABE instructor. The DRC described herself as “the connector” between the
participant and necessary personnel in a “very fluid process.” In one case, the team met with a
participant who had completed an ILP, listened to his needs and aspirations, and then described
how each team member would contribute to helping him secure ongoing supports, complete
training and coursework, and get placed or have an opportunity to complete OJT. The IRT
successfully helped the client finish training and obtain a full-time job paying $20 an hour. Also,
a VR counselor explained how IRTs complemented the “individual placement and support”
approach of VR by enabling additional training and work opportunities.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
79
IRT meetings were focal to fostering collaboration between partners in another treatment
WDA. In addition to VR and ABE representation, IRTs here included public assistance and
Employment Specialists, as well as county social and financial workers. While such diverse
representation could be “a little overwhelming,” it was “really beneficial” for the client because
it provided one space to address requirements from different service providers. Further, “we were
able to accomplish a lot more than when we work separately.” A placement specialist also
valued IRTs as a “constant informational feed.”
Similarly, the DRC from another treatment WDA characterized IRTs as the “most useful”
SDS since they convened all relevant partners, reduced duplication, and enhanced blending and
braiding. She also depicted the unfolding of an IRT, which included a case manager, VR
placement coordinator, the participant’s parent, and eventually an employer. DEI funded a paid
work experience at a hotel, while VR handled placement and coordination of the work
experience with the employer. The participant was hired after the work experience at 30 hours
per week. This AJC also adapted the IRT approach to serve students with disabilities.
DEI also contributed to Blending and Braiding Resources throughout Minnesota. As a
state leader attested, “We braid money like it’s our jobs here.” Notwithstanding the state’s
prolific coordination of resources, blending and braiding could be challenging because these
processes required knowing which dollars could be integrated and with which outcomes dollars
must be associated. Moreover, relevant processes and requirements tied to different funding
sources had to be integrated into program design and there were not “clear policies” on co-
enrollment. Thus, co-enrollment could be perceived as “double-dipping.”
Despite these complications, treatment WDAs embraced co-enrollment. One WDA
underspent its training funds for much of Round 5 due to having other funding streams cover
training costs. In turn, DEI offered training funds in cases where WIOA case managers were not
“willing to take the risk quite yet” with certain clients. The DRC here also described a co-
enrollment with VR through which that latter aided with tuition and training while DEI supplied
an internship. Another treatment WDA also harnessed braiding to finance internships and
regularly co-enrolled. For one individual, DEI funds were used to fund job coaching, and VR
coordinated the services.
As with other states, there was scant evidence of DEI facilitating Self-Employment
opportunities in Minnesota. Minnesota’s Round 5 DEI also did not move the needle much
regarding Asset Development. One WDA’s IRT model included specialists devoted to
budgeting, and it also connected participants with community-based agencies that helped with
general assistance. Another WDA tried to secure financial aid for postsecondary tuition.
Regarding TTW, one WDA was distinctive among Round 5 grantees for its expertise. In
addition to the AJC serving as an EN — and considering becoming an administrative EN — it
featured two “Ticket Ambassadors” who led a Community of Practice on TTW and CP. The
DRC was also certified as a CWIC. This WDA learned about TTW during Round 3 and “was
brought in” for Round 5. State leadership described this WDA as TTW “leaders” and reported
how it was fielding TTW requests from customers 2 hours beyond their service area.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
80
In addition to TTW outreach sessions, the AJC’s quality assurance department
periodically scanned its WIOA caseload to identify beneficiaries and engage them about
assigning their Tickets. After completing a screening form and a discussion with AJC staff about
employment aspirations, JSWDs were encouraged to “make an informed choice” and assign their
Tickets if they were poised to become “self-supporting.”
To encourage sustainability, this AJC also engaged its TTW staff with its DRC training.
As of May 2018, this WDA nearly matched its TTW revenue goal of $100,000 by generating
$95,000 in milestone payments. The AJC had a productive relationship with VR but had not
become formally involved with Partnership Plus. Another treatment WDA became an EN during
Round 5 and its DRC became a CWIC. Altogether, Minnesota assigned 51 Tickets during Round
5, more than doubling its goal of 25.
The impetus for one WDA’s customized carpentry/construction training was that many
JSWDs could not pass the Comprehensive Adult Student Assessment Systems examination but
could still derive great benefit from CP training. This WDA’s Career Navigator also spoke about
adapting classes to participant backgrounds and needs.
In treatment WDAs, JSWDs were also equipped with information, services, and
integrated supports to lead their goal-setting, as suggested by universal design. Job clubs, mock
and motivational interviewing, holistic supports such as job coaching, and interactive activities
and icebreakers were common tools for reinforcement, retention, and socialization.
In one WDA, both staff and JSWDs spoke about how “the cohort model within the group
pathway” facilitated peer mentoring and support:
“We got to know each other and trust each other. We were given plenty of time to interact and be
with one another and to learn that we were a whole group. We weren’t just on our own at all. I
never felt on my own.” — JSWD
Some JSWDs who completed a class or program volunteered to return and help peers
who were enrolled, which was “tremendously helpful.” One JSWD admitted that without peer
assistance with Excel, “I don’t think I ever would have figured it out.” Others came across job
leads appropriate for their peers and shared them with the DRC and AJC staff to pass along.
Cohort members also exchanged phone numbers, carpooled, and “had our fun times.” As focus
group participants attested, these universal design features combined to optimize motivation and
conviction in employment aspirations and instilled the capacity to navigate adversity in job-
seeking journeys.
Minnesota treatment WDAs showcased advanced proficiency in multiple SDSs,
particularly TTW and CP. There was limited evidence of Customized Employment being
delivered with “full fidelity.” Each treatment WDA received “pretty intensive” training on CE.
However, according to a state leader, Minnesota’s VR was “struggling on the whole” with CE.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
81
Each treatment WDA was enhanced by a DRC and other key staff proficient in serving
JSWDs. State leadership highlighted one WDA’s superlative “leadership with decision-making
ability.” This was evidenced by the WDA taking initiative in promoting TTW and the DRC
training as well as “transforming their whole agency” to better serve JSWDs. This AJC also
conducted an employability assessment to address high-priority needs of JSWDs.
Implementation Summary iii.
To accomplish its goals and objectives, Minnesota prioritized several strategies. In
addition to strengthening state-level bonds with VR, DEI aimed to buttress links with State
Services for the Blind, Minnesota Department of Human Services, Disability Services and Adult
Mental Health Divisions, and the new Olmstead Implementation Office. An array of programs
and funding sources were synergized (e.g., WIA Titles, Carl D. Perkins Vocational and
Technical Education Act of 2006, TANF, mental health, intellectual and developmental
disabilities, transportation, housing, corrections/justice, aging, registered apprenticeship and pre-
apprenticeship programs). Moreover, “local CP partnership teams” united disability providers,
employers, and service systems to expand awareness of the value of hiring JSWDs and to
increase their engagement with paid work-based internships, apprenticeships, and learning
experiences. Through co- and dual enrollment, funds were blended and braided locally and at the
state level.
DRCs coordinated the implementation of IRTs and ILPs, including provision of
accommodations, alternative assessments and curricula, support services, assistive technologies,
and personalized strategies. IRTs involved CP partners (e.g., Minnesota Family Investment
Program/TANF employment service providers and ABE), VR counselors, supported
employment providers, CWICs, county social services, financial workers, 18–21 programs,
correctional probation, disability advocacy staff, community medical and mental health, veterans
staff, and others. DRCs facilitated “assertive outreach and engagement” and “person-centered
planning” (e.g., disability Benefits Planning, health care access issues, CE strategies) as well as
advocacy services. In addition, DRCs advised about quality improvements, including the
production of formal feedback surveys from JSWDs. This oversight was to complement
Minnesota’s role as a core partner in the Alliance for Quality Career Pathways. DRCs were also
to be “trained experts” with TTW and promote Ticket assignments, as well as connect JSWDs to
vital health care coverage, such as Medical Assistance for Employed Persons with Disabilities.
The state’s Disability Benefits 101 tool was also to be utilized to help JSWDs optimize benefits
and health care coverage while working.
Other features included the integration of universal design principles, peer mentor
supports, financial literacy education, and dual enrollment. Minnesota also improved data
tracking, such as by adapting the state’s Workforce One system to monitor DEI enrollees’
activities and outcomes. Tracked data was to include population descriptors, service types,
funding sources, utilization of CE and other specialized supports, and customer feedback.
Finally, DEI sparked a CP-focused messaging campaign that provided information about training
and employment.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
82
Minnesota Round 5 was jointly spearheaded by its Department of Employment and
Economic Development, WDA, and VR so as to integrate this work into state systems. DEI also
built upon multiple statewide CP efforts. The Joyce Foundation’s Shifting Gears Initiative, which
lasted from 2008 to 2012, galvanized the state’s FastTRAC Adult Career Pathways system that
coalesced education, workforce, and human services systems to achieve enhanced outcomes in
“high-demand careers” for low-skill, low-income adults. The state’s CP infrastructure “grew
exponentially” when state and local entities participated in the ETA Career Pathway Institute in
2010. These initiatives established both a state CP system and several local/regional CP systems,
with buy-in from the Departments of Employment and Economic Development, Human
Services, and Education, the state college and university system, and philanthropic and
community partners.
Round 5 featured three treatment WDAs, one of which was a comparison site during the
state’s Round 3 grant. One treatment WDA was in the rural southwestern part of the state,
another was in the Minneapolis metropolitan region, and another in Monticello. Minnesota’s
treatment WDAs exhibited high capacity to achieve positive employment outcomes. Physical
and communications accessibility were monitored by the state’s Office of Diversity and Equal
Opportunity. In addition to gauging the physical accessibility of AJCs, monitors scrutinized
policies and practices and initiated a “complaint” procedure process. Minnesota also connected
Office of Diversity and Equal Opportunity and DEI leadership at the state and local levels,
including through “a series of meetings” to discuss Section 188, AJC certification, and
dissemination of common trainings and curricula around accessibility. This work included TA
from NDI and the Governor’s Workforce Development Board (GWDB).
Programmatic accessibility was also spearheaded by state leadership and initiatives,
including equity grants and One-Stop operating consortia that promoted methods for universal
access to living wage employment. Minnesota’s WIOA plan integrated disability employment
throughout. Quarterly DEI meetings involving state and local leadership included discussions
about universal accessibility, such as DEI accommodations and AT policy and procedures, as
well as hands-on activities like a tour of the PACER Center’s Simon Technology Center.
Despite these efforts, treatment WDAs confronted barriers to physical accessibility. State
leadership intimated how WIOA funding constraints made it difficult to “sustain the bricks and
mortar” of existing facilities, stunting potential for accessibility and universal design
enhancements. Moreover, a lack of access to quality public transportation, child care, and
housing were cited by state and local leaders as significant obstacles. At the local level, Career
Navigators and other AJC staff addressed these hurdles prior to training, such as through the
provision of funds for transit, gas, automotive repair, and child care.
Partnerships and Collaborations iv.
Minnesota stands out due to a treatment WDA spearheading enhancements and
dissemination of the Alaska-model “DRC 1 and 2” trainings.
37
The model provided training,
37
The DRC training emerged from Alaska’s Round 1 DEI grant. It evolved to include graduated trainings on
disability awareness and effective practices for serving JSWDs and was delivered to AJC staff and other partners.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
83
often by DRCs, to AJC Employment Specialists and other partners’ staffs on the essentials of
employment assistance for people with disabilities. This WDA developed four DRC training
modules that covered facility awareness, disability etiquette, IRTs, CP, TTW, the Guideposts,
and universal design. Not only did this understanding become “embedded” in this AJC, but DRC
training was delivered to other Minnesota treatment WDAs, involved collaboration with
employment officers, and was shared with other states as a model. In addition to assessing
knowledge gained from this training, there was discussion of mandatory DRC training for new
AJC staff and biannual refresher training to foster sustained awareness. There were also plans
around expanding the training statewide. Further, Minnesota garnered T/TA funds from USDOL
to enrich the training curriculum, including by making it virtually accessible and integrating
state-specific resources. This training was also framed as a strategy for Section 188 and AJC
certification. As well, this WDA received “parent training” and other TA from PACER,
including about the Guideposts for Success.
Another treatment WDA offered training on youth transition and on harnessing AT,
accommodations, and disability services in postsecondary training. Treatment WDAs also
offered staff tutelage on accessibility, CP, and disability awareness. Across treatment WDAs,
respondents offered that NDI T/TA was instrumental in promoting TTW, DEI implementation,
and systems change. One staff member suggested she could benefit from training on engaging
clients with mental illness.
Integration was galvanized by state leadership, which was “very intentional” about
forging a “seamless interface” between VR and local WDAs . DEI being a “co-managed
initiative” in Round 5 signified much progress from Minnesota’s Round 3 and the Disability
Program Navigator initiative, which involved “a lot of pushback on the VR side.”
An emergent partnership with GWDB also enabled Workforce to be included on state
committees regarding disability and CP. As well, DEI strengthened relationships between
Workforce and State Services for the Blind.
State Leads facilitated regular calls among DRCs and separate calls with local and state
leadership as well as in-person quarterly meetings. These convenings were valuable to DRCs,
aided with “constant” communication, and helped disseminate visions of state “senior
leadership.”
Each treatment WDA featured co-location and improved collaboration with VR, which
expedited co-enrollment and helped secure additional funding for training and tuition. According
to one DRC, “we really complement each other in that we find ways to help clients be
successful.” Another attributed better collaboration with VR to focusing more on “our”
customers rather than “your” or “my” customers. The third WDA spoke of “much more
integration” between VR and CP, which built “a nice bridge to get more access” to supports.
Treatment AJCs also demonstrated much collaboration with ABE, local community and
technical colleges, and youth transition staff. These partnerships fostered information-sharing
regarding job contacts, including by engaging business services, and access to services like job
coaching and English as a Second Language.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
84
Locally, DRCs cultivated and strengthened bonds with entities within and outside the
AJC, including veteran services, adult rehabilitative mental health services, manufacturers,
employment and training centers for JSWDs, and community action agencies that helped with
needs like housing and energy assistance. In the more rural WDA, the VR employment
coordinator asserted that “partnering is essential because we don’t have a lot of resources here.”
Partners provided awareness and TA, such as through presentations by college disability
specialists. In turn, treatment WDA partners benefited from AJC trainings, such as PACER
workshops. Two DRCs were specifically praised by VR stakeholders for their exceptional
communication and collaboration, and one was noted for facilitating “very fluid” engagement
with employers and trainers. Collaboration was also advanced via IRTs, which included as many
as seven service providers.
A common theme across sites and stakeholders was “truly meeting jobseekers where they
are at.” This involved soliciting JSWD input on what supports they needed and helping them
“connect the dots.” One DRC applauded DEI’s “very individualized” approach, which was not
“one size fits all.” JSWDs met with the DRCs if they were interested in WIOA services and/or
CP training. AJC Employment Specialists were trained in motivational interviewing, which
helped uncover “what’s inside of JSWDs and what motivates them” and whether DEI services
might be appropriate. Employment Specialists encouraged JSWDs to follow their “dreams and
hopes.” This “genuineness and authenticity … allows [counselors] to move with [clients] as they
move in their lives.” Customer choice was also promoted through personality and interest
assessments, career exploration, and a “job search club,” in which JSWDs selected topics to
collectively probe information on employment and training opportunities. In addition, these
clubs explored job search processes, from résumé and cover letter writing to interviewing.
JSWDs from this WDA underlined how “we were here on our own accord” with “no
shame … no judgment.” Multiple participants also reflected on how their internships were ideal
fits; one proclaimed “it brought my life back.” DRCs also connected clients with
accommodations, including ergonomic desks, screen readers, and extra time to take tests,
enabling them to complete coursework and job activities. In addition, evaluations during and
after internships helped participants gauge their progress and areas for growth.
Leadership from another treatment WDA reflected on adjusting guidance for JSWDs
“based on their needs” and educational backgrounds and by “not assuming things.” This
approach led to more formal or more personalized CP planning. JSWDs from this WDA
discussed the benefits of peer mentoring, mock employment scenarios, and mental health
counseling.
Another DRC met with customers to ascertain employment opportunities and related
goals and receive assistance from Employment Specialists to secure resources necessary to attain
their objectives. DRCs also helped JSWDs pursue their desired Career Pathway, which meant
“striking a balance between encouraging them to think about next steps on their employment
journey and providing guidance without judgment when certain steps proved daunting.”
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
85
During our site visit, this DRC showcased her capacity to support JSWDs, including by
providing direct care for an individual who experienced a seizure. Focus group participants also
cited the DRC’s “excellent listening skills, supportive and resourceful nature, and ability to help
them through training by securing accommodations and supports.” One JSWD submitted: “The
DRC is very interested in our lives to help us get to where we need to go.”
In another treatment WDA, the DRC coordinated the AJC’s multiple “well-integrated”
supports for JSWDs, including from Career Navigators, VR counselors, economic assistance
workers, and resource room staff. This team furnished assistance such as career and college
counseling, including postsecondary accommodations; résumé, soft skills, and interviewing
support, such as supplying interview clothes; pro bono legal advocacy; and job coaching. Focus
group JSWDs affirmed that the DRC provided pivotal reinforcement throughout their
engagement with the AJC, which was key to retention and program completion. This impactful
encouragement was also crucial to the training progress of JSWDs in another treatment WDA
according to focus group participants. Led by the DRC and Career Navigator, JSWDs were
supported throughout their training, including through mock interviews by an employer. One
JSWD remarked, “I couldn’t have done this without them.”
Notwithstanding these promising capacities and developments, state leadership reported
how WIOA budgetary constraints prompted layoffs and less funding for resource rooms. Also,
one treatment WDA was reluctant to strive to become an EN due to insufficient capacity.
Minnesota made strides in coordinating its employer engagement efforts. It evolved from
a solo approach to employer outreach, through which numerous entities would engage
employers. The state streamlined the outreach process in each WDA and via a regional strategy
led by WIOA regional and local boards. Despite this challenge, a Federal Project Officer
commended Minnesota for targeting high-growth jobs and being responsive to employers. A
quarterly report also noted that training integrated local employer input.
Locally, one treatment WDA was lauded by state leadership for its coordinated
engagement that led to “stellar” relationships with employers. Entities like WDAs, ABE, local
colleges, and the local industry council held common meetings with employers and
communicated with one voice to avoid confusion. This synergy paved the way for a fruitful
partnership with a local manufacturing employer and an employment and training center for
adults with disabilities, which culminated in a carpentry/construction training program. Per the
VR employment coordinator, DEI also enabled the AJC to offer a financial incentive to
employers that was crucial to their involvement. The Career Navigator also conducted employer
relations and groomed many business contacts to ensure clients would have opportunities for
placement upon training completion.
In another treatment WDA, the DRC stressed the importance of nurturing relationships
with workplace supervisors “because that helps us to do placement.” The AJC was intentional in
fostering employer connections that could facilitate internships with a greater likelihood of
leading to long-term employment through laddering and latticing.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
86
City, county, and state government opportunities were identified as ripe for such
advancement. The DRC also created internships at a YMCA, a community college Office of
Student Affairs, and a nonprofit agency devoted to ending youth homelessness. A JSWD from
this WDA reflected on how the DRC “really advocated” to extend an internship, which was
crucial to the customer landing a post-internship job. Additionally, the DRC collaborated with
VR to plan a reverse job fair, though there was not significant participation from JSWDs.
Focus group participants conveyed that this AJC was effective in communicating to
employers about client expectations for internships, including ensuring that internships were
active, integrated, and enhancing skills. An employer interviewee also expressed much
satisfaction with her intern and echoed other interviews in professing that those with disabilities
can serve as extraordinary mentors once they have requisite skills and a sense of self-efficacy.
In another treatment WDA, placement specialists balanced interests of employers and
consumers, such as by conducting outreach to identify employer needs and job leads before they
were listed publicly. This AJC also partnered with employers to ensure job descriptions and
accommodations were an apt fit. One challenge was finding employers willing to participate in
OJT, work experiences, and Customized Employment.
Each WDA provided diverse internships, paid work experiences, and OJT through
employer partnerships as part of its CP offerings as well as access to postsecondary education
and ABE. WBL Opportunities were also available to JSWDs. Some pathways were more
traditional and underwritten partially by Pathways to Prosperity grants. Some enrollees attained
stackable credentials, such as advancing from a personal care attendant to CNA to trained
medical aide. Others pursued interests like scuba diving, audio technology, and community
interpreting. AJCs also aided individuals pursuing new careers, those who “really just want to be
employed and working,” and those seeking help just to get their “foot in the door” and gain
skills, résumé assistance, and workplace exposure. Work experiences were fitting for these
customers.
In one WDA, interests, values, and abilities were optimally aligned in planning for
further training and education. Another WDA utilized the Virtual Job Shadow platform as a
“one-stop shop” for career exploration. The platform includes interest assessments and the
capabilities to identify suitable occupations and educational institutions, observe the job being
performed, research potential salaries, and compose cover letters and résumés.
One WDA built upon a fruitful partnership with a local hospital, including by providing
work experiences in housekeeping, laundry, and dieting. The hospital visited the AJC to conduct
mock interviews with JSWDs. The DRC from this AJC also worked closely with the youth
program to facilitate pathways for young adults, including by helping with tuition. In addition to
helping with CNA and welding pathways for youth, the AJC facilitated a collaboration between
Project SEARCH, a local high school, and VR to establish health care work experiences.
Career Pathways v.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
87
With respect to Career Pathways, treatment WDAs forged a range of occupational,
training, and education opportunities for JSWDs. DEI involvement in the state’s GWDB CP
committee helped with this effort. CP progress was also advanced by TA, including completion
of the Career Pathways Toolkit: Readiness Assessment Tool in concert with CP partners.
Minnesota met most CP measures including engaging SSA beneficiaries in CP. In addition, the
state exceeded its objectives for its 6-month average earnings. However, reporting also indicated
that the state’s retention goals were elusive. Some JSWDs did not complete their CP program
and therefore may not have acquired the skills and knowledge needed to pursue gainful
employment.
Certifications were earned in manufacturing; health care, including CNA, automated
external defibrillator, cardiopulmonary resuscitation, home health aide, personal care assistant,
and medical technology; office and administrative technology; precision sheet metal; retail
management, customer service, and sales; welding; food ServSafe; and commercial driver’s
license. Strong partnerships with local community and technical colleges and ABE were “a huge
key to CP success” across the state.
In another treatment WDA with a history of robust CP training, most pathways began
with a Career Connections Workshop. This included employer tours, job searching, and
enhanced résumé writing tailored to specific jobs and skill sets. This WDA also offered specific
CP workshops focused on manufacturing and health care careers and planned “mini career
camps” for DEI clients to gain exposure in these sectors through hands-on exercises and a
college tour.
Typically, a “bridge component” provided instruction that included basic academic
subject matters and English language occupational information to help individuals earn CP
credentials. DEI clients could also enroll in an accelerated, customized GED program before
beginning training. Training was aligned with in-demand labor market sectors (e.g., sheet metal)
based on the six key elements of CP and in concert with GWDB and employers. Upon training
completion, DEI enabled paid internships. A job search club — initiated before training ended —
helped ensure customers were not “complacent” and were constantly thinking about employment
after training. “Regular, ongoing partner meetings,” including with Youth Services, were
integral to these processes.
A 2-year effort culminated in a training collaborative focused on carpentry and
construction. The network relied on DEI funds and included ABE, a community and technical
college, a manufacturer, and an employment and training center for adults with disabilities. ABE
contracted with the employment and training center to provide bridge programming, a Career
Navigator assisted with soft and basic office skills training, and a local college contracted with a
former supervisor from the employer to furnish hands-on technical skills training. Enrollees were
referred from the county, many of whom were unemployed and lower-functioning. The plan was
to place graduates in a cabinet factory with wages of $10 per hour and benefits if full-time. This
training will likely be replicated, as the WDA now has a curriculum and requisite equipment.
There was also discussion of how to build out this training to enable further credentials and
advancement beyond entry-level employment.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
88
Several Career Pathways and accommodations were available to JSWDs.
Accommodations were available via multiple chances to take tests, use of paper copies, and
tutoring. In one WDA, the DRC assisted in establishing a more seamless application process for
disability services at local community colleges, and staff were trained on AT for postsecondary
learning. Career Navigators, ABE, and resource room staff also addressed barriers and helped
maintain engagement, including by referring clients to support services. One WDA explored an
alternative curriculum for its health care pathway to make this avenue more accessible. Despite
these accommodations, one DRC stated that mental health barriers and lack of access to
transportation and child care could hinder pathway completion.
Blending and braiding were leveraged to finance work experiences and other pathway
opportunities. For instance, in one WDA 12 youth participants continued their paid work
experience after DEI’s end since they were co-enrolled in DEI and the Minnesota Youth
Program. In another WDA, a grant from the Walmart Foundation helped incubate retail
pathways.
Outreach and Dissemination vi.
DEI outreach and dissemination of effective practices happened through various means in
Minnesota. Principally, regular DRC meetings served as critical information-sharing spaces.
These convenings were a springboard for planning capacity-building trainings across grantees
and beyond. Communities of Practice also functioned as dissemination vehicles, and one WDA
spearheaded adoption of the DRC training across and beyond the state. The DRC training
involved train-the-trainer sessions to augment its transfer and sustainability. Social media, flyers,
and word of mouth were also outreach mechanisms.
At the state level, leadership was deliberate about sharing best practices with grantees:
“It’s being timely about when we jump is what we’ve learned. Pushing too much, too fast — at
the wrong time — doesn’t help. Also, trying to be real strategic on when we help guide and push.
It has a better impact.”
Notwithstanding this discretion in terms of promoting innovation, DEI served as a
significant marketing and advocacy lever for serving JSWDs: “Just being part of an initiative
means you can push your way into things you couldn’t push your way into before.” For example,
the GWDB disability committee sought best practices from Minnesota Round 5 to promote
statewide. The committee also engaged NDI to discuss lessons from DEI as well as how to align
promising practices with WIOA implementation and Section 188 and AJC certification. DEI
services were also amplified within VR, ABE, Community Transition Interagency Committees,
Veterans Affairs, and Unemployment Insurance.
Treatment WDAs conducted DEI outreach and dissemination through various channels,
including alternative schools and transition programs to engage more youth. In one treatment
WDA, VR alerted customers on its waiting list about DEI. This AJC was also selected to present
at a statewide Joint Counselor Training Conference, where it amplified innovative practices with
IRTs, blending/braiding, and targeting transition-age youth.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
89
In another treatment WDA, the DRC launched the grant with initial presentations about
DEI for AJC and community partners. She also spoke about DEI at a community and technical
college fast-track program. A placement specialist discussed ongoing outreach and sharing of
best practices, trainings, and contacts. However, one JSWD focus group participant suggested
AJC services “should be more open to the public” and that the AJC “should notify JSWDs more
often.”
The DRC from Minnesota’s third treatment WDA initially conducted targeted DEI
recruitment through the AJC, ABE, and other partners. This DRC opted to prioritize recruitment
for the AJC as a whole: “Our workforce center programs can serve people with disabilities. I
don’t want just to recruit for DEI because that isn’t what it’s all about. It’s about getting
everyone — people with disabilities and other people — into the workforce center, serving them.
And we can serve them in all our programs, and even without DEI.” Due to this strategy, focus
group participants indicated that the AJC could raise more awareness about DEI.
Promising Practices: “Peer Mentoring” vii.
One treatment WDA exhibited how a cohort model and peer mentoring could contribute
to CP completion and retention. A job search club in concert with training facilitated an
employment focus among JSWDs and helped them overcome common pitfalls. For example,
cohort members assisted each other with Excel and other training components. As they built trust
and solidarity, they also shared job leads, carpooled, and socialized. Some program graduates
even returned and aided peers. A peer support cohort model is appropriate and effective for
JSWDs since the employment search can be frustrating and isolating. The socialization aspect is
critical; trust-building and interaction cultivated a sense of a collective effort in training and
employment journeys rather than navigating these difficult processes independently. Also, many
were learning new skills and content and could benefit from peer tutelage in this endeavor,
especially from alumni.
Rationale for Its Implementation
The DEI State Lead and DRCs suggested that opportunities “for people with disabilities
to help other people with disabilities” are integral to the sustainability and scaling of disability
employment. They particularly cited “a huge need for home health personnel” as one gap that
could be addressed for JSWDs in service to their peers. One deemed this as a potential “game
changer” and even reported that people with disabilities have asserted that theirs peers provide
the “best services.” Moreover, this pathway is a “great entrance” into CP because an industry-
recognized credential is not mandatory and it can offer part-time, flexible hours. State leadership
is now championing this strategy across systems and CP initiatives.
Why the Practice Could Be Considered Promising
DRCs and other DEI leadership conducted presentations on best practices and “strategies
for replication” at WWL conferences and other events, including on IRTs, career assessment
tools, enhanced benefits counseling, TTW, interagency collaboration, capacity-building, and
employment readiness for JSWDs (including those in substance abuse recovery). The positive
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
90
reception to this latter presentation signaled that the AJC could garner referrals for JSWDs in
recovery. Additionally, DEI leaders from one WDA, including the executive director of the
WDA’s Board, presented at a National Association of Workforce Development Professionals
conference. They focused on how DEI aided in transcending Section 188 requirements and in
conceiving accessibility as a long-term, “systems change” endeavor. These leaders also shared
practices in conference calls with WDAs in other states. One DRC co-presented with MRC on
panels. The state’s extraordinary emphasis on dissemination was epitomized by its second DEI
Best Practices Conference.
Challenges and Sustainability viii.
There were a few roadblocks to optimal implementation. In most treatment WDAs, DRCs
were pulled into case management due to insufficient staff capacity and turnover. Resource
constraints also prevented realization of universal design across sites as well as ideal TTW
implementation in one treatment WDA. Stakeholders also suggested that staff could benefit from
more training related to the DEI SDSs to expedite an all-staff and IRT approach to serving
JSWDs. Training to engage JSWDs with mental health challenges was also cited as a critical
need. Minnesota was highly proficient with blending and braiding, but state leadership
recommended more policy guidance on co-enrollment to prevent “double-dipping.”
Retention struggles were one of the most trenchant obstacles in the long-term
employment of JSWDs. Any hurdles along the way can lead to dropping out. Some focus group
participants admitted that it would have been unlikely to complete the Excel training alone.
Excel in particular is a common prerequisite, and those with little prior experience with the
software can find learning it daunting. This is true of other required skills and credentials.
Furthermore, transportation is a universal challenge; carpooling resulting from cohort
relationship-building can be a pivotal factor in program completion.
Minnesota’s Round 5 DEI bolstered the state’s ongoing efforts to build a Career
Pathways infrastructure for vulnerable groups. To that end, Round 5 included several promising
approaches to disability employment. One treatment WDA showcased how to transcend barriers
to CP via cross-systems partnerships and customized assessments, trainings, and job
specifications. This WDA also featured a fruitful coordinated employer engagement approach.
Another WDA exhibited the promise of a cohort and peer mentoring model for training and
employment searches. The third treatment WDA was distinctive nationally as an innovator in
disability employment staff training and TTW implementation. With two treatment WDAs also
engaged in Round 7 and productive alignment between state and local leadership, Minnesota
may continue spearheading innovation in disability workforce development. State leaders did
stress DEI practices are conducive to sustainability, such as IRTs.
As Round 5 proceeded, state leaders asserted that one WDA showed the most promise in
forging sustainable practices. This AJC’s DRC functioned as a resource who trained staff, while
other AJC staff took on more of the load of engaging JSWDs through practices like IRTs. The
DRC also invested significant effort into TTW, Benefits Planning, and building upon Alaska’s
DRC training. The AJC’s development manager suggested that TA was “phenomenal at helping
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
91
us shift [from a] program mentality to more of that project level.” At the end of Round 5, the
DRC affirmed that staff felt more comfortable serving JSWDs.
State and local leadership were intentional about bolstering the sustainability of DEI
roles, relationships, and practices. State leaders also submitted that planning around the
Olmstead decision promoted sustainability, including embedding CP and peer mentoring
opportunities for JSWDs in workforce development. Initially, DRCs “looked more like case
managers,” according to state leadership. State leaders discouraged this by “firmly” advising
DRCs that “if they come out and monitor us, your files cannot look the same as the employment
counselor files. … That’s duplicative services. We can’t pay for you and the WIOA person to do
the exact same thing.”
This transformation contrasted with how this WDA approached its Round 3 grant.
Despite this progress, the DRC expressed how taking a systems and supervisory approach was
challenging because she enjoyed client contact and being hands-on. Sustainability was
increasingly paramount for the other treatment WDAs as well, as evidenced by leadership
discussions. One AJC manager described the DRC as “a resource, not a doer.” This DRC did
have her own caseload, but she mostly assisted job counselors in forming IRTs and accessing
resources for JSWDs. Still, she was compelled to fulfill case management responsibilities due to
staff absences.
This DRC acknowledged the need for more staff training to embed DEI ideals:
“It’s been a constant battle to get the employment counselors to understand that I’m trying to
teach them how to work with people with disabilities and how to develop these IRTs. I’m not
going to always be here. This is something that you’re going to do and you’re going to provide
to our customers. So, I realize that I really need to push and get some training done.” — DRC
DRCs straddled the line between case management and a systems-level role. The DRC
role was characterized as a “seamless connector” who was “interwoven” between services and
supports but also maintained close contact with JSWDs as they navigated recruitment,
assessment, instruction, training, placement, and follow-up. At the end of Round 5, the DRC
assumed the role of Minnesota Family Investment Program senior vocational counselor but will
continue DRC functions and work intensively with JSWDs.
Regarding training other staff to engage JSWDs, one DRC suggested that Career
Navigators would be most appropriate to absorb the DRC training since they have “consistent
contact [with clients] throughout.” This AJC adapted the DRC training and completed it with CP
staff who will be responsible for overseeing testing and accommodations. The AJC also created a
training package for new staff and an annual training for all staff, with plans for the training to be
expanded county-wide. Furthermore, management committed to taking a systems approach to
engaging JSWDs.
In terms of funding to sustain DEI practices, one WDA was well-positioned due to its
acumen with TTW. Another WDA became an EN toward the end of Round 5. The third WDA
garnered a Pathways to Prosperity grant, which includes a DRC role. A state leader reasoned
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
92
that, outside of TTW, workforce development for JSWDs would be integrated with CP
programming, meaning that WDAs must either leverage WIOA dollars or apply for future
dollars. A local manager mentioned that CP is incorporated into all funding sources as dedicated
CP funding is limited. She also asserted that there are CP and career and technical education
champions in the state legislature.
DEI practices and collaborations should be largely sustained since there is much
continuity between Round 5 and Round 7 grantees in two WDAs and well-established
partnerships. Round 5 also solidified robust relationships with state and local Equal Employment
Opportunity personnel. IRTs were specifically cited as “more embedded in what we do” and
crucial to avoiding duplication. To sustain best practices, one stakeholder suggested there should
be financing for accessible online training and tools that can be shared widely. This would help
ensure that the state’s visionary work with the DRC training — and other innovations — endure
and reach as many stakeholders as possible.
F. State: South Dakota; Focus Area: Adults with Disabilities
Stated Goals and Objectives i.
South Dakota’s Round 5 grant was designed to enhance the capacity of AJCs to obtain
better employment outcomes among adult JSWDs through Career Pathways training.
Partnerships included the Departments of Labor and Regulation (DLR), Human Services,
including VR services, Social Services, and Health; Western Dakota Technical Institute; the
South Dakota Workforce Development Council; and the Workforce Diversity Network of the
Black Hills. In addition, local industries and employers played an important role in developing
opportunities for CP programs as well as providing opportunities for JSWDs to interact with
employers, prepare a résumé to target certain occupations, and engage in support services such as
Work-Based Learning and Apprenticeships.
Social Dakota aimed to accomplish the following outcomes:
• Improve employment outcomes of adults with disabilities and maximize their economic
self-sufficiency through CP approaches, including academic and employment transitions
leading to industry-recognized credentials and 2- and 4-year degrees (a minimum of 25
adults annually). Outreach presentations will help recruit JSWDs who are dislocated
workers, disabled veterans, and/or part of the long-term unemployed and underemployed.
• Build on current WIOA services provided through AJCs and established CP programs to
place JSWDs in high-demand, high-wage occupations (e.g., trade/transportation/utilities
and health care services):
o At least a 25 percent increase in adult education services;
o ILPs for all participants;
o All participants assessed in need for short-term technology skill training; and
o By project end, 90 percent of participants were to report overall satisfaction with the
“universal design employment delivery system.”
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
93
• Create systemic change through increased partnerships across service delivery systems
and via replication of proven workforce development strategies throughout the public
workforce system (e.g., through a “partnership committee consisting of agency and
business representatives” and joint disability training). This was to be measured in part
through tracking agency involvement in blending and braiding and through monitoring
Ticket assignment and revenue.
38
DEI Service Delivery Strategies ii.
South Dakota served adults with disabilities, including the following subgroups: disabled
veterans, individuals in need of English as a second language, dislocated workers, and long-term
unemployed and underemployed individuals. South Dakota increased the use of IRTs, including
through statewide training and written guidance. This effort helped establish the grantee’s IRT
model, which provided opportunities for leveraging resources and identifying the need for
existing SDSs. In one treatment WDA, IRTs involved in-person meetings with diverse
stakeholders that were described as “really working well” and gave clients “power” to direct
teams based on their employment goals. Another WDA showed fidelity to the IRT model by
facilitating ARC before IRTs. IRTs combined with ARC in the third treatment WDA included
behavioral health professionals, parents, and other supportive individuals, including DRCs.
Supports ranged from daycare to vehicle repair.
South Dakota also implemented TTW, Partnerships and Collaborations, Blending and
Braiding Resources, and systems change. However, qualitative data did not suggest that all
participants completed ILPs or were satisfied with universal service delivery in which
individuals had opportunities accessible to as many individuals as possible.
Elements of Customized Employment were present in South Dakota, such as Discovery,
negotiation with employers about job carving, and tailored job descriptions, but there were few
efforts to develop long-term CE arrangements after the grant period, which was a key feature of
the grant. CE was implemented in its component parts rather than its full implementation.
South Dakota conducted surveys of IRT members. In addition to gauging satisfaction
with IRTs, a survey was used to measure how IRTs supported outcomes; the extent to which
IRTs facilitated Blending and Braiding Resources; satisfaction with a shared employment goal;
and how IRTs influenced time needed to serve a client.
Blending and braiding and co-enrollment were common DEI practices, such as VR
funding a job coach and WDAs supporting work experiences. Blending also allowed for some
JSWDs to have clothing for job interviews and funding for transportation.
39
Each WDA
promoted Asset Development activities, including financial literacy, credit counseling (e.g.,
Credit When Credit is Due coursework), tax assistance, and guidance on accessing TANF and
SNAP benefits. Help with these efforts was provided by Wells Fargo, state agency accountants,
38
South Dakota Round 5 project narrative
39
AJC Title I staff from one WDA discussed Self-Employment as an SDS, but there was no evidence of any DEI
clients pursuing this option.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
94
and the Black Hills Special Services Cooperative, which provides education, training, and
employment support services. There was also evidence of aid for housing, child care, and
transportation. DLR partnered with the Department of Social Services to include DLR
informational materials in SNAP and TANF statements, as well as with energy assistance
applications. In turn, AJCs distributed information about the Department of Social Services. The
state also made multiple efforts to kick-start TTW activity at DEI sites but opted to leave this to
VR after realizing the low volume of Ticket holders at grantee sites. Coupled with VR’s
expertise and foothold in this arena, state leadership reasoned TTW was “not a good a business
model.”
Implementation Summary iii.
South Dakota’s Round 5 grant aimed to maximize CP approaches through forging cross-
systems partnerships, Blending and Braiding Resources (including through TTW milestones),
and provision of services designed to ensure that any JSWDs could access support and training
services through their AJC. JSWDs had access to a range of training opportunities, from non-
credentialed basic computer courses to OJT and credentialed occupational skills training. CP-
focused strategies expanded utilization of IRTs, which featured career coaches and career
specialists; alternative assessments; ILPs; dual enrollment, stackable credential programs, and
blended learning; paid work experiences, internships, and other work-based opportunities; and
intensive wraparound and support services (e.g., transportation, academic and career counseling,
and financial aid). According to JSWDs, IRTs were validated through surveys and interviews of
involved staff, observations, and “post-assessment results.”
South Dakota’s Round 5 grant built upon its Round 2 activities. It centered on the West
River/Central area of the state. One treatment WDA was in Pierre, another was in Spearfish, and
the third in Rapid City. Rapid City’s implementation in particular was bolstered by the local
presence of a “regional career center,” the Black Hills Special Services Cooperative, and the
Western Dakota Technical Institute. Other than the State Lead, much grant leadership emanated
from this region.
WDAs demonstrated a high capacity to serve JSWDs during the grant period as each
WDA enhanced both physical and programmatic accessibility, such as with automated doors;
more disability-friendly cubicles, parking, and bathrooms; and AT (e.g., ZoomText). Toward the
grant’s end, AJCs were equipped with iPads to enable Skype access for customers seeking to
connect with partners outside of their WDA. Each WDA’s programmatic accessibility was
enhanced by local Adult Education Literacy and supportive service providers. Accessibility
enhancements led to increased participation by JSWDs in WIOA Title I programs.
40
And DakotaLink was an integral partner in maximizing utilization of AT.
41
Still, there was room
40
Based on quarterly reporting, South Dakota reported the following: 34.7% of the 2015 WIOA Adult enrollment
had a disability. This percentage increased to 42.81% in 2016, and after the first two quarters of PY 2017, it was at
48%. DEI also eclipsed its goal of expanding adult education service utilization. Only 11 such clients reported a
disability in 2013; this number steadily increased to 93 in 2016. As of end of 2017, DEI was serving 25
underemployed and 14 dislocated workers and 10 veterans.
41
Through DakotaLink, South Dakota provided AT devices and services for individuals when “functioning is
impaired due to a disability, injury, or aging.”
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
95
for improvement in physical inclusivity. For instance, in one WDA there was concern about how
the “cubicle farm” was not conducive to confidential conversations and disclosure with JSWDs
and other AJC staff.
The AJCs supplied transportation assistance for JSWDs. They also completed a proposal
that provided for more accessible transportation in the metropolitan area. Statewide, there was
much effort to make DLR’s website and other communications accessible, including a Job
Seekers with Disabilities resource page. The state also facilitated training on AT and the Job
Accommodation Network;
42
this training was posted on the DLR website and prompted AJC
changes regarding signage and basic electronic devices to assist JSWDs.
The DEI State Lead relayed that South Dakota did not approach universal design “very
well,” including being “written up” for not doing ADA reviews. At the same time, state
leadership asserted that AJC programming was universally accessible. A State Lead partnered
with VR to generate informational material encouraging businesses to have multiple ways to
apply for jobs, responding to concerns about the online application process for JSWDs. Also,
there is a link to ODEP’s universal design page on DLR’s JSWD resource page. Toward the
grant’s end, South Dakota prioritized streamlining access to services for JSWDs to reduce
burden.
Locally, in addition to disability-friendly modifications to AJCs, AT partners such as
Access Elevator and Lifts, Inc., provided self-paced and multi-modal assessment and learning
programming, including with the Barton Reading and Spelling System for clients with dyslexia.
One DRC furnished consistent reminders about the difficulty and length of certain Career
Pathways and encouraged clients to consider avenues of varying complexity and to pace their
plans to ensure that individuals have time to complete CP training while maintaining a lifestyle
that meets their family and employment requirements.
In another treatment WDA, Career Pathways road maps served to both visualize
employment journeys and motivate JSWDs. As one DRC stated, these visuals could be “hung on
their fridge [to] mark off what they’ve done, what they’ve completed, where they want to go.
That’s a big self-esteem booster as well.” This DRC’s public relations background and
experience serving individuals with disabilities helped her advocate for JSWDs and support them
in developing self-confidence regarding their employment goals. Continuous support from
DRCs, AJC colleagues, and job coaches, combined with motivational interviewing, encouraged
aspirational discovery, self-regulation, attainable goal-setting, and persistence among JSWDs.
One WDA exhibited multiple elements of inclusivity. JSWDs had ready access to
multiple modes for job searching and applying for employment, including a CP web tool. While
assisting with several JSWDs’ job searches, a DRC gauged their comfort level with IT and
helped build confidence with using computers. A JSWD focus group also reflected on taking
self-paced computer classes to improve their comfort with computers and learn how to prepare
résumés and related documents. Some DRCs and AJC staff provided background knowledge to
42
The Job Accommodation Network “is the leading source of free, expert, and confidential guidance on workplace
accommodations and disability employment issues.” It is sponsored by ODEP.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
96
stimulate disclosure, such as talking about whether a JSWD had an ILP to support her education
and future academic and employment plans. Finally, the AJC manager conducted mock
interviews with JSWDs to cultivate a sense of self-efficacy that individuals need to enter the
labor market.
Few DRCs served in a full-time role. Those serving rural areas engaged relatively fewer
JSWDs than more populous WDAs. DRCs approached their roles differently in South Dakota as
some provided primarily case management services while others focused on job placement. Most
DRCs had the ideal background, expertise, and skills for the position. DRCs and AJC staff grew
more equipped to serve JSWDs, such as with navigating self-disclosure so that JSWDs could
transition into soft skills training, skill development, and, eventually, employment.
Each WDA has ample CWIC capacity, typically through Partnerships and Collaborations
or VR. However, capacity challenges remained regarding serving JSWDs with mental health
barriers, handling disclosure, and securing access to transportation for JSWDs.
DRCs and other AJC staff were also supported and informed by abundant and well-
coordinated T/TA from state leadership. This tutelage not only raised awareness regarding
serving JSWDs within AJCs, partner organizations, and the employer community, but helped
overcome turnover and administrative restructuring. Staff trainings included sessions on Access
for All, mental health first aid, IRTs, business engagement, motivational interviewing, and
person-centered thinking; this aspirational thinking may lead to “positive control and self-
direction of people’s own lives.”
43
AJC staff also participated in other cross-trainings, such as a
disability symposium and “scenarios” to better understand partner agencies and their processes.
Partnerships and Collaborations iv.
Integration and collaboration across WDAs were established by a state-level interagency
cooperation, particularly between DLR and VR. This partnership enabled consistent and
productive messaging, resource leveraging, and cross-training. This synergy “trickled down” to
local offices, facilitating more and higher-quality services for JSWDs as well as less
fragmentation and enhanced cost-sharing. Per one DRC, the union of DEI and WIOA “really
increased our services to anyone with a disability;” VR and AJC ending up having “invaluable”
conversations about clients. Another DRC corroborated how VR counselors actively engaged
DLR to explore shared cases and “everyone’s comfortable.”
Early struggles with interagency communication and “silos” were remedied by additional
training on IRTs. Enhanced coordination was epitomized by “agency IRTs” in each WDA, a
novel approach to partnership meetings. IRTs included title agencies as well as employers,
United Sioux Tribes, and local transit agencies. These systems-level resource teams were most
prominent and productive in one treatment WDA. Agencies also partnered in offering statewide
conferences, trainings, and webinars about disability employment topics like motivational
interviewing.
43
Orthogonal. (n.d.). Using aspirational thinking to elevate your product design. Retrieved from
https://orthogonal.io/insights/a-simple-way-to-use-aspirational-thinking-to-elevate-your-product-design/
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
97
Partnerships and Collaborations were furthered via co-location, Blending and Braiding
Resources, co-enrollment, referrals, and coordinated business tours. One AJC and VR agency
referred via a “two-way street” that allowed JSWDs to access DEI services through either
agency. The Black Hills Special Services Cooperative and Western Dakota Technical Institute
were noteworthy partners that contributed to DEI implementation in each WDA. One DRC
characterized co-location of WIOA Title programs as “huge.” Other DEI partners included Jobs
for Veterans State Grants, the Departments of Education and Corrections, Job Corps, the Jobs for
America’s Graduates advisory council, Board of Regents, Service to the Blind and Visually
Impaired, reemployment services, and the Great Plains Tribal Chairmen’s Health Board.
Treatment AJCs benefitted from “increased interaction” with a local Career Learning Center and
an adult education provider, culminating in expanded options for assessments, training, work
experiences, and job coaching.
In addition to planning trainings and dissemination of promising practices, such as
trainings on Job Shadowing, State Leads facilitated coordination through monthly and quarterly
reports and data collection. Employers and local colleges were also involved in cross-training.
DRCs fostered internal AJC communication with “open door” policies by valuing VR as “the
experts,” sending informational emails, and by dedicating portions of meetings to partners.
However, there were challenges around data-sharing and creating “streamlined” referrals.
JSWDs, DRCs, AJC staff, partners, and employers all indicated that AJCs and local
employers were customer-centric and the employment process was customer-driven. In one
WDA, a JSWD proclaimed, “Nobody’s made any choices for me … they might steer you. … I’m
not pushed,” and another echoed this sentiment, “Same with me … I make my own choices. The
DRC just steers me in the right direction.”
One DRC suggested that a collective effort helped JSWDs lead their employment
journey: “With anybody you’re going to tailor the services to what their needs are. So, it’s just
working individually with people and determining what they need and how to get that need
satisfied, or pointing them in the right direction. We’ve got great community resources and
employers, so if we don’t handle something, I can usually refer them to an agency in town that
they can get services from, too.” In another treatment WDA, the DRC appeared to be very
intentional in letting JSWDs know “this is about you … We really want to give [you] that
power.”
The DRC and JSWD interviewees from one WDA testified that the AJC advocated for
individuals to pursue “what they want.” JSWDs also confirmed the AJC has lessened its reliance
on staff. It ultimately allowed for more independence in seeking employment among JSWDs.
Aspirational and person-centered discovery, which staff members were trained on, was
balanced with pragmatic, ongoing career planning that included multiple entry and exit options.
As a result, JSWDs were more engaged with diverse JSWDs through their IRTs. Stakeholder
interview testimony was reinforced by evidence of customer satisfaction surveys regarding IRTs.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
98
Nonetheless, there was scant evidence that JSWDs were involved in strategic planning
despite one JSWD who successfully lobbied to make the AJC’s parking more inclusive. Also,
JSWDs expressed a desire for more long-term supports and individualized assessments.
South Dakota illustrated several innovative practices related to employer engagement, as
reflected by its designation as an ambassador for the DEI Business Engagement Community of
Practice. Like other robust facets of South Dakota’s DEI, its employer engagement stemmed
from state-level leadership. For example, in partnership with VR and the South Dakota Retailers
Association, the state’s DLR Cabinet Secretary Hultman conducted a webinar with businesses
about hiring JSWDs.
At the local level, one treatment WDA most exemplified promising business engagement
practices through a multipronged approach. In a “public relations role,” the DRC regularly
advocated for JSWDs with many employers to transcend “the fear that you’re going to lose
money.” The DRC also coordinated business tours with “high-demand, high-wage” companies.
One employer expressed gratitude for coordinating approaches to provide trained workers for his
company. In addition, the DRC facilitated employer involvement in agency IRT meetings.
VR staff conducted most of the employer outreach in another treatment WDA. However,
the DRC spearheaded statewide planning for promotional media around disability employment
and National Disability Employment Awareness Month (NDEAM). The AJC manager testified
that such promotion “opened up the eyes of employers.” The DRC also consistently advocated
for JSWDs with employers using “person-first” conversations. Additionally, partners noted that
employers were willing to take part in IRT meetings. However, they were not heavily involved
in strategic planning at the AJC.
Another WDA maintained that consistent, productive, and personalized relationships
with regional employers are essential to providing services to JSWDs. The DRC attended local
job fairs and maintained contact with local schools, while the AJC manager participated in
chamber meetings and WDA or state-level meetings and ceremonies. The manager also sent staff
across the region to do employer outreach, which bolstered job listings at the AJC, NDEAM
coordination, and OJT arrangements. Also, the DRC was mindful to keep in regular touch with
employers to ensure “they don’t forget about us.” Both the DRC and AJC manager conversed
with employers about individual JSWDs, their skills, the types of work for which they qualified,
and possible accommodations. Notwithstanding these favorable relationships, the WDA did not
furnish collaboration with employers that could guarantee long-term placements beyond
temporary OJT or work experiences. This challenge was common across treatment areas.
Moreover, there was no mention of employers being involved in agency IRTs. Despite these
proactive engagement activities, employer biases and insufficient accommodations endured
according to interviewees.
Career Pathways v.
Regarding Career Pathways, one treatment WDA stood out due to featuring the most
vibrant infrastructure, buttressed by its proximity to the region’s technical school and leading
adult education/support services provider. This WDA also took the lead in producing and
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
99
disseminating a Building Pathways toolkit and CP visuals to guide JSWDs, in alignment with the
technical school’s pathways model. CP road maps depicted multiple entry and exit points for a
variety of employment journeys and included information on suitable education and
certifications, career exploration, supportive services, financial assistance, and expected wages.
As well, the WDA generated documentation on Job Shadowing for staff, clients, and employers.
Statewide trainings incorporated this guidance and also promoted best practices around
securing accommodations for individuals with disabilities to complete their GED. The South
Dakota Workforce Development Board created a statewide joint-funded CP committee with the
Department of Education, Board of Regents, and VR to develop an accessible and integrated CP
system. Part of this effort was building a Career Pathways development web tool to be used by
multiple partners and to track data. Webinars were conducted on using the tool, pathway
information was being gathered, and JSWDs were being recruited.
The other treatment WDAs had less immediate options for Career Pathways, though
jobseekers could connect with training programs beyond those areas (e.g., medical coding and
pharmacy technician). One DRC indicated about a quarter of DEI clients earned additional
certification through either school or online training.
Work experiences, OJT, internships, and Job Shadowing were common DEI offerings but
did not guarantee long-term placements. VR and DEI often complemented wages for paid
opportunities. Outside of one treatment WDA, there was not much evidence of Work-Based
Learning that incorporated school-based instruction. South Dakota also featured technology
training and digital literacy assessments in its CP offerings.
Altogether, 74.4 percent of DEI clients received a digital literacy assessment. Forty
participants received technology training. High-demand pathway trainings included banking,
network administration, computer science, welding, truck driving, CNA, auto mechanic, and
electrician.
Outreach and Dissemination vi.
South Dakota Round 5 featured executive champions for JSWDs and bottom-up
dissemination of promising practices. Governor Daugaard pinpointed his own family experience
with disability as part of his efforts to encourage businesses to hire JSWDs, which included
the Ability for Hire initiative. The state built upon its cache of promotional videos from Round 2,
including videos for employees and employers on hiring, accommodations, and ADA in the
Work Place.
This media was supported by a statewide “road show” to promulgate effective practices
for serving JSWDs, including ARC, IRTs, Job Shadowing, and CP. A DRC, along with her
AJC’s Title I representative and partner leadership, led this effort. This exhibition was reinforced
by written guidance distributed statewide and by ongoing trainings, webinars, and conferences
with staff, partner, and employer involvement. NDEAM was another avenue for widespread
dissemination.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
100
One WDA epitomized the state’s commitment to disseminating promising practices about
serving JSWDs. In-person trainings were complemented by written documentation on IRTs,
Communities of Practice, and email blasts. Dissemination also involved employers via business
tours and partner meetings.
Another treatment WDA was intentional about integrating DEI services and resources
with WIOA. The AJC relied on its “very good relationship” with VR and Access Elevator and
Lifts, Inc., to recruit clients and also conducted outreach in local high schools. To promote
effective practices, DEI “success stories” were sent to the DEI State Lead. The DRC also helped
with planning for NDEAM and video production.
Promising Practices: “Employment Tours” vii.
The South Dakota Division of Developmental Disabilities and the state’s Council on
Developmental Disabilities are members of the Supporting Families Community of Practice.
South Dakota is one of 11 expansion states in the national Community of Practice for Families of
Individuals with Intellectual and Developmental Disabilities. South Dakota developed a state
work plan to ensure that family networks provide family-centered case management and expand
in-home services. South Dakota also created “a comprehensive, coordinated set of person-
centered and family-centered strategies.” Employment tours were designed to acclimate JSWDs
to the work environment, provide information on their employment interests, and increase the
employment rate for JSWDs in the state.
South Dakota focused on occupations in high-wage, high-demand industries based on
data from the state’s Labor Market Information Center and insight from technical institutes, the
local Society for Human Resource Management chapter, and business services representatives.
Prospective employees were engaged to learn about several companies through tours and
interviews with staff. AJC attendees included WIOA core and “other required” partners (e.g.,
VR, TANF, and SNAP) as well as from technical institutes and the Workforce Diversity
Network.
Tours and interviews included inquiries about hiring needs and qualifications, available
training, compensation, and accommodations. Employment tours offered staff opportunities to
“experience the culture of the business” and its physical layout. AJC staff devised a “customized
workforce strategy” and shared it with prospective employers. The strategy was designed to
facilitate businesses’ recruitment of suitable candidates with disabilities and included a South
Dakota-produced protocol on each aspect of employment and the interview process.
Rationale for Its Implementation
Employers are often approached by many entities about job openings and candidates.
This was confirmed by an employer who participated in a treatment WDA’s DEI partnership
meetings, who also affirmed that coordinated business tours are much more productive. A
standard protocol and process ensures employers are not overwhelmed and that efforts by partner
agencies are not duplicated. Ultimately, this efficient approach to networking should lead to
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
101
more placements and more optimal matches between JSWDs and employers. It also improves
collaboration between workforce and employers and among WIOA partners.
Why the Practice Could Be Considered Promising
Data collection revealed the same issue as had surfaced in our evaluation of the
Employment First State Leadership Mentoring Program. South Dakota’s approach provided a
template and documentation for creating synergy between workforce agencies, employers, and
JSWDs. Raised awareness about business needs and JSWDs’ capabilities laid the groundwork
for employment matches and enhanced referrals. Indeed, a South Dakota DRC testified that
“connections” from business interviews and tours “have opened that door for job shadows, work
experiences, and on-the-job trainings.”
Challenges and Sustainability viii.
Due to the state’s rural nature, convenient access to trainings and high-demand pathways
can be elusive for many JSWDs. In addition, capacity challenges complicate serving JSWDs
with mental health issues, facilitating transportation access, and engaging Ticket holders. A
trenchant dilemma, not unique to South Dakota, is expediting employment beyond short-term
OJT, internships, or work experiences. Persistent employer biases and lack of accommodations
contribute to this problem.
Notwithstanding a significant funding void with DEI’s sunset, which will limit monies
for assessments, training, supports, and dissemination customized to JSWDs, there were many
signs that disability awareness and promising practices for serving JSWDs will be sustained in
each treatment WDA. Even with the two DRCs who adopted a more case management approach
— and despite each DRC serving in that role part-time — grant leaders were intentional about
integrating DEI practices and policies into WIOA processes so as to forge system changes and
maximize resources. One DRC asserted DEI was designed to promote “long-lasting, big change.
… This is a way to make changes, not just spend this money.” Further, at each treatment AJC, all
DEI staff served JSWDs and received pertinent ongoing training. Each DRC was absorbed into
their respective AJCs at DEI’s end.
The fruitful collaboration spurred by DEI, particularly between DLR and VR, should
persist. State and local leadership testified to this systems change accomplishment.
Consequently, there will be “no right door or wrong door” for JSWDs to seek services. IRTs and
partnerships will also endure, and the state’s IRT approach should cement integration. As one
DRC stated, “I [envision] all of our relationships continuing, the IRT approach continuing. … I
don’t see anything for the participant changing. We’re still going to have all of those services,
and I may just be wearing a different hat again. It’s just a different title.”
In this WDA, JSWDs will likely be enrolled into Title I programming.
44
This DRC’s
public relations approach also bodes well for the sustainability of awareness and promising
practices related to serving JSWDs.
44
Title I programs include job search, education, and training.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
102
Grantees also engaged a local career learning center, technology training, and adult
education providers about continuing technology assessments and trainings after DEI. Still, one
DRC was concerned that there might not be a substitute for DEI funding to continue the valuable
assessments and technology training provided by partners.
To garner financing for DEI practices, stakeholders mentioned tapping into Title I and III
funds as well as TANF and SNAP allocations. A prominent partner also suggested that statewide
philanthropy could help sustain CP programming. TTW will remain within the purview of VR.
In sum, South Dakota’s implementation was exceptional when measured by its
innovation in several aspects, namely via agency-level IRTs, a dissemination road show, and
coordinated business engagement. As well, one region in particular showcased productive
synergy between the AJC, local partners, and state leadership. This led to a solid and inclusive
CP infrastructure. Moreover, disability awareness and service collaboration for JSWDs in the
workforce system has been significantly elevated after two DEI Rounds. Nevertheless, accessing
and maintaining long-term quality employment options for this population remains a quandary.
VII. GRANT IMPLEMENTATION ROUND 6
A. State: Alaska; Focus Area: Youth with Disabilities
Stated Goals and Objectives i.
The DEI Alaska Youth Works project served youth with disabilities ages 14 to 24
(including subgroups such as intellectual/developmental disabilities, adjudicated youth, homeless
individuals, mental illness/substance abuse, and out-of-school youth). The project proposed
building on the success of Alaska’s DEI Rounds 1 and 4 grants to expand capacity of the AJCs in
assisting individuals with disabilities in obtaining self-sustaining and meaningful employment.
Alaska has a significant geographical reach, with its service area spanning six WDAs:
Anchorage/Mat-Su, Northern, Interior, Southwest, Gulf Coast, and Southeast. Each region
encompasses large geographic service areas and includes urban and rural communities.
Alaska Round 6 goals were to:
• Increase Partnerships and Collaborations to develop a cohesive approach to serving youth
through partnerships with existing systems and programs as well as other providers of
services to youth with disabilities.
• Build upon Alaska’s Career Pathways system by increasing accessibility, retention, and
successful attainment of industry-recognized credentials to include the University of
Alaska system, Regional Training Centers, apprenticeship programs, school-to-work
programs, and employer-based training.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
103
• Increase employment opportunities for youth through additional business partners and
provide disability awareness and accommodation training and education on the
advantages of hiring youth with disabilities.
• Capitalize on Section 503 and Vietnam Era Veterans’ Readjustment Assistance Act final
rules by assisting federal contractors in hiring youth with disabilities and using summer
youth employment, OJT, and other Work-Based Learning. Alaska DEI will:
o Increase the number of AJCs operating as ENs from 6 to 10;
o Create systems change within the existing CP system to increase participation of
youth with disabilities; and
o Increase family awareness and involvement in supporting youth toward independent
futures.
DEI Service Delivery Strategies ii.
As a youth-serving state, Alaska was required to implement services in congruence with
the principles of the Guideposts to Success. In addition, they proposed to focus on IRTs and
Blending and Braiding Resources. Alaska’s IRTs convened stakeholders from a wide variety of
community-based programs and service providers to identify and align services and supports that
contributed to a jobseeker’s career goals. Alaska IRTs were most often created by DRCs, who
leveraged public and private resources to improve employment outcomes. Blending and Braiding
Resources was an essential component of IRTs; however, resource leveraging could occur at
both the services and systems level. Blending and Braiding Resources is a financing strategy that
can integrate and align multiple disability and employment grants and their funding streams to
broaden the impact and reach of services provided and achieve statewide economies of scale.
Therefore, Alaska’s choice of prioritizing IRTs and Blending and Braiding Resources
corroborated their intent to build upon the Partnerships and Collaborations that they developed
during their earlier DEI Rounds.
DEI SDSs, particularly IRTs, Blending and Braiding Resources, and Benefits Planning,
were firmly embedded in the Alaska DEI. This, perhaps in no small measure, was facilitated by
the step-wise development and implementation of services commencing with their first adult-
focused DEI grants and continuing through Round 6 with its focus on youth with disabilities.
During our interviews, we found that IRTs and ARC were generally known by DRCs even if the
terminology was not always articulated. However, they were occurring functionally across AJC
locations by developing partnerships with each JSWD and having DRCs facilitate opportunities
for training and employment. Since Alaska DRCs were certified as CWICs, attention to Benefits
Planning while customers were developing job goals was also embedded into the IRTs.
DRCs provided a number of examples of direct intervention with employers on behalf of
JSWDs. For example, one DRC described how an applicant was turned away because his speech
was slurred and he walked with a limp — the receptionist thought that he was drunk. With
follow-up and what the DRC described as “educating employers,” the employer was made aware
that the jobseeker’s speech was slurred because of a stroke and the DRC reinforced why the
applicant was a good job candidate. Afterwards, the interview went well and the individual was
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
104
hired. Attention to “grassroots” employment and case management, while maintaining a systems
focus that included CP and related support services, characterized the work of Alaska DRCs.
Most DRCs appeared to have general knowledge about CP and to a lesser extent the
Guideposts to Success. DRCs talked about the relationship between employment, the
employment environment, and Guideposts and youth development principles even though they
may not have been conversant in “Guideposts” terminology. CP approaches were most evident in
the University of Alaska VR Pre-Employment Transition Services partnerships, where youth
gained exposure to certificate-bearing curricula relevant to the range of Alaska growth industries.
They also learned soft skills.
As reported prior to the grant end date, all WDAs implemented IRTs, Blending and
Braiding Resources, some aspects of CE, Asset Development, and Guideposts when serving
youth, and they implemented partnerships for Self-Employment/Entrepreneurship when
necessary. They clarified how some service strategies were important tools to use when serving
either youth or adults with disabilities. It is important to underscore the value of Alaska’s
approach to training DRC 1s and 2s as a tangible example of integrating WIOA services with the
DEI. That is because training AJC Employment Specialists as DRC 1s with the training often
provided by the funded DRC 2s meant that the DEI had access to internal AJC staff who
engaged customers in WIOA services and linked them to DEI services.
DRC 2 duties included updating their DRC 1s on any local and state disability initiatives,
including an orientation to the ABLE Act.
45
They also followed up with their offices to keep
them informed on how they could help people with disabilities related to the provisions of the
Act. Since DRC 2s were in charge of all assistive technologies in their region, they had the role
within their AJCs of making sure their AT was working, documenting what types of AT they had
in their region, and making sure that all their frontline staff were up to date and trained on the AT
equipment and methods. So, although DRC 2s had a definite role as case managers in
outreaching to, engaging with, and providing direct DEI services to customers, they also worked
to enhance systems-wide capacity to better serve people with disabilities using AJC services.
WIOA Youth services were provided by community partners under contract. Their focus
was helping to get youth into employment, and they worked with DEI DRCs and with DVR and
other partners to leverage funds and knowledge and help youth get where they needed to be as
they transitioned from school to work.
Alaska did an effective job of implementing ENs and TTW, facilitated by its Partnership
Plus arrangement with VR. Any cross-systems partnership may have subsequent implementation
challenges, but DRCs and the State Lead and the AJC and VR TTW services were working well
together to resolve difficulties in obtaining and providing services to TTW enrollees. At the AJC
level, Alaska Round 1 and Round 4 staff also trained JSWDs as DRC 1s, a model for other states
and referenced as a promising practice. Thus, a “no wrong door” policy to employment was
45
Internal Revenue Service. (2019). ABLE accounts - Tax benefit for people with disabilities. Retrieved from
https://www.irs.gov/government-entities/federal-state-local-governments/able-accounts-tax-benefit-for-people-
with-disabilities
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
105
developed that, at a practical level, supported cross-staff communication and integrated services
planning. This systems-wide training effort was one of the indicators of sustainability; after DEI
ended as a project, its methods are likely to remain embedded within AJCs, VR, and other
partner agencies.
By the end of the third quarter of 2018, the Alaska grantee reported that they had
significantly exceeded their goals: enrollment + 360 percent; CP entry + 227 percent; and
entered employment: +245 percent. They nearly achieved their credential earned goal (98%) and
average earnings goal (96%). Retention data was unavailable at the time of this report.
Implementation Summary iii.
Alaska is a regionally, ethnically, and economically diverse state characterized by
economies and job markets that can vary extensively across regions. Site visits, focus groups,
and observations corroborated that the DEI provided employment services that helped youth
JSWDs obtain employment, from transition camps where youth living in mostly rural juvenile
detention facilities were flown in to Nome, Alaska, for intensive 2-week life and employability
skills trainings, to paid internships at AJCs and at provider agencies that, in several participants’
estimations, helped them to become employed. It appeared that Alaska was “meeting customers
where they are at” and on track to achieving positive employment outcomes for youth with
disabilities.
Although Self-Employment was not selected as an area of focus by any Round 5 or 6
grantees, the geographical and economic circumstances in Alaska often substantiated support for
Self-Employment/Entrepreneurship where wage-based employment was largely unavailable.
These pursuits could be in arts and crafts or other cottage-type businesses, or businesses that
were part of mainstream products and services.
Partnerships and Collaborations iv.
The cross-agency and cross-systems partnerships that continued their evolution from
Alaska’s Round 1 and Round 4 DEI grants were both evident and strengthened through the
efforts of its Round 6 grant. Alaska cultivated relationships among public agencies serving
JSWDs at multiple levels. At the state policy level, the DEI State Lead was a frequent attendee
and advisor to state-level policy conversations regarding employment of JSWDs. A
representative from the Governor’s Council on Disabilities and Special Education offered how
valuable it was to have access to the DEI in developing cross-agency partnerships. The Council
demonstrated a strong commitment to supporting rights and economic opportunities for people
with disabilities and secured funding from multiple sources (DEI, Partnerships in Employment,
Employment First State Leadership Mentoring Program). Furthermore, it viewed these programs
as part of a matrix of opportunities to support a comprehensive, collaborative approach to
employment of people with disabilities. Alaska is also an ABLE Act state.
46
The Governor
46
Achieving a Better Life Experience (ABLE) Act (Public Law 113-295) was signed into law on December 19,
2014. The ABLE Act amends the Internal Revenue Service Code of 1986 to create tax-free savings accounts for
individuals with disabilities.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
106
signed Employment First legislation. The Governor’s representative noted how DEI leadership
was an important technical resource for advising on future grants and projects that contributed to
a positive, mission-driven consensus on integrating services and resources supporting the health,
housing, and working lives of JSWDs in both policy and practice.
When working in extremely rural communities as a resource-leveraging DEI partner, VR
and its Tribal VR component had to negotiate often complex relationships among and within the
governing authorities of tribes and with generic services and systems, such as public education
and employment of youth with disabilities. Both DEI and VR representatives remarked on the
challenges of providing services to youth with disabilities living in the most “rural of rural
communities.” In these locales, challenges were significant in connecting youth to jobs while
also navigating through barriers in access to basic daily living services, poor telecommunications
infrastructure and public transportation, and the impact of the state’s economics contributing to a
paucity of jobs. This multifaceted effort required collaboration and trust-building among diverse
public, private, and tribal partners, which was evident in Alaska Round 6.
“First thing they do is if you don’t know where to start, they take interest tests and learn your
strengths and your personality.” — Youth Juneau
“Before coming to the job center, my only experience was a work readiness class, it was
horrible. You write a poor résumé and look at three colleges. It is so general that it doesn’t help
you at all. There is a career counselor at HS but I never met with them.” — Youth Juneau
Employer engagement occurred at multiple levels. First, the business development teams
located at AJCs connected with employers to link jobseekers with training and employment and
were a resource to DEIs in helping youth with disabilities secure jobs. The Business Employer
Services Teams were employer outreach and employer engagement programs that operated VR.
Because of VR’s strong interface throughout the state with DEI, they also worked on behalf of
DEI customers.
Youth involved in the DEI had positive things to say about their experiences and access
to AJC services and opportunities for Job Shadowing, job accommodations, WBL Opportunities,
and CP training. They often said that their experiences contrasted favorably with the career
guidance and development assistance they received in high school and college. Youth spoke
particularly well of the support that staff and employers provided and how they were being
listened to and assisted in developing their occupational goals: “Yeah, they listen to me and I feel
comfortable to have a say.” DRCs also made positive differences in the lives of youth. This may
be an indication of how systems change naturally occurs; when the “program” becomes so
embedded in an agency and connects with employers, it becomes indistinguishable and
embedded as a routine practice that the agency “normally” provides JSWDs. It was very
apparent from youth in focus groups that they felt empowered by employers, listened to by
DRCs, and respected within the AJCs and in DEI.
Grant funding included the creation of internship opportunities at job centers in Juneau,
Anchorage, Mat-Su, Kenai, and Fairbanks. The internship program began in November 2016.
Youth had opportunities to work in office settings with customers, but also to provide their input
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
107
into ways that center services could improve accommodations and services for youth with
disabilities.
“I would say this internship program helped me exponentially, actually. It kind of gave me a
direction on what I want to do with my degree in college.” — Youth, Fairbanks Alaska
“Being at an office all day didn't seem like the greatest job in the world. I kind of steered away
from that, but through my 6 weeks I would say I got more comfortable answering the phone, and
I kind of enjoyed the interaction with people. I would say that was like my favorite part of my job
was the interaction with people and helping them.” —Youth, Fairbanks Alaska
Career Pathways v.
Regionally, Northwest Career Technical and Training conducts Transition Boot Camps
for youth in detention facilities and high school seniors to prepare for and obtain career-track
development; employers in targeted growth occupations are engaged as partners and potential
employers of youth who acquire credentials. The DEI developed an MOU with the University of
Alaska Southeast to connect students to training, certifications, and job opportunities after
completion of CP exploration courses. DEI and VR collaborated to provide summer employment
opportunities, another example of outreach to employers and over 150 businesses that were
participating in the Transition Boot Camp project.
VR brought together the Department of Education and Early Development, Office of
Children’s Services, and an organization delivering foster care services and created the first
transition camp. Using VR and other braided resources, they flew 24 Office of Children’s
Services foster youth from around the state to the transition camp in Anchorage for employment
job goal development, soft skills, work experiences, and life-skills training that expanded from 6
to 12 camps a year.
Alaska’s growth industries include marine technologies, fishing, mining, and
construction, among others. But the economic and employment considerations vary so
dramatically across the state that flexibilities in the types of jobs and careers that youth with
disabilities can access, especially after returning to their home communities, must be considered.
For example, a young person may need to leave their village to learn about and gain experience
in a growth sector like maritime work or health services. But that entails being separated from
their community of origin and it may not be the work that is available to them when they return.
Also, many rural communities engage in subsistence occupations. A young person may have
career-level credentials, but they will need to work in subsistence-level hunting and fishing to
help provide for their family and communities. VR and DEI understood these issues and
embedded flexibilities into their services to provide maximum accommodation for youth,
including foster care youth, to participate in CP training and work activities.
Youth also developed new skills like negotiating urban environments and diverse peer
and social relationships while in training as positive youth development outcomes. From the
sustainability interview, the close connection between the participating AJCs and DEI was
evident in successfully implementing CP.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
108
Outreach and Dissemination vi.
Alaska’s outreach and dissemination strategy was multi-tiered and related to their
partnership development efforts at both the state level and across the AJCs that hosted the DEI.
The DEI State Lead provided briefings to the Governor’s Council on Disabilities and Special
Education so that progress and challenges were known to policymakers who then had the
information needed to support the project. Across regions where the DEI was located, DEI
program materials were available at AJCs for participant recruitment; the DEI also used
traditional (flyer, announcements, etc.) marketing materials to attract customers. But DRCs also
marketed the DEI for the purposes of developing partnerships with other agencies and leveraging
resources. For example, in the Mat-Su region, the DRC provided information about DEI and
developed partnerships with Job Corps to take referrals of DEI participants with some interest in
a specific field. In Fairbanks, the DRC communicated regularly with VR, whose counselors were
placing students into training and employment. Alaska was also effectively using information-
sharing not only to swell enrollment of participants, but also to build partnerships based upon
mutual self-interest that strengthened and can potentially sustain the program.
Promising Practices 1: “Training DRC 1s” vii.
Grants are necessarily time-limited. A consequence of an end to a grant is that staff who
were hired and trained to perform essential functions usually move on to other jobs. However, if
new knowledge and practices are firmly embedded into systems and programs so that their staffs
incorporate new ways of approaching problems and using methods that were demonstrated
effective over the life of the grant, the potential for long-term sustainability is improved.
Rationale for Its Implementation
AJC staff, as well as partner agency staff who worked with the DEI, were trained from
the DEI as “DRC 1s.” They were trained in the fundamentals of working with JSWDs; gained
familiarity with DEI employment practices; and increased their awareness of the services that
other partners brought to the initiative that both embedded knowledge in generic service systems
and reduced service siloing. When Alaska’s DEI funding diminished in June 2018 and its
employment of DRCs ended, the State Lead still had access to their statewide network of DRC
1s to continue to provide many of the services that DRC 2s provided, including enrollment into
TTW.
Why the Practice Could Be Considered Promising
This systems-wide training effort (acknowledged by Alaska’s DEI partners as very
effective) is a significant indicator of sustainability. By not confining expertise in serving JSWDs
to DRCs hired by the grant (“DRC 2s”), improved methods have the potential to become
universal. By mid-2018, this strategy demonstrated its value after the Alaska DEI found itself
without the funds necessary to continue to support its whole cadre of DRC 2s. The State Lead,
supported by depleted DEI funds, assumed the role of a “statewide DRC” through the remainder
of the project. And although the level and intensity of services provided directly to customers by
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
109
DRC 2s could not be maintained, DRC 1s across all DEI WDAs knew enough about DEI
practices and partnerships that they could take on most of the direct employment support services
formerly provided by DRC 2s. Consequently, through partnering activities and familiarity with
the DEI grant infrastructure, counselors from VR and partner agencies know enough and are
experienced enough to potentially continue the effective practices that the DEI introduced and
make progress in achieving the systems change goals that Alaska envisioned. What remains to be
seen is not only how DRC 1s continue DEI services, but how partner leadership steps in to
support their work in the same manner — and with the same effectiveness of the Alaska DEI
State Lead — after that position is no longer funded.
Promising Practices 2: “Coordinated DEI and DVR Approaches to Serving Youth viii.
with Disabilities with Criminal Justice Backgrounds”
Rationale for Its Implementation
Division of Juvenile Justice youth with disabilities are a challenging population to engage
in formalized training and employment services. Often a major impediment is that they are
served by multiple systems (criminal justice, disability, VR, housing and allied services, etc.)
that are unaligned and lack partnership agreements. However, the Alaska DEI Round 6 project
implemented a partnership with VR, the Alaska Department of Education and Early
Development, the Division of Juvenile Justice, and Alaska’s Educational Resource Center to
provide transition camps to rural Alaska and juvenile justice centers. The resource-leveraging
and partnership-building strategy, combined with a merged career and life-skills curricula with
an ability to draw youth with criminal justice backgrounds from their very rural home
communities, is an example of resource-leveraging among partners to serve youth who not only
have significant disabilities and criminal records that could hinder employment, but who may
never have visited nor lived in any place outside of their very rural communities of origin.
This partnership provided the transition camps to rural Alaska and juvenile justice
centers. The project also featured direct engagement of DRCs providing DEI services. For
example, the DRC in Juneau worked directly with the therapeutic justice system facility where
the youth were incarcerated for approximately 2 years, providing employment soft skills training
and career readiness services in groups and individually, while the DRC in Kenai worked with a
detention facility on work readiness skills, and, throughout, the VR Pre-Employment Transition
Services program helped fund the employment readiness components of the partnership. The
transition camps are not only a good example of “in-reach” to a challenging group of youth with
disabilities, but also of the necessity and value of creating cross-systems partnerships to leverage
knowledge and resources and overcome system barriers.
The DEI State Lead reported that “each of the camps are unique in their schedule of
activities. However, each camp included a DRC to talk about DEI and job center services. They
also hear from various speakers and are able to visit local businesses. Students are asked to
complete a person-centered plan that focuses on their future goals, their supports, and how they
will achieve their goals. The camps also provide career exploration, work readiness training,
counseling on postsecondary education, and Work-Based Learning experiences. [And during
2018,] most camps are also able to offer a college credit for attending and participating.”
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
110
Why the Practice Could Be Considered Promising
Assisting youth with disabilities who are in transition from criminal justice systems to
develop good work habits and skills is a difficult challenge, and it is made more difficult when
the youth reside in very rural communities where both job prospects and range of work
experiences may not be prevalent. The Alaska DEI developed its strategy with the understanding
that problems of that nature are best addressed by partners working together, communicating
frequently, and sharing resources. How they achieved such strong partnerships and the
subsequent outcomes are worthy of more exploration and explication so that other communities
may learn from their experiences.
Challenges and Sustainability ix.
As discussed previously, the Alaska DEI had to address significant implementation
challenges related to geography, diverse economies, and barriers that youth with disabilities
faced in accessing services. They encountered gaps in Benefits Planning assistance that they met
by certifying some of their DRC 2s as CWICs and found that some were more suited to the task
than others. They faced the typical systems-related barriers related to fragmented work readiness
and career development systems that they addressed through engagement of school districts, a
close working relationship with VR, and a concerted effort to change the cultures of AJCs to
become more welcoming to people with disabilities. And, they addressed attitudinal barriers that
often result in preconceptions that people with disabilities are not able to work by training
frontline staff at AJCs in disability awareness, the value of accommodations, and DEI service
strategies. Also, transportation was a challenge, especially for JSWDs living in extremely rural
areas.
The most significant challenge related to funding. The Alaska DEI reported in August
2018 that they had depleted their grant funding during the quarter; their DRC 2 staff were laid
off as of June 1, 2018. However, largely due to the strength of their partnerships, this grant met
or exceeded its use of selected SDSs and sustainability. Despite this significant reduction in their
DEI staff, the State Lead maintained her role as a primary interface with state leadership and
across the AJCs. Her work was aided by the state’s DRC 1s since all their participating job
center staff received DRC 1 training. Consequently, they reported that the frontline staff in the
AJCs would continue to provide disability coordination even after the DEI ceased formal
operations prematurely.
Despite the significant challenges noted above, the Alaska DEI made notable progress in
achieving systems change that supported the education and employment of youth with
disabilities. A key to implementing their planned, strategic approach to sustainability was their
strategy of cross-agency partnering to implement services (particularly with VR Pre-
Employment Transition Services and the AJCs in each of their regions). The Round 6 DEI
continued its Round 1 and Round 4 efforts to universally train AJC staff as “DRC 1s” on the
fundamental aspects of serving people with disabilities so that a “no wrong door” to employment
culture and climate was developed across participating AJCs. This approach ensured that after
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
111
DEI funding ended, the continuation of many DEI service components (particularly ARC, IRTs,
and TTW) and services to youth with disabilities adhered to DEI principles and practices.
The partnership with VR is continuing although some aspects of the resource-sharing had
to be curtailed due to the lack of DEI funding since the DEI paid for 30 percent of the Pre-
Employment Transition Services program for serving youth. They also indicated that some of the
Career Pathways are sustaining operations due to ongoing partnerships with education and
training entities developed through DEI. Internships at the AJCs, where youth would gain job
experience and soft skills as well as provide recommendations for improvements at the Centers
for serving JSWDs, were discontinued although they had about 30 youth who went through the
program over the grant period, most of whom continued their education or went on to
employment.
The DRCs in their original role at the AJCs are gone, and TTW is no longer sustainable
since there are no DRCs to manage the program. Per leadership, almost 170 SSA beneficiaries
were served during the Round 6 grant period and almost $500,000 in TTW revenue was
generated. More than half of the DRCs from Round 6 continue to work with the same treatment
WDA after the grant period and are available to provide their disability-related expertise to AJC
staff. For example, there is one in Kenai who is working with TANF services. One who works in
Fairbanks is now the WIOA person. In Wassila, Alaska, the former DRC is now a VR counselor.
The impact of the DEI and the DRC 1 training continues to be felt at the AJCs.
“The DRC level-1 training is mandated for job center staff. The stark contrast is that before we
refined it in our own way, when they were approaching persons with visible disabilities, they
were uncomfortable and would automatically send the clients to VR. After they had the training
on disability etiquette, DRCs helped with the needs of each client to find whatever services might
be needed. Looking at the population of JSWDs, I think we can now provide them with much
better help all around.” — DRC
Regarding TTW, by the end of the third quarter 26 active Tickets were assigned. Of
those, 57 percent were employed and 38 percent were working full-time at or above substantial
gainful activity level. To date, Alaska generated $487,756 in total Ticket revenue. However, they
reported that by the end of the third quarter, the $5,880 in Ticket revenue was a decrease
ascribed to the length of time that it took SSA to catch up on payments.
During the sustainability interview, DEI leadership offered some very insightful
reflections on the DEI. As part of that conversation, they offered that while the concept of
universal access to AJCs for all jobseekers was laudable, the time-limited nature of a grant and
promotion of the universal access concept alone could not hit the mark that dedicated programs
with ongoing funding to attract and serve jobseekers with disadvantages can do. They compared
DEI to the Jobs for Veterans State Grants that are allocated to states on an ongoing basis and
provide dedicated Disabled Veterans’ Outreach Program and Local Veterans’ Employment
Representatives who are located within Centers and whose job very much mirrors what DRCs
were doing, but specifically for veterans. One stated that, in embedding services for all JSWDs at
AJCs, “Frontline staff can help anyone that comes in the door and that is good. But there should
be specialized staff, too. They can handle things that other staff can’t.” Disabled Veterans’
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
112
Outreach Programs and Local Veterans’ Employment Representatives even have an ongoing
training institute that the DEI incorporated into its national initiative, but under a separate
contract, not on an ongoing basis. To paraphrase the Alaska leadership’s recommendations: “If
the DEI is to be an incubator for systems and services change serving JSWDs, then it is best
followed by a nationally administered categorical funding arrangement similar to [Jobs for
Veterans State Grants] that fully incorporates and sustains improved outreach, access, and
services for JSWDs within AJCs across the country. That’s what’s needed. It’s not siloed. It
creates ‘ownership’ of the mission.”
While it is unfortunate that Alaska DEI funds were depleted prematurely, there is
evidence that the initiative made progress at the systems and services levels. With strategic
foresight by training AJC and partner staff in disability awareness and DEI service delivery
methods and creating a statewide culture of collaboration, strengthened in many cases by MOUs,
the Alaska DEI contributed to improving access to WIOA services for youth with disabilities.
They engaged hundreds of employers and made significant inroads into collaborations with high
schools and postsecondary colleges and certificate-issuing CP programs. They were respectful of
the cultures of various Native American tribes and worked with and through their tribal
leadership to reach adjudicated youth with disabilities who might never have had access to
education and employment and training programs. They evidenced creativity and flexibility in
their approaches to services and partnerships, realizing that conditions “on the ground” in rural
and urban communities can greatly affect implementation. That most services and partnerships
continued statewide, even on a reduced basis, with only the DEI State Lead picking up when the
former DRCs had to leave, is in itself a commendable achievement. Obviously, the number of
TTW enrollments, sustaining the level of partner involvement, and the numbers of youth who
could be served across its WDAs were adversely affected when the statewide DEI team was
reduced from six to one. However, the foundation appears to be well established and the
essential features of the Alaska grantee appear to remain and be sustained, only awaiting other
funding opportunities to rekindle their efforts.
B. State: Georgia; Focus Area: Youth with Disabilities
Stated Goals and Objectives i.
The Georgia DEI Customized Career Pathways Project addressed the employment needs
of youth and adults with significant disabilities. The grantee implemented their project with a
focus on Career Pathways by customizing access to credential-based education and training
offered through the community and technical college system in agriculture, health care,
hospitality, and logistics. Their project privileged the creation and sustainability of cross-sector
partnerships and infused evidence-based and promising practices for JSWD employment.
Georgia’s Round 6 goals were to:
● Facilitate and strengthen partnerships with a CP focus between VR, the state WDA,
disability and employment providers, and the community and technical college system;
● Provide expanded access to career technical training and education resources in identified
industry sectors by using AJC services in coordination and collaboration with disability-
specific resources to increase participation in existing CP, and by designing education
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
113
and training programs that are customizable and accessible for jobseekers with significant
disabilities;
● Increase the diversity of job and employer types participating in CP and infusing CE
methods into job placement, including the use of job carving;
● Use IRTs to align policies and programs, including through the use of customized support
while navigating among various collaborative stakeholders;
● Expand access to short-term subsidized work through strategies including WBL
Opportunities, OJT, and other direct work experiences; and
● Provide customized and paid internships rooted in Blended and Braided Resources.
DEI Service Delivery Strategies ii.
DEI SDSs — particularly IRTs and elements of CE and WBL with an intention of
alignment with CP — were incorporated in the Georgia DEI. We heard much about the use of
IRTs. From listening to monthly DEI administrative and TA calls and from grantee interviews,
we found that the concepts and processes associated with IRTs were being absorbed by DRCs
and also by co-funded VR staff.
DRCs were most often viewed as the point person for convening IRTs. But because of
the benefit of having VR staff who also received training in DEI practices — and since VR staff
were co-located with DEI at one AJC — there was a greater likelihood of including VR at the
beginning of the IRT process rather than after the planning ended. DRCs received training and
support from NDI on the essentials of ARC and IRTs as well as on the foundational elements of
Discovery, including individual and group career mapping. CE was not generally implemented
with model fidelity. However, its principles were embraced by DRCs and their VR partners and
aligned with person-centered planning approaches.
47
Acknowledging that CE could be difficult to use with employers, one VR partner offered:
“It can be hard to find employers that are willing to do Customized Employment or supported
employment. Some clients need that. You can train them and some will be ready, but some need
extra support, so you need to get employer buy-in around that.”
The career-mapping approach to Discovery was not only gaining traction within the new
Atlanta WDA, but it was becoming aligned with employment services provided by the AJC:
“Group career mappings are now being provided in conjunction with WIOA services in the
Augusta DOL Career Center — a great partnership! This means that Title I, III, and IV under
WIOA are working together in serving Georgians with disabilities.” Other evidence also
indicated that Discovery-based CE elements were being used.
Georgia also implemented TTW and ENs and tried to recruit beneficiaries into TTW.
Georgia created an EN within 6 months and engaged in Partnership Plus as the DEI/VR
relationship helped to better collaborate with and recruit TTW beneficiaries. Georgia reported
1
“Person-centered planning” refers to methods that privilege the decisions of the participant and engages them
directly in the establishment of their employment goal and development plan. This is in contrast to planning that is
administered and under the control of staff.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
114
steady growth in Ticket assignments and had 40 percent of their goal of 30 Ticket assignments.
They also began to plan and deliver TTW outreach events. The addition of the Atlanta WDA
also meant expansion of TTW to additional counties. The grantee expected the reimbursement
amount to continue to grow, especially after the implementation of SSA outreach events in
WDAs that implemented DEI. Although the TTW program progressed well, DEI leadership
allowed that more needed to be done to better inform DEI staff and partners about when to offer
it to participants: “It’s been successful in that we could get the EN up earlier in the project, but
the EN needs more support within DEI I think.”
DEI SDSs were provided with evidence of implementation across some WDAs. Some
participants enrolled in WBL and eventually acquired jobs. Other participants participated in CE
and Discovery, with an emphasis on the use of career mapping. However, we do not have data
confirming that every participant received Discovery as a required service. IRTs seemed to have
evolved organically from Georgia’s efforts at bringing VR and other partners to the table.
Georgia delivered the DEI SDSs that it proposed, and its service delivery efforts were augmented
by facilitating cross-systems partnerships.
Implementation Summary iii.
Georgia prioritized the development of sustainable cross-sector agency linkages as a
necessary ingredient for creating job opportunities for JSWDs in occupational fields that aligned
with the state’s plans for economic growth, as well as with CP that coincided with the
Governor’s High Demand Career Initiative. The state proposed to better prepare customers for
success in CP-focused jobs and careers by utilizing IRTs, Discovery, person-centered career
planning and ILPs, supported education, and customized apprenticeships and paid internships
that used CE techniques, including job creation and job carving. Georgia’s approach to
implementing cross-sector partnerships and on-the-job work experiences prioritizing CE aligned
with supported education to achieve CP outcomes.
Georgia began by targeting two WDAs that covered a total of 16 counties. Georgia’s
focus was to create opportunities for JSWDs by customizing Career Pathways in high-demand
industries and by facilitating cross-system collaboration and expansion of the capacity of AJCs
to serve customers with significant disabilities. Two variables had to be addressed in achieving
these goals. First, targeting high-demand industries (particularly the focus areas of agriculture,
health care, hospitality, and logistics) meant that the state would need to be successful in
partnering with the technical college system, continuing education institutions, and relevant
credential-delivering entities, and it would need to develop an understanding of the hiring needs
of high-demand industries that it hoped to engage in the program.
Georgia Round 6 also addressed a number of supply-side variables that affected the CP
pipelines, the range of direct services available that aligned with the state’s intent to use CE-
driven methods, and systems-related issues that impeded JSWDs’ access to and use of these
services. For example, all DRCs were certified to provide benefits counseling, and all
participants were offered these services. Georgia addressed the systems-related impediments to
achieving those goals while working to improve JSWD access to AJCs operating in these
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
115
counties and to improve the training and employment services that JSWDs need to support their
CP job and career goals.
In Georgia, coordination at the services level between VR and AJC staff historically
happened infrequently. Georgia’s approach was not only to engage VR as a strategic partner, but
also to negotiate an arrangement to jointly fund one full-time and one part-time VR counselor to
serve as a bridge between VR and DEI. This created fertile ground for leveraging partners’
resources, including Title I resources and TTW. By the end of the project, this coordinated
approach was working well enough for Georgia to request a contract amendment expanding the
DEI to co-hire a second VR counselor to serve the Atlanta Metro area and two additional
counties. Furthermore, Georgia hoped that facilitating familiarity with each partner’s service
approaches and requirements would mean long-term improvements in the services and supports
needed by JSWDs.
Although we did not find evidence that the project was successful in penetrating high-
demand industries, partners including AJC managers, community providers, and VR
representatives acknowledged that the presence of DEI strengthened the capacity of systems to
support JSWDs. Georgia did succeed in aligning services and improving systems capacity to
help participants with disabilities prepare for training and employment.
Partnerships and Collaborations iv.
The Georgia Round 6 grant intended to develop and grow the collaboration between DEI,
Title I, and VR to address issues of service coordination. The DEI State Lead asserted that cross-
agency service integration was crucial to enabling long-term support and sharing service
burdens, including CP. Typically, there are many “hoops to jump through” to realize
collaboration between Title I and Title IV, systems that “don’t know each other.” DEI provided
an avenue to spur a “beautiful marriage” between Workforce, VR, and CP; “it was like magic
happening” with “profound” benefits. The most common shared services included vocational
counseling and guidance, Title I case management services, Discovery career exploration,
assistive work technology evaluation, equipment, and job development and placement services.
The DEI State Lead and DRCs were acknowledged for their leadership in creating and
nurturing partnerships with WDAs. One AJC manager offered that the “DRCs have made a lot of
effort to grow partnerships.” As well, the WDA manager made the observation that the DRCs
did much to strengthen rapport and trust between VR and the AJC, including overcoming
misconceptions that the DEI was duplicative of VR services. Partners also had positive things to
say about the DEI, such as the Salvation Army and ARK. Both partners provided referrals to the
program and AJC WIOA teams with which the DRCs interfaced.
The DEI/VR partnership and co-funding of two VR counselors was a potentially
replicable best practice to overcoming service silos that occur between VR and workforce
programs. Georgia’s efforts to develop Partnerships and Collaborations was one of the strongest
aspects of its project, and data demonstrates that the initiative succeeded in this critical
component of systems change. One AJC manager offered corroboration of the effective
DEI/AJC relationship: “People refer to the DRC all the time. The other thing with the DRC is
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
116
that she shows me the contacts she’s made and the partnerships developed, and we have a
tracking spreadsheet for all of our customers that indicate if they’re DEI. I’m not in the dark.”
We heard from WIOA and VR respondents that both systems could do a better job of
employer engagement and of coordinating their respective business development services so
JSWDs can have an equal shot at getting a job. The methods for engaging employers through
AJCs — where treatment WDAs would promote their job candidates with disabilities —
appeared to be through job fairs, brochures, and internet marketing. One VR representative
suggested that DEI contributed to more robust employer engagement.
As of Georgia’s final quarter of the grant, promising steps were made in connecting
JSWDs with training and in linking employers with qualified applicants. The state noted that a
number of (unspecified) participants received on-the-job work experiences and that most of them
were expected to move into permanent unsubsidized employment at the end of their trainings.
Five participants in the newest Atlanta site were in the process of obtaining eligibility for WIOA
Title I training, and in Augusta one participant was being considered to start training in the new
Cyber Security Training Center. One employer who DEI partnered with to customize a job for a
client with a significant disability requested more referrals, including from VR. It was also noted
that it was more difficult to engage participants, partners, and employers in the rural areas that
the project served. Georgia Round 6 made a good effort to implement employer partnerships and
assisted participants in obtaining jobs that privileged CP.
Individuals who participated in the DEI had positive things to say about their
involvement with employers and the type and level of services they received. There appeared to
be a consensus on staff responsiveness to the needs and interests of JSWDs and on the attempts
to accommodate the impacts of their disabilities on their training and employment. One JSWD
testified: “When we first talked, I had a lot of interests, and she acted like everything I said
mattered. She made it all about me. She has her requirements she has to follow, but they’re
helping me pursue what I want to pursue.”
However, one customer expressed some dissatisfaction with the pace of service delivery
and suggested that the DEI could do better by increasing its staffing level and strengthening
partnerships with employers: “The process can be slow. It could be faster. Maybe they need more
employees. I feel like I’m a month behind. And there’s occasionally a disconnect between
employers and JSWDs. They don’t always follow equal opportunity stuff.”
There was positive indication that JSWDs were productively accommodated. One
customer relayed that a DRC secured an adjustable chair with a footrest and headrest to help
JSWDs sit more comfortably in an AJC. Georgia Round 6 also largely implemented the intended
SDSs with a customer focus, particularly regarding the implementation of IRTs. Less evidence
supports the use of the full range of CE (e.g., job carving, negotiating, and job creation with full
“In VR, we use employment consultants, and there’s a new division that focuses just on that. I
know that WIOA people in DEI do more business outreach. Now we have the benefit of both
of those [resources] because there are two teams looking for employment opportunities.” —
VR Representative
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
117
employer participation in the process). However, DRCs were conversant in the principles and
practices associated with Discovery, which is quintessentially a customer-focused assessment
and planning method. Participants submitted that in the early stages of goal-setting, JSWDs
engaged in planning and job development to a great degree and their preferences for employment
and training were respected.
Career Pathways v.
A CP job development focus required Georgia Round 6 to connect JSWDs to college and
credentialing programs that provided them with the qualifications to compete effectively for jobs
in high-demand industries. At the other end of the pipeline, we searched for evidence that the
DEI was engaging those high-demand industry employers and facilitating their interest in hiring
qualified JSWDs. We found some evidence that those goals were being met, although perhaps
not on the scale that was envisioned.
Although the complete CP continuum that was presented in Georgia’s proposal was not
in evidence across the three DEI WDAs, we found a promising general orientation toward CP
services with the majority of the customers served by the DEI. There is also evidence that
Georgia’s DEI leadership remains committed to focusing their CP development efforts on
segments of the labor market that included high-demand occupations.
“Both [treatment] WDAs have received sector strategy development grants from our office in the
past 7 months. Both of them picked manufacturing, which is handy because we identified that in
the DEI grant, so there’s potential for integration. They’ll work with employers in those areas to
develop those pathways.” — WDA Staff
In addition, there is evidence that the DEI and its AJC partners worked to strengthen
collaborations with technical education schools to better prepare participants to attain credentials
associated with high-demand occupations. One partner proclaimed, “Our partnership with the
tech school is helpful. We send out the curricula and are open to tweaking them for the
employer’s needs.”
Outreach and Dissemination vi.
Georgia DRCs may continue to use DEI strategies and be involved in communication
with external partners and potential referral sources through presentations, signs, banners, and
other methods. Partners who were aware of the DEI were anxious to refer customers to it. In part,
this was because DEI had easily accessible resources for training and support and that the DRCs
were highly regarded in the WDAs.
Individuals with disabilities entering the AJC were referred by Employment Specialists to
the DRC when appropriate. By co-funding two counselors with VR, the DEI effectively opened a
line of communication within VR from which referrals to the project could flow. However, it
appeared from the perspectives of some of the partners that we interviewed that the DEI could be
among the state’s “best kept secrets.” More needs to be done to advertise DEI services after the
grant period.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
118
Promising Practices “Pathways for Success” vii.
Rationale for Its Implementation
Prior to DEI, a customer with a disability typically received an automatic VR referral.
Workforce and VR systems tended to operate independently, where JSWDs were likely to be the
responsibility of VR only rather than integrated in a universal employment services system.
Furthermore, it was not a given that VR counselors would receive training in emerging effective
practices like IRTs or CE.
Of particular significance was the DEI’s and VR’s co-funding and supporting of initially
one — and later two — VR counselors who traveled between WDAs to work with customers.
This was a significant innovation since historically many state disability initiatives nationwide
have had difficulties engaging VR as a contributing partner. Not only did the jointly funded
counselor assist jobseekers to take advantage of opportunities that the DEI, AJCs, and VR had to
offer, the jointly supported staff took on the role of bridging systems and as a translator to help
facilitate increased understanding of each partner’s culture and services.
Why the Practice Could Be Considered Promising
By training co-funded counselors on these techniques — coupled with the training they
usually received as VR counselors in using supported employment methods — the Georgia DEI
filled a gap in cross-agency training. VR and WIOA staff and the DRCs endorsed the co-funded
VR counselor as contributing to the nascent development of a more collaborative, seamless
approach that bodes well for sustainable collaboration.
However, even though having a co-funded VR staff assisted in reducing silos and
facilitating local systems change, the state-level relationship between VR and the workforce
system was challenged by significant turnover among VR senior staff. Nevertheless, by working
diligently at the state level to sustain communication and collaboration in part through senior-
level meetings and presentations, Georgia drew the two systems together to achieve systems
change. The State Lead attested that Georgia was “sharing the load and expertise across systems
and getting a lot of benefit.” In the meantime, at the local level, the co-funded VR counselor
helped to demonstrate how the VR and WIOA systems could align in ways that provide direct
benefits to VR and WIOA co-enrolled customers. In addition, staff from DEI, WIOA, and VR
agreed that they would explore how to sustain the position before grant funds ended.
Challenges and Sustainability viii.
The Georgia DEI sustained some turnover at the DEI leadership level, and for a time the
DEI State Lead was away on leave. NDI training, as well as our planned site visit, was put on
hiatus for a number of months in 2017. Regular TA and administrative teleconference calls
resumed in 2018, and our site visit occurred in April 2018. Although the program took some time
to build momentum and increase staff knowledge and capacity, after stabilizing it generated
momentum and appeared to make good progress in almost all areas. As referenced above, it
appears to us from available data that the hoped-for intensive focus on CP and on building
employer partnerships did not occur at the pace that Georgia anticipated. Therefore, it appears
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
119
that Georgia Round 6 was more heavily weighted on the supply side (participant preparation) of
the employment equation than on the demand side (strategic and implementation involvement of
employers). Also, the state’s amount of TTW revenue, while expected to increase, may be
insufficient for the DEI to continue to co-fund the two VR counselors after the grant period. In
essence, Georgia’s challenges were not unlike those of many DEI grantees in that they were slow
to start during the first year, made progress in the second year, and gained good traction by the
time the 3-year grant ended.
As referenced above, there are three elements of Georgia Round 6 that bode well for
sustainability: (1) the strength of AJC and VR partnerships implied that both systems were
becoming more informed of DEI practices and could do a better job serving JSWDs; (2) co-
funded VR counselors embedded DEI practices within the operations of VR, which can help
sustain services even in the absence of DRCs; and (3) the stream of TTW revenue, although
modest during the grant period, could be a source for potentially sustaining co-funded VR
counselors if it continues to increase.
The DEI State Lead confirmed that TTW activity will continue after the grant period,
including by facilitating more local areas in becoming ENs. The DEI State Lead was also hired
in a new “Disability State Lead” role and is charged with expanding DEI strategies, such as
IRTs, accommodations, and AT statewide through developing a “robust technical assistance
system.” Toward the grant’s end, AT partners created webinars on serving JSWDs and
harnessing AT. State leadership also created an “online academy” for local workforce areas,
which will include guidance on serving JSWDs. In addition, the DEI State Lead, VR, and the
Workforce Innovation Technical Assistance Center are planning a demonstration grant to
promote IRTs in local areas that were not DEI treatment sites. As well, Work Incentives
Planning and Assistance (WIPA) services will be expanded in one treatment WDA where
demand was high. Finally, the Georgia State Lead desires to expand Project SEARCH across the
state. Two Round 6 DRCs were hired by Title I at grant’s end; one is continuing DRC
responsibilities and the other is focusing on employment services. The full-time co-funded VR
counselor was hired by VR to work with clients with developmental and mental health
disabilities.
There is much to commend the Georgia DEI Round 6 project for, including its strong
leadership, productive partnerships, and the strength of the employment preparation services it
provided to JSWDs. While more needs to be done to align job preparation and placement
functions with high-demand occupations as well as to improve employer engagement, we believe
that the state has these goals in its sight. Georgia’s challenge will be how to keep momentum
going for months and years after their grant period.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
120
C. State: Hawaii; Focus Area: Individuals with Significant Disabilities
Stated Goals and Objectives i.
Hawaii Round 6 specified the following goals and objectives:
1. Increase AJC staff competencies to provide critical local resources by providing training
on Disability 101, Customized Employment, Career Pathway systems, job
accommodations, Asset Development, ILPs, and Benefits Planning.
2. Leverage funding and resources across public workforce development systems from
partners to provide additional services to individuals with significant disabilities,
including IRTs, Asset Development, and Partnerships and Collaborations.
3. Expand AJC capacity to effectively serve individuals with significant disabilities. Hawaii
will employ a direct mail strategy via social media to recruit individuals with significant
disabilities to utilize AJC services. In addition, AJCs will strengthen relationships with
disability-serving agencies to allow for multi-point entry/exit.
4. Utilize CP strategies to complement AJC services with flexible accommodations for
individuals with significant disabilities.
DEI Service Delivery Strategies ii.
The pre-transition DRCs prioritized IRTs, Blending and Braiding Resources, CE, and
Asset Development. Blending and braiding funds and leveraging resources, including TTW,
were implemented to a greater or lesser extent on all of the islands during Round 6. Because of
DRC turnover and the difficulty in convening the first cadre of part-time DRCs, both new and
old DRCs required retraining, which primarily focused on methods associated with Discovery,
including career mapping principles. However, we did not hear or see that CE was being used
with fidelity, and there is no information in quarterly reports that provide an indication of the
quality of SDS implementation. Likewise, for Blending and Braiding Resources, we did not
obtain data about Partnership Plus relationships with VR, and since VR was not on an Order of
Selection, we suspect that any progress in that area was stalled. And, we have no indication that
Asset Development services were being offered, although the Center on Disability Services
conducted Benefits Planning training in early 2018 and one on Asset Development was reported
in April to be held soon. Generally, we found an interest in, and some examples of, Hawaii using
some aspects of IRTs and CE but without the scale that they had intended.
Ramping up ENs and TTW services was a challenge for Hawaii. Prior to the departure of
the previous State Lead, there had been one DRC who was cleared for suitability determination.
But, since the loss of the statewide lead for the grant, the new DEI leadership expected that
Hawaii would, for the second time, have to obtain clearance for the TTW program before they
could reestablish services. In April 2018, we heard that income from Tickets was being
generated, and on Maui they recently received their first milestone payment. Otherwise, our data
indicates that more progress was needed to reenergize the TTW program.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
121
During the final year of the project, Hawaii’s request to amend the Scope of Services to
provide DEI participants with (1) paid work experience; (2) OJT; and (3) supportive services was
approved by USDOL. Outreach and engaging customers appeared to be a challenge for the first
cohort of Round 6 DRCs. A contributing factor was that they had limited time allocated to the
program that was more than filled by working directly with customers who came to the Center or
who had been referred by VR or other sources. It was apparent that DRCs were struggling with
the most effective way to get the word out, and we heard that even when it came to publicizing
the program to obtain referrals, the cultural nuances of working in Hawaii had to be honored.
And, even more recently, we heard through follow-up interviews that appropriate and effective
methods of outreach and dissemination continued to be a challenge that many DRCs faced.
Our overall assessment related to SDSs is that there is not enough evidence to confirm
that Hawaii’s Round 6 DEI was using the DEI strategies that they proposed to use to build
sustainable systems-wide capacity for serving JSWDs in AJCs, nor can we state with certainty
that the project met the individual outcome goals that the state identified when they applied for
the grant, primarily due to the constraints described above. Evidence tends to corroborate that
AJC JSWDs received employment services from their DRCs and accommodations were
available, although AT equipment was not always accessible to customers at some AJCs.
Capacity-building also includes how well the program institutes and maintains an effective
outreach, referral, and dissemination plan. Despite these challenges and concerns, the capacity
for the DEI to increase the numbers of individuals it served appeared to grow.
Implementation Summary iii.
Hawaii Round 6 was implemented in four WDAs: Hawaii (Big Island), Oahu, Maui, and
Kauai. Hawaii and Maui had the clearest orientation to DEI practices and Partnerships and
Collaborations, which may be related to their prior experience implementing their Round 2 DEI
grant. While staff turnover and the predominance of part-time DRCs challenged each of the
WDAs, an understanding of core DEI practices was particularly less developed on Oahu and
Kauai. The economies and also the infrastructure (transportation, access to AJC services,
employer commitment to hiring people with disabilities, access to skills training and VR
services, etc.) for creating job opportunities also varied across the islands. For example, Kauai
obtained a large solar grant where JSWDs enrolled in solar training, which led to job
opportunities at solar farms. Health care opportunities, including registered nurse certifications
and related allied credentialing, were also available.
The state has three population centers, which creates a challenge for JSWDs to access
training and employment opportunities if they live too far north because most job opportunities
are located on the east side of the island. Transportation options are generally more available on
Oahu, facilitating access to jobs, and the dispersion and extensiveness of the tourist, health care,
and retail industries offer a great variety of potential training and career options for JSWDs than
are available on the other islands.
“Outreach — but you need to go out. Ads and marketing don’t work. You need to go out. It’s
word of mouth. That’s part of our culture.” — DRC
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
122
In late 2017, the contract for WIOA services and AJC operations that the Workforce
Development Division (WDD) had was let to another contractor. On Hawaii, WDD is no longer
the AJC operator or the Title I provider; the county is providing both functions. On Maui, WDD
has an extended contract for providing WIOA Title I services, but the county operates the AJC.
The youth provider is Maui Community College. The DEI does not have a staff person in Maui.
On Kauai, WDD is the WIOA Title I provider and the county operates the AJC. The transition
did not affect Oahu, which is the provider of adult and youth services. It also runs the AJC. The
change in contractors precipitated a significant turnover of DRCs when those who were not
retained by the new contractor were laid off.
Hawaii presents a unique situation. When Hawaii received a Round 8 DEI youth grant in
late 2018, their DRCs were a mix of in-place Round 6 DRCs and new DRCs. All past and newly
hired DRCs were assigned to work with both Round 6 and Round 8 JSWDs. Hawaii renegotiated
their regional DEI contracts to ensure that the DRCs in place were not impacted by the
uncertainty of the regional and center-level Title I contracts to assist in continuity.
TA providers and the University of Hawaii met with the DEI State Lead in March 2016
and TA activities were eventually resumed. TA providers worked with the DEI State Lead to
develop a new customer flow for participating WDAs and engaged with newly hired DRCs in
April and May of 2018 to bring them up to speed on DEI implementation and the use of SDSs.
Also in 2018, the long-tenured state VR Director retired and the progress that had been made to
that point in creating better cross-systems partnerships was significantly slowed and had to be
regenerated by his replacement and a new DEI State Lead. VR went to an Order of Selection as
well, which significantly narrowed their doorway to accept new applicants. TA ended when the
national DEI training contract was assigned to another vendor during the year. That left the
University, new DEI leadership, and a mostly new cadre of DRCs to essentially complete the
work of Round 6. Hawaii determined to make the best of the situation and essentially picked up
capacity-building with an almost-fresh start. The University began to provide fundamental T/TA
to the new DRCs, who were then doing “double duty” serving customers from Round 6.
Much of the training they gave to the first cadre of DRCs had to be retaught to the new
group. Refresher training to the DRCs who were still with the project was also necessary. They
were allocated to the project with 0.25–0.5 full-time equivalent per person. The new cadre was
allocated to the project at a greater full-time equivalent (average of .42% among six DRCs) than
their predecessors, allowing them to focus their time more heavily on DEI. However, capacity-
building is a function of how well staff remain with the project and develop and transmit
“institutional knowledge” to promote sustainability. Regarding the impact of staff turnover on
sustainability, DEI leadership was asked if any of the Round 6 DRCs stayed on past the end of
that grant and were therefore available to contribute their knowledge to future other AJCs: “The
Oahu people are the same. On the Big Island things turned over. Maui turned over, too. There
are none on Kauai. On all of the Islands we only had three–four DRCs.” Thus, Hawaii had to
revisit many staff capacity-building activities midstream with new staff.
Data is limited in analyzing how well capacity-building occurred since the changeover
because some of the more knowledgeable DRCs and DEI State Leads are no longer with the
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
123
project. Nonetheless, Hawaii regained some of the ground it lost for its Round 6 project. The
DEI State Lead was also involved in Round 6 and had a position of authority within the Hawaii
Department of Labor. This provided a level of leadership continuity that could benefit the
program and contribute to its ability to partner with other state agencies, including VR.
Acquiring a Round 8 grant gave Hawaii the opportunity of having the Round 6 DRCs allocated
at substantially higher full-time equivalents, which may have been beneficial to JSWDs served
through Round 6.
T/TA was provided through limited on-site and distance learning methods. While these
two training entities established a dialogue and preliminarily agreed to provide complementary
and non-duplicative training, this did not begin to gain traction until well into the second year of
the grant. By that time, disruptions to the program were already beginning to occur, including
major DRC turnover due to a change in the AJC contractor described above. The staff turnover
made it difficult for both providers to deliver any training with regularity and
comprehensiveness. While the University trained DRCs, its Center on Disability Services
remained the primary DEI training entity. Since a number of DRCs who had received training
had left the program and new ones took their place, a significant amount of training on
previously trained topics also had to be repeated.
Another capacity-building challenge was the amount of time that the first cadre of Round
6 DRCs could spend on DEI-related responsibilities. Since the majority of their time was
devoted to AJC-related obligations outside of the DEI project, the DEI State Lead encountered
resistance from stakeholders who objected to the time demands placed on the DRCs for
participating in training sessions. This made it difficult for them to not only absorb but gain
experience applying their training while working with JSWDs. Some DRCs indicated that the
full-time equivalent allocation issue and staff turnover at AJCs were not new challenges and that
staff typically needed to perform a number of job roles.
Hawaii Round 6 intended to enhance the capacities of participating AJCs to improve
employment outcomes by providing a program specifically focused upon outreach and service
delivery for JSWDs, privileging the use of DEI services, including IRTs, ARC, and CE. IRTs
were implemented on the Big Island and Oahu and to a more limited extent on the other two
islands. In essence, across all treatment sites the DEI could be seen as a “lightning rod” attracting
JSWDs to AJCs, but with the potential for developing strong Employment Specialist-customer
relationships and employment support services in ways that people with disabilities may not
have experienced at AJCs prior to DEI. With the caveat that we have no present evidence
indicating that AJC cultural change continued with the transition to the new WIOA contractor,
some level of DEI-AJC cultural change was beginning to occur prior to the transition.
“The DRC funding allocation was part-time, and getting the attention of jobseekers was
difficult. Even small things, like getting a manager’s approval for training was an issue.
Once we were awarded DEI R8 (which included some Round 6 DRCs), we reallocated staff.
Now, we don’t really have part-time staff. All the R6 DRCs are full-time from now on, and
those that aren’t will be blended/braided to focus on VR-related activities or other things that
relate to disability.” — DEI Partner Representative
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
124
Although customers we interviewed were almost always unaware that the services they
were receiving were specifically from the DEI rather than from an “improved” AJC, it appears
that the presence of a DRC was important in helping them get jobs. The most frequent indication
of AJC capacity-building to support JSWDs was participants’ expression of satisfaction that they
were getting the types of hands-on, supportive services they needed to succeed in employment.
Partnerships and Collaborations iv.
When describing the genesis of cross-agency partnerships in Hawaii, it is important to
underscore the influence of the state’s culture as a contributing factor. Data obtained from
interviews with DRCs, DEI leadership, and partner agencies corroborated the importance of the
value that Hawaiians place on developing personal relationships and a sense of community. We
heard on a number of occasions how a DRC, a DEI leader, or an AJC manager knew a leader or
staff from another agency because they knew their family, or went to school with them, or some
other friendship-related association. Relationship-based partnerships, while often challenged by
partner self-interest or administrative challenges, were nonetheless both necessary and culturally
compatible and a critical element in Hawaii’s attempts to build systems and service capacity.
VR partnered with DEI leadership and DRCs. VR representatives acknowledged that the
partnership was sometimes difficult, with role differentiation concerns expressed by both
partners. DEI leadership met with the new administrator, who expressed willingness to
coordinate services. With VR on Order of Selection, which increased the difficulty that people
with disabilities had in accessing VR services, having the DEI work with jobseekers and with
WIOA opened up opportunities to discuss how to partner. Initially, VR was reluctant to partner
with DEI on summer youth programs, but the new VR administrator recognized the need,
especially for Pre-Employment Transition Services. The DEI and VR partnership on summer
youth employment programs continued throughout the grant period, but because they increased
their full-time equivalent allocation for their Round 6/8 DRCs, these projects were accelerated as
DRCs had more time to work on them.
Other partners included the Department of Developmental Disabilities, Adult Education,
Developmental Disabilities Council, Department of Mental Health, and Medicaid Home and
Community-Based Services. On Maui, the DEI connected with the Developmental Disabilities
Council coordinator. This was characterized as a relationship that helped the DEI serve more
“[The DRC] is just one person, so it’s hard for her to provide all of these services. I guess I
don’t see a difference? She provided these services to me a while ago. I’ve known her for a
while. I like my services from [the DRC]. There’s no other staff, just her. She checks in with
me, makes sure I’m going to class, and asks if I need any more services. She emails me or
calls me. She keeps me in the loop. I would say she contacts me two or three times a month.
She’s helped me look for a job. She helped me apply for state positions.” — DEI Participant
“When I heard about the DEI, I just thought of it as another case manager. I didn’t really
know how well it works.” — VR Representative
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
125
individuals with intellectual and developmental disabilities. Anecdotally (and particularly
relevant as the DEI focuses on youth with Round 8), they needed to connect with education, and
that connection was described as challenging. Although we heard that they had not made great
progress on Oahu, they connected with the high schools in Kona. They reported that the DEI had
success in expanding its partnership with Waipahu.
The DEI State Lead and DRCs also coordinated and participated in meetings to link
WIOA Youth and Adult programs in an attempt to braid resources from the various agencies to
provide youth with wraparound services. The DRCs assisted in the process by providing
information about accommodations, diversity, and inclusion in the workplace. WIOA Youth and
Adult programs provided information about the possibility of placing youth in “work
experiences” to help them explore various occupations while having opportunities to learn and
practice the necessary soft skills to succeed in employment.
Although JSWDs may not have associated the services they were receiving specifically
with DEI, a number of them praised the services and supports they received from their own
DRC. Most typically, they talked about getting access to workshops on résumé writing and
developing career choices, but much less about managing their benefits while working and had
little to no familiarity with TTW. JSWDs drew positive associations between the help they were
getting and a particular DRC’s knowledge, expertise, and willingness to listen to them and
encourage their input into their employment plan. When one customer was asked whether he felt
that he had a say in his training and employment plan, the customer replied:
There may be a plausible reason why Hawaii DEI JSWDs across the four participating
islands’ AJCs did not easily equate the services they were receiving as generated by the DEI. All
pre-contract transition DRCs spent most of their time on other non-DEI tasks at the employment
center as Employment Specialists, and in many cases had been employed there for a long time.
They simply did not have enough time allocated to engage in disseminating information about
the program to their communities and constituents. While the DEI was operating, it is easy to see
that customers were likely to regard services as just better, but not as having been created by a
new and distinct program.
Hawaii implemented AJC job fairs to reach out to employers. DRCs also designed and
implemented reverse job fairs in which individuals interested in employment set up booths to
showcase their employment credentials, including a résumé, education and training, and
recommendations from prior employers. This occurred while employers and recruiters walked
around the job fair and asked job candidates to discuss their skills and interests. A recent reverse
job fair in Hawaii was attended by 154 employers, legislators, and college and high school
students. DRCs reported that approximately eight students were successfully placed in
employment and three students received job offers. Access to JSWDs through employer
participation in AJC job fairs was also corroborated by the employer we interviewed. And
“Definitely, and they come up with ideas I never thought of too. This is a great office. The
DRC is great; she doesn’t give up. This is probably the best office I’ve been in. I was in
another for about 1 week.” — DEI Participant
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
126
although the employer representative indicated that his company hired two people with
disabilities, he was unclear about whether or not they were served through the DEI or were
referred through regular AJC channels: “I don’t really know what the DEI is, besides the
acronym. I make all the hiring/firing decisions at my company/branch. I don’t look at if they’re
in a program, just whether they fit the job.” Hawaii employers may have a willingness to
consider job applicants with disabilities who meet their hiring needs, but a more strategic
approach to engaging employers is necessary, including one that focuses upon Career Pathways.
Career Pathways v.
With Hawaii partner the University of Hawaii, Center on Disability Services, as the
primary T/TA entity, the grantee continued to provide new training content to DRCs, including
web training on the six Career Pathways available in Hawaii: Natural Resources, Health
Services, Business, Public and Human Services, Industrial and Engineering Technology, and
Arts and Communication. In March 2019, the DRCs attended WWL training conducted by a
national expert to enhance their understanding of how best to work with youth with disabilities
and prepare them for working with summer youth participants. In addition, the DRCs attended
the Job Accommodation Network training to gain knowledge that could assist them in working
with employers to provide on-the-job supports for employees with disabilities.
However, we found no compelling evidence that Hawaii’s Round 6 DEI had implemented
and sustained partnerships with community colleges or credential-issuing technical schools in
support of a CP approach, although jobseekers appear to have been provided information about
the types of jobs employers were seeking to fill in their regions.
Outreach and Dissemination vi.
As discussed earlier, getting the word out about the unique services of the DEI was a
challenge, particularly with very part-time staff. Some partner agencies also appeared to be
unclear about the DEI. But another challenge was that, by design, all outreach was supposed to
be done by direct mail from the SSA list given to the EN. However, when they got the TTW
clearance, SSA told them that they stopped giving out that mailing list. Although we heard that a
request for clarification and advice was sent to SSA and ODEP, Hawaii did not hear back. It
meant that the first cadre of DRCs would have to do unanticipated work regarding outreach
when it had not been written into their grant proposal. It is unclear if improved dissemination
practices were proposed in their Round 8 proposal that could have been used in the remaining
months of Round 6 since we do not have access to Round 8 materials. Also, since ramping up the
TTW initiative created difficulties discussed in Section IV, accessing SSA lists would be
problematic even if it proved possible.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
127
Promising Practices vii.
We are unable to identify a particular promising practice designed by Hawaii Round 6.
We have identified a characteristic of the grantee that may be instructive for other DEIs:
resilience of staff and partners in their efforts to overcome significant structural challenges and
deliver services to as many JSWDs as they could after they dealt with staff and leadership
disruptions. Despite disappointment that so many obstacles occurred, the leadership and partners
we talked with always evidenced optimism that over time Round 6 would achieve success even
though they may not have been able to meet their performance targets. That they persevered
while salvaging and moving forward with their Round 6 grant speaks well of their determination
to improve competitive, integrated employment opportunities.
Challenges and Sustainability viii.
Hawaii Round 6 encountered a number of challenges in building the capacity of their DEI
and WDAs to achieve their stated goals and improve services for JSWDs. The first relates to the
DEI’s ability to train, support, and sustain its DRCs, thereby ensuring continuity of DEI SDSs on
behalf of its customers. The project was implemented with two training providers whose training
offerings often overlapped. The Center on Disability Services continued its association with DEI
as a training provider from the time of their Round 4 grant. The Center on Disability Services,
DEI partners and collaborations, and DRCs regularly made trips to meet with staff on all islands
and maintain consistent, responsive communication for problem-solving and knowledge
exchange.
Hawaii Round 6 challenges were administrative, structural, and service related, but
perhaps a valid observation was that they encountered barriers that were cultural as well. We
heard time and again that it was difficult convening people and that scheduling events was a
challenge, attributable to an island culture that has a different orientation to time than on the
mainland. On a number of occasions, our scheduled interviews just did not happen because a
person never made it to the location even though transportation had been arranged. But cultural
influences may even affect job applicant skill sets as this employer referenced.
Other challenges emerged that are in many respects similar to those faced by JSWDs.
These included lack of transportation, lack of market-relevant skills, fear of losing health care
benefits, and perceptions of people with disabilities as not “employment ready.” During the final
sustainability interview, leadership noted that another challenge associated with Round 6 was
how the DEI was perceived by participating AJCs: “The DEI program is supposed to connect all
the systems that are in place. It’s not in isolation. Instead, the DEI was seen as a program rather
than a system-building situation.”
When asked during the final sustainability interview about the services that were of
greater importance and impact in addressing their DEI Round 6 goals, the DEI State Lead
responded that blending and braiding was very helpful, and to some extent the IRTs as well,
noting that the IRTs were where the blending/braiding happened. They also offered that
blending/braiding was harder after VR went to Order of Selection; they could not offer resources
beyond what they had available under those restrictions.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
128
Evaluating the Hawaii Round 6 DEI is difficult because of the myriad challenges and
changes that the program underwent as well as the long period of time when we were
unsuccessful in gathering current information through either attendance at regular DRC
teleconferences that were suspended or through review of quarterly reports, where insufficient
detail was available to us to accurately report on their continued progress across all Systems
Change Coding Scheme domains. In addition, we were unsuccessful in extracting and analyzing
Participant Individual Record Layout (PIRL) data to substantiate individual outcomes
attributable to the project. Although we conducted follow-up remote training for the new DRC
cadre on the purpose and process of the independent evaluation, including a tutorial on entering
data into the Participant Tracking System, our contact with and information about the progress of
DRCs was limited.
Despite these considerations, there is much to commend and also to be appreciated in the
Hawaii DEI effort. They addressed their lack of full-time equivalent DRCs by creatively co-
tasking DRCs with both Round 8 and Round 6 responsibilities at higher full-time equivalents.
Their partner, the Center on Disability Services at the University of Hawaii, has been with DEI
since Round 2, is well-respected for its expertise in CE and other DEI practices, and makes its
team accessible across the islands with regular site visits. It is also a T/TA team that understands
how the pace of life and the importance of role relationships on the islands can affect the culture
of work and thereby nuance the delivery of formal training. DEI leadership noted that their
Round 6 grant was a game-changer and that they expected that, because of the introduction of
DEI SDSs at participating AJCs and the partnerships they developed, IRTs and blending and
braiding will continue. “With all the changes and bringing in partners and the blending and
braiding, we have WIOA funding and VR for those persons who qualify. For youth we have DEI
Round 8. If we can find apprenticeships programs, that would be good. We have summer youth
programs from grants.”
While the project did not develop a strong familiarity and working knowledge of CE,
leadership noted that using some elements of CE helped catalyze a more person-centered
approach to serving people with disabilities at their AJCs. As one leader offered, “We want to get
into a less intense kind of Customized Employment. Everyone is supposed to be doing assessment
and ‘Discovery’ is assessing. Maybe we don’t understand Customized Employment fully.”
Although by most objective standards it may seem that Hawaii Round 6 was not as
successful as they had hoped, it is apparent that some significant gains were made despite their
implementation challenges. Perhaps the impact of the Hawaii Round 6 DEI is best expressed by
a member of its leadership team who said: “It was a valuable program. It extended awareness of
a talent pool that people didn’t know about. That generated more interest in our work. It
increased awareness. We’re sold on it. The governor’s office is sold on it, too. We have a
connection with that. Without the DEI grants, we wouldn’t have this awareness of employment of
JSWDs. It takes ongoing persistent work and effort to make people aware of the situation.”
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
129
D. State: Iowa; Focus Area: Adults with Disabilities
i. Stated Goals and Objectives
Iowa has been active with each of its DEI grants, including collaborating with operations
managers, supervisors, employers and WIOA partners. DRCs report limited access to education
and training for JSWDs and “many stigmas” that hinder their hiring. DRCs and Employment
Specialists focus job seekers’ skills and access to accommodations. Individuals interested in
employment and/or training meet first with a DRC to discuss available support services and
potential barriers that may include limited literacy, employment experience and/or outstanding
adjudication and/or fines. Some job seekers are transferred to VR to prepare for work or to retain
employment through job coaching, while others may enroll in internships and/or work-based
learning opportunities to acclimate to new work environments. A lot of what happens when
individuals enroll in DEI depends on where each job seeker is in the workforce development
system. Some job seekers enter the workforce development system through Iowa Works where
they go through an integration process that includes registering with the AJC and determining
what level and type of service each individual requires.
Many Iowa job seekers connect with local community colleges to engage in academic
and/or Career Pathways services. Individuals with blindness or autism, for example, may receive
instruction on desktop technology such as operating printers. Others may learn soft-skills to
engage with colleagues and function within a certain work environment. DRCs play a leadership
role in this process by helping JSWDs find employment and training. Their role is to collaborate
with job seekers to determine their level of interest in employment and training services and help
them reach their goals of employment, housing and/or transportation.
All Iowa DRCs are Certified Work Incentive Counselors (CWICs) that meet with WDA
leaders once per month to discuss the influx of JSWDs and their employment and training needs.
Some DRCs prepare quarterly reports to state workforce agencies, Title I Adult Dislocated
workers and youth operations managers in the three Iowa WDAs. While one Supervisory DRC
oversees the project, all of the DRCs and Employment Specialists serve JSWDs. The latter is
more of a subject matter expert that oversees the intake, training and job placement processes for
a given WDA, than a Case Management oriented DRC. Iowa also implements WIOA services
across the state through a referral system that uses multiple AJCs and their core partners. The
services that are provided depend primarily on the type of disability and level of severity of each
JSWDs, as well as their career plans and interests. DRCs also collaborate “on how to help
JSWDs” depending on their stated needs and concerns while concerns related to compliance and
accessibility depend on the needs of each job seeker with support from a DRC or Employment
Specialist.
DRCs also review partner locations to ensure that they are accessible, both physically and
programmatically. Communications with partners within the treatment WDAs are monitored to
ensure that they are available to all JSWDs and avoid duplication of staff and services. Partners
include WILA Adult and WILA Youth services for the blind, Youth Development Services,
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
130
Adult Education and Literacy, the Department of Collections and WIOA mandated partners.
Iowa provides full-day training for job seekers, including tours of local businesses and
occupational role-playing related to workplace accessibility and etiquette.
Iowa collaborates with the local prison system where incarcerated individuals with
disabilities comprise over 80% of all inmates in the state. In order to identify these individuals
before they are released, DRCs and community service providers discuss each job seeker’s
employment related interests as well as barriers to employment, which provides information that
can help individuals prepare for employment. During this process, job seekers may attend job
fairs, practice interviewing skills and prepare to answer specific questions about their
background and work experience for employers. DRCs facilitate this process by meeting with
each job seeker that enrolls in a WDA to discuss their employment backgrounds and interests.
ii. Proposed Service Delivery Strategies
DRCs focus on systems change by creating universally accessible opportunities for job
training and employment. As an adult-serving state, Iowa implements services that are
supportive of Career Pathways training and TTW. Iowa’s Service Delivery Strategies (SDS)
includes IRTs, Asset Development and Blending/Braiding of Resources. IRTs are used to
convene DRCs, Employment Specialists and WIOA partners to identify and align services and
supports for JSWDs and leverage public and private resources. Blending/braiding of resources is
another essential component of this process that includes leveraging resources for Benefits
Planning, Partnerships and Collaborations with WIOA mandated partners, and training on
Financial Literacy and Asset Development strategies. These SDS are integrated and aligned with
multiple disability and employment grants and their funding streams to broaden the impact and
reach of DEI and WIOA services. Iowa also has created a region-wide team whose tasks is to
discuss business service issues from job training to job placement, soft skills, employment
opportunities and enrollment in TTW.
Most Iowa WDAs provide pre-employment classes at Correctional facilities that include
mock interviews, connections with employers for enrollment in work release programs, access to
mini-job fairs including reverse job fairs in which job seekers provide information to prospective
employers. Having access to a DRC helps employers and HR directors develop training
programs on disability etiquette, soft-skills and building relationships with Correctional facility
staff that eventually engage with the WDA to provide employment and related services.
Initially, there was some resistance to the idea of a Correctional facility work release
program through the DEI. However, WDA and correctional facility leaders became more open to
the idea “and walls kind of came-down.” The program was inspired by a partnership of both
agencies, despite initial push-back. More recently, the program expanded to include integrated
industry teams with community support services divided up into sectors to promote the
employment of JSWDs, connect them with local employers and determine their needs with
regard to employee skills and interests. Along with the WDAs, “we kind of jump started what is
now established through the Workforce Development Board by partnering with our WIOA
partners, Correctional Facilities and community-based agencies.”
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
131
Iowa also helps job seekers develop Reverse Career Fairs through its Business
Engagement staff. Reverse Career Fairs allow job seekers to develop informational documents to
share with employers as they peruse fair booths to learn about each job candidate, their
occupational interests and employment background. Job seekers also have opportunities to attend
business tours and discussions about the hiring process, accommodations, work environment and
wages. This information is collected and used in job search classes for individuals that come to
an AJC interested in employment.
“In our office, we have a great staff that is willing to support our JSWDs as people just walk
in and ask for information from the administrative staff. They then make an appointment to
meet with us. A number of people come in right off the street and we’re very well-versed in
how to begin the process that eventually leads to employment.”
“You know, the thing about our region is that we have a large area, and so because of that,
with the barriers that some of our clients have, we have to be out of the office quite a bit. I
think that makes the staff equipped, trained and ready to help people who walk in the door,
because we’re not always going to be here.”
iii. Proposed Outcomes:
● Partnerships and collaborations include adjudicated individuals, blind individuals,
individuals with psychiatric diagnoses and former sex offenders.
● Increased awareness and involvement in supporting JSWDs toward independent
futures;
● Achieve the following individual outcomes: access to benefits counseling, job
coaching, support services (e.g. transportation, child care, housing etc.), access to
community-based agencies, soft-skills development;
● Reverse employment fairs in which individuals provide information to employers
regarding their skills, interests and work experience.
● DRCs function as systems change agents, case managers, facilitators and mentors;
● A collaboratively engaged approach to workforce development that includes a
collaboration of DRCs and Employment Specialists, numerous partners from
WIOA mandated partners to links to programs for incarcerated individuals with
disabilities, sex offenders and individuals with psychiatric diagnoses.
iv. Partnerships and Collaborations:
DRCs receive and distribute information for job seekers through the AJCs. Individuals
may have questions about their unemployment support or budgeting that determine the kinds of
services and supports that are available to them. By collaboratively engaging with staff
throughout the WDA, including DRCs and Employment Specialists, Iowa has access to
numerous partners that “come to the table at the same time to figure out who can best meet the
needs of our JSWDs and how we can bring resources from each partner to the table to add
value. That’s been a huge benefit.” Iowa accesses WIOA and related services through various
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
132
partners. This process is facilitated through a referral system that is accessible without visiting an
AJC, and as a result, expedites the receipt of services for individuals that have mobility issues,
limited transportation and/or other barriers to employment and training.
Iowa also operates a region-wide DAC that includes multiple partners. A standing
committee was formed in 2017 to focus on macro-level issues such as job availability, in-
demand jobs, Career Pathways training and support services provided by community-based
agencies. A subcommittee of the DAC also provides outreach to partners and individual job
seekers when they complete an application that enrolls them in the WDA and ensures that the
AJC has up to date information on their employment status. Business services teams, which are
tasked with outreach to local employers, use Title I adult/youth, dislocated worker, Wagner-
Peyser, VR services, Adult Basic Education and Department for the Blind resources. These
services include accommodations such as accessible work-spaces and software.
Iowa also identifies JSWDs through entities that serve individuals with disabilities from
residential to day treatment programs that provide support to sex offenders, SSA beneficiaries,
TANF recipients and other individuals that have barriers to employment and training. Business
services teams focus on employee skills, training opportunities and managing or retaining
employees with disabilities that may benefit from customized employment strategies such as job
sharing and task reassignment. Through a customized employment process, job seekers, in
collaboration with an employer, establish employment conditions that are necessary for their
success.
DRCs acknowledged that there are numerous opportunities for collaborations through
both WIOA mandated and non-managed partners. DRCs often facilitate new partnerships and
collaborations by “First, recommending a certain need for a given customer. Who can provide
or meet this job seekers’ need at this time?” Partners then have an opportunity to discuss what a
tailored plan for a given job seeker might look like. In some instances, a job seeker may not be
eligible for one program, but would be eligible for another program; “We look at the big picture
needs of each job seeker and figure out with all the entities involved who can contribute funds to
the services this job seeker needs.” Co-enrollment is necessary in many situations because job
seekers are often in more than one program and may require multiple services. It also allows
DRCs to work bring all partners to the table. DRCs commented that “it’s ultimately the key to
any job seeker’s success.”
Access to WIOA has effectively expanded the availability of services. DRCs emphasize
planning services locally through the WDA and bringing WIOA mandated partners into the
discussion as they arrange resources and supports for each JSWD. According to several DRCs,
initially, with the Disability Program Navigator it seemed to be a separate initiative that
functioned autonomously. DEI and WIOA however, are now very integrated into the package of
available services. “That’s definitely the biggest change we’ve encountered.”
Iowa however, does not always rely on its DRCs as the subject matter experts. When
Employment Specialists or other staff needs information on how to assist an individual, “that’s
when we go to the DRC. It’s not at all immediately.” They place many individuals with
disabilities into employment without ever getting to a DRC. This process expedites access to
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
133
employment and subject matter experts such as DRCs can be part of the process, but only when
they are needed for discussions about specific accommodations and/or suggestions for training or
enrollment in TTW. Etiquette is another issue that is reviewed with staff on a continuous basis to
make the AJC environment “comfortable and inviting to all JSWDs.”
v. Employer Partnerships:
Iowa partners with local colleges and community-based agencies. There’s an extensive
list of about 30 individuals along with a county coordinating agency for service providers (CPS)
that addresses issues from DEI grant expenditures to flexible spending accounts, TTW and the
specific needs of a given WDA. Iowa has also created outreach materials such as brochures that
advertise employer events and meetings that resonate with the business community with the goal
of creating more collaborative relationships with the DEI and AJCs. Iowa is also supportive of
the business community and when any member of the DEI or AJC community goes out and
speaks with employers, they present information on behalf of the entire workforce development
system to increase the number of partners involved in the development and implementation of
services; “that’s an ever-growing aspect of how we do business services in the region.”
However, partnerships with the various WIOA partners, has been mixed. According to
some DRCs, core partners understand that they are expected to collaborate with DEI and WIOA.
However, there are some partners who feel as if their services are too multi-faceted that
providing information on all of the available WIOA services would be too much information for
employers. Iowa’s WIOA referral system coordinates each individual’s specific needs.
Sometimes it’s a meeting with one job seeker or multiple job seekers that engage with local
employers at the AJC to identify employment opportunities that fit their skills and interests.
Employers don’t want to be “bombarded with 20 people at one time asking for
information about employment opportunities and hiring issues.” In most cases, employers want
to know that they can work collaboratively with each AJC and WIOA partner to ensure that job
seekers are placed in an appropriate training and employment situation. Iowa is focused on
getting people into training and employment that can build their repertoire of skills and services
in a way that meets the needs of local employers. Iowa is focused on Career Pathways training
which includes partnerships with WIOA service providers and businesses that lead to the
development of skill-based programs that train job seekers to do a certain job and develop soft
skills that help them acclimate to their new employment situation.
The implementation of Career Pathways training varies for most job seekers. Initially,
there tends to be barriers to getting a job seeker to engage in the employment process, including
the selection of an appropriate training program. Iowa facilitates this process by involving
additional WIOA and non-WIOA partners in an IRT to obtain basic services and resources such
as transportation, childcare and access to community-based agencies. Needless to say, Iowa
DRCs and job seekers meet barriers during this process. Disability is always a factor as DRCs
often discuss physical and/or psychiatric challenges to employment and/or career futures.
Equipped with more resources through an IRT, partners tend to get more involved in the process
of identifying and providing resources; by putting more heads together, we can do a better job of
supporting JSWDs.”
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
134
“We’re going to have to stay more general, define each employer’s needs and then bring in
the partners that can provide more specific information. We need to max-out all the training
opportunities we have. But one thing that is often missing is Work-Based learning
opportunities, like on the job training and apprenticeships. That needs to be added.”
With regard to Career Pathways training, most DRCs agreed it is necessary to have
access to “on and off ramps” to ensure that each job seeker has “temporary pathways” that
allows them to have entry points where they can begin the training process and take a respite
from it. Initially, Iowa was focused on two year programs but recognized the need for short-term
training opportunities, work-based learning and on-ramps and off-ramps that allow job seekers to
meet both their personal and career goals.
“I would just add also that there’s a lot of hand-holding. Job seekers can potentially need a
basic level of support as far as confidence to get started. I think there should be a good one on
one talk with each job seeker. It’s face to face at times and there’s definitely hand-holding
involved in helping them along the way.”
“We gained a lot of knowledge from NDI technical assistance. My only feedback is that
sometimes the training is not as localized as we would have liked. Some of the materials could
have been maybe more geared towards Iowa. Otherwise, they were very helpful. That would
be my feedback of the technical assistance.”
vi. DEI Services:
Case management is a primary component of DEI. It appears to be a more important
component of the employment process then a job placement or enrollment in a Career Pathways.
Case management with a ticket holder includes a considerable amount of support before focusing
on work incentives and employment. DRCs often use a modified Discovery process which is an
evidence-based alternative to standardized assessments and evaluations and is closely aligned
with person centered planning that involves getting to know an individual before supporting
them in developing a plan for employment.
Initially, Iowa used automatic ticket assignment to identify TTW beneficiaries. However, this
approach did not take into consideration that not all beneficiaries are eager to transition to full-
time. Currently, Iowa is focused on working with individuals “who want to better their situation
and/or seek additional or new employment opportunities.” DRCs engage individuals in the
process of Discovery which begins with case management and WIOA services that may be
needed by newly enrolled job seekers. For example, Benefits Planning and IRTs are central to
the process of employment as some TTW beneficiaries and JSWDs have not worked in many
years and may need basic support services as well as an understanding of how their benefits will
be affected if they return to work. Beneficiaries that are interested in TTW also have access to
presentations and one-on-one discussions with DRCs and/or Employment Specialists who can
explain the benefits of the program and discuss specific SSA guidelines. For Iowa, success with
TTW includes strategic support services designed to help one individual at a time access DEI
services and service delivery strategies, instead of many individuals engaged in a group setting.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
135
Iowa DEI service delivery strategies emphasize IRTs and Asset Development Services.
The latter were combined to include services from DRCs, Human Immune Deficiency (HIV)
service providers and the National Working Positive Coalition to engage individuals in Career
Pathways training. Some DRCs also focus on systems change designed to improve AJC
accessibility, upgrade the skills of Employment Specialists, and provide case management
services, benefits planning, job training and work incentives for SSA beneficiaries. DRCs also
train Employment Specialists to provide better access to WIOA services following a protocol
that collects information on each participant’s employment and related skills and career plans.
Some JSWDs also engage in a detailed career planning process to identify their key interests,
create a career portfolio, enroll in literacy skills assessments and expose individuals with limited
employment histories to a wide-range of training opportunities leading to employment.
Blending and braiding of resources is an opportunity for JSWDs and their DRCs “to be at the
table” to review each individual’s options from DEI services to VR, Title I Adult Services,
Dislocated Worker services “or whatever it might be, we all have rules and regulations that
govern our programs. We also have opportunities with all of our partners where multiple
partners can find a particular service for an individual.”
“We actually hold their hand. We’re involved, intensively as case managers and maybe that
one other person that could be way more beneficial not only to them, but to us. She could be a
family member, maybe a DRC or an Employment Specialist. She could be an individual that
has successfully enrolled in TTW. As an EN, in the future, we’ve learned that it’s way more
important to be able to help a TTW beneficiary. It’s not about quantity. It’s about quality, I
guess.”
In Iowa, TTW beneficiaries sign an employment plan with the WDA that describes a path
that includes individuals that are “on the same route to employment”. Using the automatic ticket
assignment process was less effective than using intensive case management in which DRCs and
Employment Specialists identify individuals that appear to be ready to work and reduce or
discontinue their SSA benefits. Partnership Plus is a vehicle for accessing ticket beneficiaries
during the 90 day waiting period. At the end of the 90-day waiting period, tickets can be
transferred to another EN, such as a DEI EN, to access milestone and outcome payments and
receive job coaching and/or support services.
“When you’re working in TTW partnership and finding is involved, you really need a very,
very strong partnership to make that work smoothly, without any hiccups.”
At one point, we had about 265 people that were ticket eligible. Of that, we had about 70 that
were heavily engaged and interested in TTW. The rest were individuals that didn’t remain in
contact. We didn’t know their current status and couldn’t get them to respond to us so we were
spinning our wheels constantly on those individuals who weren’t actively engaged enough to
work with us.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
136
Active Resource Coordination brings together resources that are available throughout the
WDA for individual clients. IRTs access resources through WIOA mandated and non-mandated
partners and DRCs and their clients. Often, job seekers have numerous partners from VR to
mental health services to TTW and other local support services. They may also receive support
through local colleges that have Disability Program Navigators that provide oversight and
support universal access to resources and provide training on disability topics. “Iowa
employment initiative group” merged with its local leadership team comprised of several
counselors. As part of the big picture group, customized employment is an integral part of what
we do.”
vii. Capacity-Building: DRCs and WIOA Services:
DRCs are encouraged to inquire about family members and friends to establish trust and
a willingness to communicate with DRCs and Employment Specialists through Discovery. This
process often includes pieces of customized employment (CE) arranged through a team that
meets on a quarterly basis to strategize on the implementation of service delivery strategies. Self-
employment was implemented at local colleges as well, and local small business development
centers collaborated with Title I services to provide entrepreneurial skills that support the
development of new businesses. Iowa’s Asset Development services cover various approaches to
saving or investing in resources, purchasing a home or car, or developing new employment
related skills. While these activities are not necessarily primary components of the Iowa DEI,
they have been successfully adopted by job seekers interested in accruing resources and skills
and enrolling in education and Career Pathways training. Benefits counseling is a key component
of employment for SSA beneficiaries. It is provided through certified CWICs that provide
information to SSA beneficiaries and non-beneficiaries interested in developing resources
through Asset Development and access to support services such as transportation, housing and
access to community-based agencies.
Iowa also created integrated industry teams that include five sectors: manufacturing,
retail, hospitality, healthcare and construction trades. Job seekers have access to business tours
during which they ask questions about hiring, training, accommodations, and the work
environment. DRCs have done presentations on Adult Education Literacy and personality
assessments and have accessed a program called New Beginnings, which provides counseling for
individuals interested in acclimating to the work environment. Iowa also provided presentations
at National Youth Symposia that covered Youth Sexual Offender Treatment Programs and a pre-
employment training that includes business networking, business relationships and skills for
connecting with other job seekers and employers. Iowa reportedly has a high sex offender
population and as a result, difficulty finding appropriate housing and employment for these
individuals is an impediment. One potential setback is job availability in Iowa’s rural areas
which continues to be a barrier for all JSWDs.
viii. Promising Practices
Iowa has a partnership with VR. The two agencies have developed working relationships
in which they share information, resources and expertise so that there are few gaps in services
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
137
and that job seekers have access to employment and training services that will best meet their
needs. Customized employment is a promising practice that includes a specific set of strategies
that incorporates individualized planning and negotiated employment. Discovery is a component
of Customized Employment. It is a strength-based approach that collects information about an
individual’s interests, skills and employment goals. It is designed to help job seekers develop
opportunities for employment that meet their financial and social needs. Job carving, self-
employment, job restructuring, and negotiated job descriptions are all components of
Customized Employment that are used by DRCs to increase the likelihood of long-term
employment. Iowa JSWDs have been self-employed, employed as office assistants, support for
the culinary/hospitality industry and assistants in variety stores, pharmacies and school systems.
ix. Sustainability
Iowa leadership has focused on sustainability for the future of the DEI by integrating DEI
services and strategies into the AJC. Given the focus towards work-based learning, Career
Pathways and case management, most job seekers eventually find employment that’s sustainable.
Iowa also sustained some of its DRCs through TTW resources which have been focus of the
state’s TTW activities; “building sustainability once the grant’s gone and to be able to keep the
DRCs region-wide is a key goal.”
Most Iowa TTW revenue is used to sustain the DRCs through TTW revenue. Iowa had
three DRCs which was reportedly a very important resource for participating WDAs. If Career
Pathways training becomes unavailable due to limited resources, opportunities abound through
on the job training and/or employer-partnerships that provide direct employment or
apprenticeship opportunities. Iowa has also prepared an updated regional strategic plan that
incorporates business services, career and skill development systems, and collaborations across
WDAs. Another focus is experiential work-based learning, which is a process through which
individual job seekers explore and observe how employees engage in certain occupations and
how they function throughout a typical work day. Iowa is expected to sustain assistive
technology, work experiences and on-the-job training for individuals “now that they have a core
vision of how to easily talk with AJC staff.”
IRTs continue to be frequently used to access resources for job seekers enrolling in DEI. The
majority of job seekers are VR clients and/or receiving services from community-based agencies.
There is also a sizeable population of Veterans with disabilities that benefit from the use of an
IRTs to access WIOA mandated partners and community-based agencies.
Iowa also collaborates with the public transportation system in Iowa which has been a
barrier to job seekers returning to work. Because the transportation system has limited
availability and limited accessibility and limited access on weekends, Iowa DRCs met with
transportation management and allowed them to provide training opportunities for job seekers
who need help with the transportation system to travel from home to work. Training on the Iowa
transportation system was provided by DRCs. It includes information on fare changes, routes
specific to the needs of DEI job seekers, and best practices for accessing local bus and train
routes.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
138
E. State: New York; Focus Area: Adults with Disabilities
i. Stated Goals and Objectives
New York was the recipient of DEI Rounds 1 and 4 grants, with each grant focusing on
asset development, IRTs, and blending and braiding resources. New York’s Round 6 “Pathways
to Employment” grant sought to build upon the successes of its prior Rounds and its existing
Career Pathways (CP) programs for low-income adults by adapting these programs in high-
growth regions that include New York’s two participating workforce development areas
(WDAs): Herkimer/Madison/Oneida (HMO) and the Capital Region. Integrated services—
including IRTs and individual learning plans (ILPs)—were to assist JSWDs to obtain CP- and
industry-focused employment in the growth sectors of health care, technology and
manufacturing, and to enhance basic occupational skills.
In addition, the NY Round 6 grant intended to strengthen and sustain coordination across
systems by embedding Universal Design (e.g., alternative assessments), accommodations,
assistive technologies (AT), support services, and productive partnerships within workforce
development and CP systems. New York proposed to serve 300 JSWDs over the life of the
project, improving the job placement outcomes for 185 of these individuals who complete their
CP in combination with stackable credential-based education offered through the community
college system. Job seekers who are Veterans with disabilities were to receive service priority.
Engaging employers was also a priority and the workforce needs of businesses were incorporated
via labor market analyses. New York also noted in their application that its Round 6 grant was
strengthened by investment in a robust, cross-system data management platform— NY State
Employment Services System (NYESS))—allowing tracking of employment outcomes of
jobseekers served through the workforce development system.
Fundamentally, NY proposed addressing systems change for JSWDs by strengthening
cross-agency partnerships that improve CP outcomes for JSWDs. Participating AJCs and their
DRCs would partner with existing CP partners including TANF Employment Service providers,
school districts, community colleges, and Adult Basic Education, and also recruit new partners,
such as vocational rehabilitation counselors, supported employment providers, and certified work
incentive coordinators (CWICs). Employers were closely involved in sector-based strategies by
the DEI’s implementation of employer engagement strategies. DEI practices were intended to be
sustained through institutionalization of IRTs and disability service provision awareness within
AJCs; bolstering of partnerships; dissemination of best practices statewide through media,
conferences, and knowledge-sharing products (e.g., about developing Memoranda of
Understanding and staff development plans); integration of WIOA, Wagner Peyser, and TTW;
and CP-focused state and other funding.
48
Consistent with ETA’s guidance, NY Round 6’s “Pathways” project proposed to achieve six
major goals to create improved systems serving JSWDs:
48
NY DEI Round 6 proposal
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
139
• Build cross-agency partnerships and clarify roles by “embedding essential leadership,
expertise, resources, and supports from the WIOA, Wagner Peyser and Title
IV/Vocational Rehabilitation agencies” at both the state and local levels;
• Identify sector or industry and engage employers by “expanding strategies for
engaging businesses around hiring disparities in the disability population and developing
collaborative plans driven by business needs”;
• Design education and training programs with emphasis on “embedding Universal
Design within workforce development and career pathways systems,” and “developing
coordinated, flexible systems accessible to the greatest number of users”;
• Identify funding needs and sources by blending and braiding funds at the state and
local level, including “identifying and leveraging the funding sources that come with
each individual participant (including TTW) to both stretch grant funds and to leverage
formula funds through co-enrollment strategies that build system capacity to continue
career pathway work”;
• Align policies and programs by integrating, “developing, and deploying financial
resources augmented by improved leadership, goal-setting, and shared accountability to
support the state’s education and training systems that measurably increase inclusive
career pathways program development, delivery, and student completion,” as well as
through a large-scale outreach and “messaging campaign” that harnesses social, digital,
and traditional media to nurture “disability diversity confidence”; and
• Measure systems change and performance by cultivating cross-systems “data
pipelines” that facilitate “quality data collection and analysis, resulting in both qualitative
and quantitative reports on systems change.”
49
DEI state agency partners include Adult Career and Continuing Education Services-
Vocational Rehabilitation (ACCES-VR) and its advisory council, the Developmental Disabilities
Planning Council, the Office for Mental Health, the Mental Health Association, the Olmstead
Cabinet, New York State Commission for the Blind (NYSCB), and the WIOA Accessibility
Workgroup. DEI local agency partners include WIOA partners, AJC Business Services, WIBs,
community colleges, chambers of commerce, training providers, hospitals and long-term care
agencies, community-based disability-services organizations, Departments of Social Services
(DSS), advocacy groups, and economic development agencies.
ii. Proposed Service Delivery Strategies
At the services level, the NY Round 6 DEI intended to:
• Increase JSWD access to WIOA core services (including employee recruitments
and industry-connected job fairs) and support services, such as post-placement
counseling, benefits counseling, job coaching, student services (e.g., financial aid,
tutoring, and life skills), and transportation, childcare, and housing assistance
• Guarantee that AJCs are accessible in full compliance with the Americans with
Disabilities Act (ADA), with the most up-to-date AT and availability of
individual rooms for privacy when conducting assessments
49
NY DEI Round 6 Proposal pg. 6
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
140
• Furnish AJC and partner staff training on strategies for individuals with learning,
cognitive and physical disabilities
• Based on an individualized “eco-map,” DRCs and IRTs would implement
accessible curricula and an ILP that form the basis for an Individual, strengths-
based Work Plan, and addresses key retention issues (e.g., emergency planning,
building a support network, budgeting, and coping)
• Ensure that DRCs function as case managers, facilitators, mentors, and service
navigators for each participant; coordinate with appropriate agencies, services,
and referrals based on JSWD barriers (including via Partnership Plus); and train
AJC staff to assist JSWDs.
• Assign community college students to suitable Learning Community Clusters,
which involve contextualized learning
• Use braided resources for individual job education, training and employment
support by supplementing training costs payable through DEI and TTW, as well
as WIOA Title I funds
50
Ultimately, systems and services level improvements and coordination were designed so
that JSWDs could optimize training and skills enhancement to earn stackable credentials that
enable them to advance along multiple career ladders (e.g., from Certified Nurses’
Aide/Assistant to Licensed Practical Nurse), earn living wages, and achieve economic
independence.
To achieve their systems and individual-level outcomes, DRCs in each WDA were to
function as both systems change facilitators and case managers, including through leveraging
DEI and/or Individual Training Accounts and VR resources. New York also has a robust TTW
program, where the state as the lead entity retains twenty percent of TTW revenue for
management, with the remainder reallocated to WDAs for service enhancements and program
sustainability. Blending and braiding of resources was another essential component, which
included leveraging resources for benefits planning, partnerships and collaboration with WIOA-
mandated partners, and training on financial literacy and asset development. These processes
were integrated and aligned with multiple disability and employment grants and their funding
streams to broaden the impact and reach of DEI and WIOA services. The below participant
testimony illustrates the life-changing impact of such synergy:
I don’t know where I’d be without [the DRC and DEI]. My belongings were in the driveway. I
went into the AJC and asked for help. They got me a place to stay. I was willing to sleep in my
car at the AJC. And they were going to help me because I wanted their help. Anything I needed
they were there for me.
iii.
Proposed Outcomes
50
NY State Employment Services System (NYESS) administers TTW for NYS: https://nyess.ny.gov › docs
› nyessbrochure .
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
141
New York’s Round 6 targeted individual outcome goals were specified in their proposal
as below. According to a progress report for the quarter ending December 31, 2018, the NY
Round 6 DEI attained the following individual outcomes relative to their stated performance
goals
iv.
Implementation Summary
v. Capacity-Building: The Context for Capacity Building
New York has considerable experience from implementing DEI Round 1 and Round 4,
which provided a foundation for Round 6. The state was also a pioneer of TTW implementation
prior to the implementation of DEI; New York was certified by SSA as a state lead entity for
Employment Network creation during the early years of TTW. In addition, the state boasted one
of the largest cadres of Disability Program Navigators among all states participating in that
federally-sponsored initiative. As a result, New York became a mentor to several DEI grantees,
providing information on EN applications, benefits planning, CWIC credentialing, and milestone
and outcome payments.
vi. Capacity-Building: DRCs and WIOA Services
To enhance capacity during Round 6, New York established a Project Council that
included Workforce Development Board directors, AJC Managers, and representatives from
NYSCB, ACCES-VR, and the Independent Living Centers (ILCs). In terms of DEI-sponsored
trainings, DRCs arranged tutelage for staff and partners about universal services, ADA,
reasonable accommodations, disability etiquette, assistive technology (AT), and TTW. All DRCs
were also trained in benefits counseling.
Instead of sending all JSWDs to DRCs, New York created a universal access platform,
including larger chairs and screens, specialized keyboards, accessible software, screen readers,
and adjustable work stations. One of the participating AJCs saw customers—who were also
clients with the State Commission for the Blind (NYSCB)—visit specifically to utilize its video
magnifier and closed-circuit television system. Statewide AT updates that occurred during
Round 6 were beneficial. NYSCB in particular regarded the upgrades as “really helpful,”
including by making AJCs “more welcoming” and facilitating job searches. A DRC was also
tapped by the Employee Assistance Program to help employees with disabilities at Albany
Medical Center with accommodations and community resources. The Technology Related
Assistance for Individuals with Disabilities (TRAID) Program was also utilized to help JSWDs.
Some DRCs also focused on systems change designed to improve AJC accessibility.
DRCs trained AJC employment specialists to provide better access to WIOA services, following
a protocol that collected information on each participant’s employment and related skills and
career plans. DRCs also upgraded the skills of employment specialists, provided case
management services, and expedited access to benefits planning counseling, job training and
work incentives support for SSA beneficiaries. Some JSWDs also engaged in career planning to
identify participants’ interests, create a career portfolio, enroll in literacy skills assessments, and
expose individuals with limited employment histories to a wide range of training opportunities
leading to employment.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
142
AJC staff and partners were also trained on engaging JSWDs living with HIV/AIDS (e.g.,
disclosure and confidentiality), and on data entry. Moreover, there was ample cross-training with
ACCES-VR, NYSCB, ILCs and organizations offering support for JSWDs with HIV/AIDS. As
well, ACCES-VR offers weekly service orientations on disability employment topics at AJCs.
In a final call, one DRC mentioned how he was overextended during the grant,
suggesting that New York could build more capacity to serve JSWDs in treatment regions.
vii. DEI Services
Throughout the grant period, DRCs worked with AJC staff and partner agencies to
identify, recruit, and enroll JSWDs into CP programs and TTW, including through referrals from
ACCES-VR, local service agencies, colleges, and schools. DRCs developed IRTs and engaged in
blending and braiding of resources—often with ACCES-VR—to provide CP training and access
to WIOA services for SSA beneficiaries and other individuals with disabilities. DRCs were able
to leverage many community resources, cultivated positive relationships with partners, and took
advantage of DEI’s flexible structure to generate innovative strategies.
On a regular basis, DRCs communicated directly with JSWDs by telephone or in person
to discuss the benefits of TTW and CP programs. DRCs also conducted assessments of aptitude
and/or literacy and numeracy, reviewed employment histories, and accessed benefits planning
services. AJCs also provided bus passes to JSWDs with transportation barriers, and helped some
clients obtain driver’s licenses. In concert, all of these elements enabled DRCs to successfully
serve clients with significant barriers (e.g., mental health issues and criminal justice
involvement) and very little paid work experience.
Mandated partners accessed WIOA services and partner programs such as Career and
Technical Education services through local community colleges, and also worked with the
Alpine Rehabilitation and Nursing Center in Herkimer County and an AJC to establish a
Certified Nursing Assistant (CNA) training at Alpine’s facility. TANF—a required WIOA
partner—provided employment supports for individuals returning to work.
Local areas had considerable flexibility in how they utilized their training and related
resources. In some cases, JSWDs pursued available job opportunities that required limited
training and offered minimum wages. Others opted for more robust training and services,
including support in employment preparation and acquisition through IRTs, and in leveraging
supportive resources such as CA$H (Creating Assets, Savings and Hope), an asset development
program that helps individuals with financial literacy, tax preparation, and asset-building.
New York also facilitated systems change by cultivating a level of comfort among
partner agencies in serving people with significant barriers, including obstacles related to
language, poverty and/or disability. DRCs were tasked with addressing systems change and case
management support for individuals with visual, psychiatric, and physical disabilities. New York
Round 6 was also unique in prioritizing JSWDs living with HIV/AIDS, and was able to
successfully place multiple JSWDs with significant barriers (e.g., substance abuse and criminal
justice involvement).
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
143
Other service delivery strategies included asset development, ILPs, work-based learning
opportunities and apprenticeships. Asset development included utilization of a “Hand on
Banking” tool, support from Volunteer Income Tax Assistance and MyFreeTaxes, and
workshops on TTW. Apprenticeships involved placement assistance from DRCs and support
from HIV service providers. Further, the National Working Positive Coalition was pivotal in
engaging individuals in CP training. Regarding customized employment (CE), the state lead
emphasized the role of business services in conducting customized recruitment and matching
JSWDs’ needs with openings. Otherwise, there was not significant activity with CE, nor was
there with self-employment.While there is evidence DRCs are helping to build better linkages
through systems, the balance between DRCs’ attention to systems and direct services is more
heavily weighted towards directly assisting program participants. Based on conversations with
the DRCs, AJC staff, and partners, it was apparent that the DRC-led systems change activities
related primarily to partnership building and to training staff and coordinating cross-trainings.
DRCs offered that it will be challenging for AJC employment specialists to sustain the level of
individual attention that they provide to DEI participants. However, it is apparent that DRCs are
modeling what it takes to serve JSWDs successfully, and participants are demonstrating that with
proper supports and the use of DEI practices they can succeed in career education, training, and
employment.
With respect to TTW, treatment regions are under contract with the statewide
administrative EN. Two DRCs had limited experience with TTW, while others provided CWIC
services at ACCES-VR. DRCs were instructed to focus on “quality and not quantity” for Ticket
assignments. One DRC also attended monthly meetings about Partnership Plus. According to
December 2018 NYESS reports, treatment regions each generated more than $177,000 in TTW
revenue.
We did not find much evidence that a systemic approach to integrating Universal Design
(UD) was being implemented as the grantee proposed. One example of New York’s intent to test
and demonstrate UD principles was development of a customized AJC workshop for a blind
customer at an AJC. However, despite New York’s intentions to integrate more UD in each pilot
site—such as utilizing SMART Boards to enable multi-sensory workshops—the state was
hindered by DOL/ETA and DEI regulations (i.e., prior approval was needed to make AT
purchases over $5,000, and equipment purchases could not be made in the last year of DEI).
Consequently, we find that systemic training on UD principles and targeted application of its
practices were not implemented during the grant period.
viii. Partnerships and Collaborations
As described in a quarterly report, New York’s core mission of the Round 6 DEI project
was to implement systems change through building cross-agency partnerships, clarifying roles,
and engaging employers. New York made significant efforts to engage partners serving JSWDs.
The grantee bolstered its collaborative arrangement with ACCES-VR. Through a
partnership with ACCES-VR and DEI WDAs that began in Round 1, State Leads and DRCs
developed working relationships with SSA personnel. By the end of Round 6, ACCES-VR
counselors were present within treatment AJCs at least twice monthly.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
144
Partnership development also focused on engaging NYSCB to better serve its customers
through DEI; on collaborating with the Office of Temporary Disability Assistance to serve
TANF recipients who meet DEI criteria; and on serving customers of ACCES-VR and ILCs. In
concert with these partners, DRCs developed jobseeker groups that led to benefits planning
sessions and monthly meetings. These events helped “people with visual impairments feel
comfortable to come [to AJCs]…meet the staff, and eventually get them registered” for services.
Individuals with visual impairments described AJC managers as “very supportive.” However,
one DRC suggested in a final call that his connection with NYSCB could be stronger; he
indicated the same about the local ILC.
Nonetheless, the state lead asserted that enhanced collaboration with ACCES-VR and
NYSCB resulted in much improved experiences and opportunities for JSWDs:
[AJCs] knew about ACCES-VR. They knew about NYSCB. But now [those agencies]
are actually coming to the AJCs and we are working together for these participants.
So [JSWDs] are feeling more comfortable and welcomed. And we’re getting them jobs
quicker, with more opportunities.
The Northeast Association for the Blind was also a crucial partner, helping to improve access to
DEI services for JSWDs who are blind or visually impaired.
In addition, The DEI State Lead and DRCs worked with the Department of Health and
the AIDS Institute to provide a train-the-trainer opportunity designed to serve people with
HIV/AIDS. The training involved material intended to allay the concerns that people living with
HIV may have when considering employment. The training also offered guidance on providing
trauma-informed services for people living with HIV. Other training topics included health
maintenance services, self-disclosure and confidentiality issues, identification of gaps in work
histories, concerns about discrimination by employers, and access to health care coverage and
other benefits.
DRCs from one region connected with the Damien Center, a local agency that supports
those living with AIDS and HIV through counseling, food preparation, and access to medication.
One participant commented:
“It’s great! The Damien Center, a local agency, is a partner and they’re not WIOA-
mandated.”
DRCs also sought partnerships after finding that AJC staff were less well-informed about
recognizing and accommodating hidden disabilities within their Centers. The DRC located at the
Herkimer/Madison/Oneida AJC co-facilitated a training on trauma-informed care with the local
area Psychiatric Center to assist employment specialists to better understand the impact of those
disabilities on employment success.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
145
The Legal Aid Society (LAS) also became a DEI partner. A LAS liaison helped JSWDs
fill out Supplemental Nutrition Assistance Program (SNAP) applications, building off a TANF-
DEI collaboration. As well, a LAS lawyer attended DEI meetings to help individuals with
HIV/AIDS with SSA periodic medical review requirements.
Referrals from Northeast Career Planning for the Disabled, Inc. also helped to place
several JSWDs into employment. A CNA program partnered with healthcare facilities through a
“Workforce Collaboration” facilitated by the Director of Training and Programs at Empowered
Pathways to help develop career pathways. This effort also included collaboration between
Oneida County Workforce programs and Oneida County Department of Social Services to help
local agencies serve JSWDs in the community. In addition, one treatment site partnered with the
Veterans Administration by inviting veterans’ employment specialists to tour an AJC, and
suggesting collaboration on TTW and AT.
A DRC elaborated on the fruits of DEI training facilitated by a partnership between an
AJC and a local community college:
We have a lot of JSWDs in training. We have 9 who have completed or nearly completed job
training. There’s a program between Onondaga Community College and the center. The AJC did
CNC training for 10 people for free and also got employers involved in the process. The
customers chose the employer and the latter decided what they wanted in an employee. Now
they’re doing apprenticeships. One in particular has been a huge success. We have Healthcare
classroom training. Manufacturing is an employer here.
There are also more practical needs that are fulfilled by partners—such as the Department
of Social Services—including clothing, furniture, transportation, housing, and childcare. WDA
staff also attended business council meetings, in which business services representatives
discussed opportunities for training and employment and events like employment fairs and asset
development summits.
DRCs reflected during their interviews upon how Round 6 strengthened connections,
which led to better information-sharing with staff and customers, enhanced access to services
like benefits planning and community resources, and productive relationships with other
agencies, training providers, and employers. Likewise, ACCES-VR reported more exposure and
contacts within AJCs, facilitated by the presence of DRCs. In a post-grant call, the state lead
pinpointed improved communication and referral processes between AJCs and community
agencies as one of the grant’s defining achievements.
The development of partnerships is a strong feature of the NYS DEI across both
treatment sites. Linkages are primarily facilitated through efforts of DRCs, and most are informal
rather than formalized relationships. The next step of joint participation in resource allocation
(e.g., braiding funds) and developing memoranda of understanding to formalize roles and buy-in
has not yet taken place. While New York’s DEI partnerships achieved the purpose of
implementing DEI services and contributed to achievement of its benchmarks—and are
noteworthy by reaching out to agencies serving people with HIV/AIDS—the lack of formalized
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
146
cross-systems collaboration does raise questions about how well the arrangements will sustain
after the DEI ends.
ix. Customer Focus
We conducted a participant focus group at each of the two treatment sites. Individuals
who participated in these sessions had positive comments about their experiences in the program
and about using AJC services. Some participants referenced the help they were receiving in
accessing education and training leading to a career path. Most commented on how the DRCs
treated them with respect and followed up quickly and consistently with communication and
direct assistance. The below quote from a participant is representative:
I am very excited because things are working so well in my life. I am able to work around my
own schedule. As soon as I pass my RN, I can bring my Ticket anywhere. [In terms of] moral
support, the DRC is my biggest cheerleader. Her direction—with me coming into the office and
writing letters and proofreading—not that I need to be told what to say, but I needed that
guidance to become more professional. She helped me to coordinate with all these people
DRCs were a key resource for AJCs, working directly with customers with diverse
barriers to employment who accessed AJCs and disclosed a disability. One DRC described the
process of working directly with JSWDs as below:
Any person who comes in the center who discloses any disability will speak to me. I can help
those who are interested in healthcare with trying to get a job or get retrained for that field. I
also work with ex-offenders. With the Round 6 funding, I’ve worked more with ACCES-VR to
help customers get training and reach career goals. I also work with the local community
college. They refer students who have special needs or problems and other people who need
help.
DRCs offered more than systems and services knowledge; they were empathetic listeners,
allowing DEI participants to relate their own self-assessment of strengths and needs and their
own hopes and goals for future employment. One DRC testified, “It’s good to hear their story
too. You don’t just need to go straight down the questionnaire, sometimes you just let them talk
and tell their story.”
DRCs’ personal attention and customer focus were acknowledged and much appreciated
by DEI participants in focus groups. Among the endorsements they provided were the following:
There were hurdles to jump through and the DRC has been very knowledgeable and helpful. She
guided me through all the roadblocks I had, [and was] very knowledgeable and empathetic. She
goes another mile with me all the time. She makes it work.
DRCs also facilitated meetings with jobseekers to provide group benefits planning
sessions for individuals with visual impairments. This allowed individuals to meet DEI staff, and
to eventually enroll in DEI and WIOA services. Our data suggests NY Round 6 did well
advocating for and supporting AJC customers and other persons with disabilities who were
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
147
referred to the project. Data also indicated that AJC managers and staff valued the presence of
DRCs at their AJCs. One partner stated:
I think the fact that staff are aware of an individual who can provide more in-depth assistance to
these individuals is a big help—someone who can spend more time with these individuals
compared to what other staff can spend. It allows that person to meet with someone more in-
depth in discussing what their needs are and what services are available for them, compared to a
normally scheduled appointment.
x. Employer Partnerships
New York held business council meetings with WDA Boards to build familiarity with
and potentially sustain DEI practices. DRCs leaned on Business Service Representatives to
facilitate reaching out to local employers to discuss job and on-the-job training (OJT)
opportunities, client matches, and tax credits for hiring JSWDs. One treatment region sponsored
a reverse job fair, the first such event in the state. Twelve businesses attended, and three JSWDs
were engaged for interviews. Additionally, National Disability Employment Awareness Month
events proved to be fruitful for engaging employers and promoting the hiring of JSWDs.
At the systems level, robust efforts were made to engage businesses. DEI representatives
attended the National Apprenticeship Partner Roundtable in Rome, NY—organized by the
American Apprenticeship Initiative—which included 50 local business representatives and a
wide range of stakeholders to learn about such topics as the benefits of hosting or becoming an
apprenticeship partner, information on tax incentives and funding for OJT opportunities. The
Madison County Business Service Representatives provided information on hiring individuals
with disabilities and on identifying resources, funding, and support services for individuals with
disabilities and their prospective employers.
Although the NY Round 6 proposal indicated the state’s employer engagement would
result in participants acquiring jobs, evidence suggests the primary outcome from business
outreach is that familiarity and collaboration with the DEI is positively affecting some
employers’ perceptions about people with disabilities in the workplace. Multiple employers
talked about how they identified qualified job candidates by connecting with DRCs. One
employer looked within its business to determine how it could best retain talent among
employees with disabilities already on staff:
At the base of our decision-making to pursue this is we have quality employees with longevity in
the system now. Handicapped or not—and I don't mean to sound callous on that either—why
would we not invest in them, knowing that they are our future, because they're already invested
in us?
xi. Career Pathways
New York’s Career Pathways program is organized as a sequence of steps, each of which
is associated with a specific credential or job advancement opportunity. The program is designed
to enable participants to increase their job skills—including soft skills—and earn a credential
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
148
recognized by employers within targeted industries. Support also includes basic literacy and
math skills training linked to industry skill standards, and certifications and/or licensing
requirements that lead to in-demand jobs. Traditional classroom instruction is combined with
other activities, such as contextualized learning, work experiences and internships. Developing
partnerships with community colleges and employers is critical to achieving CP goals.
One CP college partner offered that the collaborative relationship between the college
and DEI was mutually beneficial, and another example of braiding resources for career-focused
education:
The students who come to me want to further their education but they don’t know how to do it. I
guided them to the DRCs who are great. They helped the students. One got a partial scholarship
through his work, but couldn’t afford the rest; the DRC was able to help him get money for the
rest.
In one treatment region, the DRC nurtured a strong relationship with the nursing
coordinator of a Board of Cooperative Educational Services to facilitate CNA training. This site
also partnered with a local rehabilitation and nursing center to establish a CNA training.
A DRC from the other treatment region cited DEI training funds as a huge linchpin for
successful healthcare pathway completion for JSWDs. This region also explored community
health worker internships for DEI enrollees. Another DRC was proud that DEI facilitated JSWDs
progressing from working in “Burger King” to working in hospitals.
In addition, New York developed a “soft skills job club” utilizing TTW revenue. New
York also was able to fund OJT opportunities at up to ninety percent through DEI, including in
manufacturing. All customers we spoke with very involved in their CP process. Though there
were a limited range of possible choices—due to the focus on healthcare and manufacturing—
the customers seemed all to be very happy with where they were, where they were going, and the
process in which they were supported to arrive there.
Altogether, data suggests the NY Round 6 DEI made strong inroads with cross-agency
CP partnerships. Consequently, DEI forged increased access for customers with disabilities to
credentialing programs that could qualify them for career-level employment within the health
care and manufacturing occupational sectors, and potentially other occupations where skills and
credentials may be transferable. Moreover, DEI’s progress in braiding VR, DEI, AJC, and
financial aid funding directed towards CP outcomes appears to suggest that a diversity of JSWDs
entering AJCs should have a greater likelihood of obtaining support for credentialed jobs with
career potential, rather than entry-level, low wage employment.
xii. Dissemination
Sharing of best practices occurred primarily via conference calls, passing along success
stories, and hearing from guest speakers (e.g., from ILCs, the Positive Working Coalition, the
Empire Justice Center, and the National Disability Institute [NDI]). Also, both DRCs and AJC
partners were proactive in their communication to external partners and potential referral
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
149
sources. Partners who became aware of the DEI were eager to refer customers to it. In part, this
may be due to the fact that DEI provided easily accessible resources for training and support.
The DEI also contributed to information-sharing on service to youth for an in-state
conference, and on creating asset coalitions. One treatment region presented its financial
assessment tool during an NDI Community of Practice webinar, and NDI requested information
on New York Round 6’s curriculum and trauma-informed training for engaging those living with
HIV/AIDS and LGBTQIA populations. DRCs also presented on panels with partners, such as on
career exploration for students with disabilities at local schools. In essence, the DRCs and AJCs
did well to get the word out about DEI and how it can benefit JSWDs. However, we found no
evidence of a proactive plan of dissemination aligned with strategic goals for any particular
project component (e.g., sustainability, capacity, or employer participation in project leadership).
Going forward, associate business services representatives will be key to disseminating practices
related to serving and hiring JSWDs, according to the state lead.
xiii.
Promising Practices
Description
Admittedly, DEI grantees should incorporate cross-systems partnerships into their
strategies as a matter of course.. Nonetheless, it is significant that NY Round 6 opened a wider
doorway for potential AJC customers by seeking out referral partners that serve people who are
blind, ex-offenders and living with HIV/AIDS. The state also addressed AJCs’ difficulties
serving people with mental illnesses and other hidden disabilities through training and technical
assistance. The DRCs proved capable in providing direct services for people who are often
underserved at AJCs, and AJC staff were more comfortable referring this clientele to DRCs.
One DRC asserted:
I always say that we help people with some kind of barrier, whether it’s language, poverty,
disability, etc. I think sometimes by virtue that I’m the DRC, they would just send them to me,
and wouldn’t even register them like everyone else. It’s all about universal access.
A. Essentially, through a determined effort to establish partnerships with agencies whose
customers generally did not have success accessing and using AJC services—and
augmenting the effort with awareness-building and training to reduce AJC staff anxiety
and help them better serve these customers—the DEI laid the groundwork for AJCs to
better serve jobseekers with hidden disabilities and those who are blind.
Rationale for its Implementation
AJC staff and DRCs commented that jobseekers with hidden disabilities and those who
are blind could be better served if AJC staff understood more about these disabilities, the impact
of those disabilities on employment, and strategies for serving such jobseekers effectively. Since
the DEI’s primary objective is to assist JSWDs in achieving economic self-sufficiency and
employment by engaging AJCs, it made sense to prioritize working with agency partners serving
various disability demographics, especially populations who confront stigmas like JSWDs do. As
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
150
well, Project Council members already had connections with the Damien Center and the New
York Aids Coalition.
There is also value to the DEI tapping into the knowledge and partnerships other agencies
have developed to facilitate creation of more seamless service systems. For example, seven
WDAs across the state provide support services, job training, volunteer opportunities and CP
services, including those implemented by NYSCB. NYSCB also offers loans for employment-
related items and an independent living program that provides mobility services, rehabilitation
teaching, low vision services and devices, social casework and adaptive equipment. Partnering
with NYSCB potentially opens the doorway to new resource braiding opportunities. Bringing
other partners into alignment with the DEI and AJCs can benefit the partnering organizations as
well. DEI’s outreach and engagement with agencies serving people who are blind and those with
HIV/AIDS means that those jobseekers can potentially benefit from IRTs, CP, and other DEI
service delivery strategies.
Why the Practice Could be Considered Promising
Developing partnerships is fundamental for any DEI grantee. However, the focus and
energy the NY Round 6 DEI brought to partnership development to widen its reach to JSWDs
served by other agencies—who might not otherwise receive such services—is a promising
approach with the potential for longer-lasting impact. One DRC described how partnerships
could have a direct impact on JSWDs:
There was a flyer that I forwarded to Computers for the Blind. They told me that it wasn’t
readable. I talked to the DEI State Lead and she ran it up the chain, and that’s a change that will
hopefully be happening. The Legal Aid Society in this area has become a partner with the WDAs.
We now have a liaison that will help JSWDs fill out SNAP applications and send them in.
There’s a lawyer also who attends meetings with the WDAs. She helps people with HIV/AIDS
with the SSA medical review. There are smaller [partnerships] for getting people clothing or
furniture. Whenever I meet [a potential partner], I try to let the manager know about them.
A DRC also discussed building upon established partnerships to increase understanding
of employment of people with disabilities, such as by conducting disability-related trainings for
AmeriCorps and the housing/homeless coalition. One training included discussion of the link
between poverty, disability, and employment with social services groups Community Action and
Circles Oneida County. There was also a cross-training with Legal Services of NY. As
referenced earlier, engaging trainers from a Psychiatric Center to present to AJC staff on the
effects of trauma helped AJC employment specialists gain a better appreciation of support needs.
One employment specialist reflected on the value of cross-training:
Further, the work that DRCs did with partners helped ensure smoothly coordinated
services between partners and the AJC. Outside partners also indicated a significant increase in
their knowledge of the AJCs as a consequence of the connecting work of the DRCs, and
therefore a greater inclination to actively partner with the workforce development system.
xiv. Challenges
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
151
Key challenges included developing trustful, working relationships with JSWDs with
significant visual, psychiatric and physical impairments to engage them in services and mentor
their progress, while at the same time working to help sensitize AJC staff to JSWDs’ particular
needs.
Resource constraints also thwarted optimal implementation of DEI practices. One DRC
remarked about being overextended in providing both direct and systems-focused work. AJCs
have their own constraints; in one instance, CP enrollment had to be curtailed due to a lack of
AJC capacity to do career assessment. Despite promising gains made in developing cross-agency
collaborations, stakeholders suggested the challenge of losing the DRC role. One partner
explained:
JSWDs need someone to believe they can do it. Without someone doing the individual work,
people looking for DEI or WIOA services may not get it. I don’t think anyone at the Center
would really take it over and facilitate the process. But I would at least like to see someone here
from a partner agency for a couple days a week that could do benefits planning.
Concerns were also aired about sustaining TTW efforts. Some DRCs worried that
beneficiary outreach and demonstrating how TTW can assist employers may dissipate after the
grant ends due to the relatively small group of engaged employers. The implication is that
developing TTW partnerships can take a lot of time that AJC staff may not have. One AJC staff
commented on the need to have more resources dedicated to engaging TTW customers:
I market to agencies and staff to tell them to invite clients on SSI/SSDI, and I talk about benefits,
work incentives, etc. Without a point person for it, though, I don’t know that anyone could take it
over. I often will look around, and realize that I’m the only one…I don’t think they realize the
funds and the benefit that TTW can provide.
Notwithstanding significant investment in and attention to assistive technology, treatment AJCs
also struggled to access or fully optimize AT. Multiple stakeholders expressed a desire for more
accessible furniture (e.g., adjustable desks), and issues remain with visual accessibility.
Equipment was available in one AJC, but a DRC needed appropriate software to activate this
AT. One DRC experienced challenges in engaging partners to resolve these issues.
In a post-grant call, the state lead revealed how DRCs were initially “very frustrated” with the
amount of effort needed to identify and establish relationships with existing career pathways
systems. DRCs had to spend significant time forming and integrating pathways, including by
cultivating relationships with businesses. Still, once connections were made, the CP strategy was
effective.
Another impediment to optimal implementation was that one treatment region did not
have a partner that serves those living with HIV/AIDS. As a result, this WDA had less
engagement with this population.
xv. Sustainability
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
152
Perhaps a DEI partner we interviewed framed DEI aptly by saying it is a “project, not a
program.” AJC staff were leaning on DRC support much less frequently as the project came to
an end, indicating that DEI was successful in changing staff attitudes and behaviors.
Additionally, the state lead asserted that DRCs were effective in training AJC managers and staff
and partners in the IRT model so that it became “common practice.” She mentioned how IRTs
would continue without DRCs as evidence of systems change, and that the IRT process was
instrumental in aligning resources on behalf of JSWDs. Blending and braiding with ACCES-VR
and CP systems should also endure.
One Round 6 DRC is assuming the role of employment systems manager at the Damien
Center, continuing her trajectory as a systems and services change agent; she was a DRC since
Round 1 and a Disability Program Navigator. Thus, the connection between AJCs and JSWDs
living with HIV/AIDS should solidify. A train-the-trainer curriculum for engaging JSWDs living
with HIV/AIDS—which will involve an employment counselor from all 85 AJCs statewide—
will also help institutionalize this service within AJCs. One session was conducted in June 2019
with partner staff, including the Damien Center, ACCES-VR, NYSCB, and an ILC.
One other DRC will remain employed in the Capital region through TTW funds, while
the two DRCs from HMO will not. The state lead reported that the last few months of the grant
were dedicated to ensuring remaining staff can fulfill DEI roles, including understanding
relevant resources, referrals, and partners. She also emphasized how every AJC in the state
provides universal access, regardless of the presence of a grant like DEI, and that AJC managers
were “extremely on board” with continuing DEI practices.
Regarding sustainable financing for DEI practices, a successful TTW program may
contribute to sustainability if revenue targets are met.. However, according to several DRCs,
constant staff training and recruitment are necessary to maintain a sufficient number of Ticket
holders and job placements. Towards the end of the grant period, each treatment WDA had
several Ticket-eligible beneficiaries who had yet to connect with a local AJC. We heard from
DRCs that some participants need prodding to engage in what could be a lengthy process of hard
and soft skill development to avert the cycle of job loss they may have experienced in the past.
One AJC employment specialist reflected on the importance of TTW to sustaining DEI
practices, in addition to collaboration:
We’ve talked in the past year about putting out another RFP because we believe in the TTW
program. It can help with sustainability. Training is ongoing because we have a partnership with
several agencies. That’s what’s great about my role. I can reach out, introduce people to the
manager, and make sure the partnerships continue on. That’s my hope for sustainability.
Although New York is hopeful about leveraging funds from TTW and braiding resources
from community colleges and other grants to sustain DEI practices, there is trepidation that those
efforts take time and staff resources that may not be available to AJCs after the grant ends. The
partnerships New York established will be pivotal to continuing DEI practices... Close
connections with ACCES-VR, NYSCB, and DSS should lead to more shared customers and
more demand for AJC services; before DEI, many of these individuals did not visit AJCs.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
153
Partnerships with ILCs have also augmented benefits counseling capacity in treatment areas.
Lastly, linkages created with the healthcare sector—including with hospitals and small
businesses to hire JSWDs—should persist. Altogether, the state lead was confident that most
DEI practices and trainings will be accessible due to newly forged and strengthened partnerships.
In a final project call, one DRC asserted that treatment AJCs are “poised to be
accessible” to JSWDs, particularly those with visual impairments. DEI and partner staff
discussed how New York Round 6’s emphasis on assistive technology access through
collaboration and training could serve as a model for AJCs statewide. Technical assistance is
planned to push this approach forward, including a webinar on equal access and section 188. In
addition, after the grant ended, state leadership disseminated AT how-to “cheat sheets” to AJCs,
as well as surveys to assess whether staff were trained on AT since historically equipment has
“collected dust” due to lack of technical staff awareness.
Project staff also offered that New York is also now well-positioned to capitalize on
momentum around apprenticeships for JSWDs. AJC managers will continue to engage with the
American Apprenticeship Initiative.
While it is difficult to determine if practices from NY Round 6 DEI will be sustained–
with an equivalent level of resources to support DEI’s systems and services structure—there is
reason to believe that a number of DEI innovations on behalf of JSWDs will be embedded in
each AJC’s operations in the future.
xvi.
Summary
The New York Round 6 DEI implemented much that is commendable. It built upon the
robust NYESS data management and reporting system to better track DEI employment
outcomes; sustained and developed new partnerships; harnessed assistive technology, training,
and collaboration to engage diverse JSWDs; and continued to strengthen its TTW program..
According to the state lead, New York exceeded each of its grant goals (e.g., number of
individuals served/receiving core services/entering CP).
In addition to TTW, Partnership Plus is a potential avenue for accessing Ticket holders
and generating resources. Cross-agency partnerships fostered through DEI should endure and
contribute to accessible AJCs capable of serving a range of JSWDs, including those with
significant additional barriers.
However, the totality of data suggest that NY could have done more to train AJC staff in
DEI practices so that they are knowledgeable and skilled in using DEI strategies to serve JSWDs.
New York’s dilemma for the future is determining who will continue the forward momentum of
the systems and services progress that DEI made. But, if TTW revenue to support DRC-like
functions combines with new habits, outside partnerships, and training that assist AJCs to better
serve customers with significant employment barriers, NY will have a positive outlook for
building upon the lessons learned from implementing its DEI Round 6 project.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
154
F. State: Washington State; Focus Area: Youth and Adults with Disabilities
i. Stated Goals and Objectives
The Washington Disability Employment Initiative (DEI) Career Pathways Project
proposed to address the employment needs of youth and adults ages 18 and older, with
disabilities across the Seattle-King County and Snohomish County LWIAs by building upon its
nationally recognized career pathways program Integrated Basic Education and Skills Training
(I-BEST). The I-BEST operates in all of the 34 community colleges across the state and has
been proven successful with other populations that have significant barriers to employment.
Washington DEI Round 6 proposed to demonstrate how it could also be an effective approach
for students with disabilities, especially when combined with the additional outreach, expertise,
resources, Integrated Resource Teams, and greater business/employer participation that the DEI
would bring to the partnership. Washington proposed a systems and services change approach
with a focus on transforming I-BEST from “systems that were serving people with disabilities
inadvertently, with no particular focus of effort or coordination of resources, into cultures that
embrace IRTs as an established habit of work, and organizations committed to sustaining a
dedicated capacity to focus coordinated efforts to leverage, blend and braid among multiple
partners in support of increased access and better outcomes for people with disabilities
51
”.
Their Round 6 goals include:
• Assist the Seattle-King County and Snohomish County LWIBs in adopting the use of
IRTs as a person- centered method for collaborating across multiple service systems to
serve job-seekers with disabilities (JSWDs);
• DRCs and AmeriCorps members will convene and facilitate the IRTs for students with
disabilities participating in career pathways programs;
• Encourage and support hosting IRT meetings on college campuses, and mentor career
pathways personnel and other college faculty and staff in how to recognize how IRTs
could help them to identify and apply knowledge, services or other resources that would
improve the ability of a student with a disability to participate and succeed in their class
or program;
• The Coordinating Committee will facilitate collaboration at the systems level by
identifying emerging issues and potential problems and developing collaborative
solutions;
• Establish Wi-Fi hotspots covering one AJC in each of the participating LWIAs. The
AmeriCorps members in each area will identify all of the documents and forms, paper or
electronic, and all of the products currently used with or by jobseekers, and with
assistance from the Washington State Department of Services for the Blind (DSB), work
to ensure that as much as possible of the information and functionality provided by the
AJC is available through the Wi-Fi connection in accessible formats;
51
Excerpted from the Washington Round 6 DEI Project Proposal Narrative (pg.6)
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
155
• Provide group and individual financial education and counseling to improve credit, lower
debt and increase savings, while improving their ability to make more informed financial
decisions;
• Ensure that working-age Social Security beneficiaries who participate in this program are
provided complete, accurate and reliable benefits counseling and individual benefits
plans;
• Recruit, coordinate and manage active participation of businesses and trade associations
that are committed to using career pathways and WIOA programs and services to
improve their access to qualified working-age applicants with disabilities.
ii. Proposed Service Delivery Strategies
Building upon the systems and services accomplishments of its Round 2 DEI grant,
Washington’s proposed Round 6 strategy prioritized the acquisition of Career Pathways jobs for
participants, predicated on close partnerships with academic and credentialing institutions and
with employers. Their project privileges the use of IRTs and Blending/Braiding augmented
by Ticket to Work enrollments with the intent to build upon their I-BEST initiative that features
strongly favors customer-driven services, integrated team planning and resource leveraging. “I-
BEST project is a thoroughly integrated system of curricula connecting ABE, occupational skills
and academic tracks, that students can enter and exit at any point, depending on their individual
needs. These resources include career counseling child care, transportation, financial supports
through TANF, financial aid or other resources
52
.”
iii. Implementation Summary
iv. Capacity-Building: DRCs and WIOA Services
The Context for Capacity Building
Washington was the recipient of a DEI Round 2 grant, with participation by the
Workforce Development Council of Seattle King County, and Workforce Snohomish, that are
also their Round 6 sites. Their Round 6 project proposed to build upon the expertise, culture of
disability inclusion and programmatic infrastructure within these two WDAs to develop Career
Pathways programs that are more inclusive and effective in serving jobseekers with disabilities.
Their project was also facilitated by the systems-level knowledge of the DEI State Lead whose
participation the Governor’s Task Force on the Employment of People with Disabilities,
participation in state policy decision-making on Medicaid issues, and ability in leveraging
partners’ resources was an asset to a project with such a strong emphasis on systems change. The
Governor’s Task Force has a good track record in securing funding to support a comprehensive
approach to employment of people with disabilities, including the DEI, Partnerships in
Employment, and Employment First State Leadership Mentoring Program.
v. DEI Services
52
Excerpted from the Washington Round 6 DEI Project Proposal Narrative (pg. 5).
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
156
The Washington DEI Round 6 focus areas were Career Pathways, IRTs, and Blending
and Braiding Resources. Their progress in implementing Career Pathways and Blending and
Braiding Resources is further described in sections of this report specifically addressing
Partnerships and Collaborations and Promising Practices. This section focuses on the
perspectives of interview respondents and the data that was supplied in their Quarterly Reports
that are particularly related to IRTs and Blending and Braiding as well as Ticket to Work. IRTs
are a specific DEI implementation strategy with established methods and criteria and AJC staff
and managers appeared to be conversant in their application, if not their specific terminology.
So, for example when queried about the Worksource center’s knowledge and use of IRTs an AJC
partner offered that: “We just don’t call it IRT. It is mainly people understanding, sitting together
to work together, not giving mixed messages, that’s just how we do business”. And, from the
perspective of a DVR partner-their counselors have long adopted an interdisciplinary approach to
assisting JSWDs:
“We don’t use the term, but we do that all the time because it’s an interdisciplinary team. We
work with providers and community rehab provider for job placement, case managers, families,
etc.”
Another AJC partner offered: “How do you develop IRT? I don’t know, conceptually I do,
but not really. We have functional teams at systems insight level. Workshops, job clubs, group
services. Mostly workshops. The last one is integrated service delivery”. DRCs acknowledged
that it could be time-consuming to form IRTs, but they were an integral part of their approach to
assisting JSWDs. Customers did not necessarily know that they are receiving IRTs, but some do
acknowledge that they are benefitting from a Worksource team approach to job development:
“I thought the business solutions team has been helpful. So, it’s not just the DRC but all kinds of
teams here. the whole office. Resume building, job searching, benefits planning with the
employment specialist, it’s nice to have the DRC as a directory. Getting me the right information
and what to do”.
IRT development appeared to be successfully implemented by DRCs across the two pilot
sites. While available data does not support the assumption that IRTs meeting the established
definition were initiated on behalf of JSWDs by other staff in addition to the DRC, the
Washington Round 6 DEI reported that they coordinated and facilitated for 220 IRTs since
October 2015.
53
Their target was to conduct 680 IRTs over the life of the grant. While they were
on track towards achieving only approximately 1/3 of their expected outcomes, we find that the
IRT strategy was being firmly embedded within the fabric of the DEI service delivery
implementation plan. In fact, according to leadership responses to their Sustainability Interview
held on 6/25/19, IRTs have become firmly embedded as a method of practice within both
WorkSource Center and were deemed one of the most effective outcomes of DEI:
53
DEI Quarterly Report October-December, 2018 (pg. 1)
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
157
“With the IRTs, it’s effective because it’s powerful for everyone to work together so that all of the
resources are together to help the client. For example, if they’re working with a housing group
that can help the client along with other services”
At the AJC level, data supports the progress that DRCs made in providing training and
employment services to participants. In the estimation of a number of participants participating
in focus groups, the DEI’s presence at WorkSource Centers and their attention to individual
needs, familiarity with partner agencies for supportive services and relationships with employers
resulted in better services than they previously had experienced. DRCs appear to be involved in
both direct employment services and individual case management, but also systems-level
activities where they most frequently identify and convene partner agency staff for participation
in a JSWD’s IRT. We found that in King County, where the DEI is administered through a
private contract, a concerted effort is being made by the DRC to conduct in-reach to homeless
services agencies with the intent of assisting people in transition from homelessness that meet
DEI criteria to participate in the program. In similar fashion, the Snohomish site that is
Department of Labor-administered, has established a relationship with a homeless services
agency to access wrap-around and housing services for DEI customers experiencing
homelessness. Since JSWDs transitioning from homelessness can be among the most challenging
clientele served through the workforce development system, these efforts by the DEI to engage
individuals with significant employment barriers who are served by homeless services systems
can be viewed as an effort to improve cross-systems partnerships.
Another feature of capacity building is to improve accessibility at AJCs with the use of
technology including the promotion of W-Fi and Apps for users to better access services. The
DRCs’ efforts in this regard were augmented by AmeriCorps staff, partially funded through the
grant initially, and later using TTW revenue. It would appear that while gains were made, the
DEI still encountered challenges in achieving the level of accessibility that they had hoped to
reach.
“This office used to be a disability room. I thought it was required. It’s a good requirement.
Actually, it’s not and we were the only one in the state to have a dedicated room. We had
assistive technology in here, but that got dismantled. DEI came in and made it happen. The
equipment was here and the room was here. Not a lot of action and not much demand and not
much staff to run the stuff. DEI upgraded TTY to video phone”.
But technology is just one part of an accessibility assessment. How customers are being
regarded and helped is also an indication of an AJC’s capacity to serve JSWDs. We conducted
two customer focus groups and responses about the level of customer-focused services between
the two pilot sites and the positive influence of DRCs were similar. Participants in both focus
group, strongly endorsed and expressed their appreciation for the attention and services they
receive from their DRC. But they also expressed strong opinions about the capacity of that AJC
to provide them with the assistance they required:
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
158
“There are people but no signs where you can get help. If they can have things posted, it would
be helpful rather than a calendar where…etc. It’s discouraging”.
Participants offered that the help they receive through Worksource greatly improved
when the DRCs came on board. Nonetheless, these participants also wished that the DRC wasn’t
stretched so thinly.
“I didn’t get much help before. I would wander around and would come to the WorkSource
center, but there would be computers around for me to use, but not much other help. Things
changed when the DRC came on board”.
“I think the DRC is excellent. It’s just a complicated system. One person can be wonderful, but
getting a job is hard enough.”
“The one word I can describe as the difference between before and after DEI is “result.” But
there is more than that. There is compassion and caring. WorkSource and WIOA don’t really
care. You are a number. But the DRC cares. He drove all the way to meet me. I didn’t even know
that was an option. You need to come here all the way, drive an hour and a half, to drop your
insurance information, come back tomorrow to pick up your vouchers.”
Another feature of capacity – building is the extent to which WIOA enrollment occurs
with DEI customers so that multiple resources are available to meet the person’s training and
employment needs and project resources can be effectively and more efficiently leveraged. It
appears that positive efforts are underway to leverage WIOA resources on behalf of DEI
participants:
“We enroll them with WIOA first, funding pieces in place. WIOA people will help with the school
enrollment too. I usually make them enrolled at WIOA at that point. All the tuitions are
supported by WIOA or BFET (Basic Food Employment and Training)”
DEI leadership had another perspective on leveraging WIOA services. While DRCs
focused on its efficacy in meeting individuals’ training and employment needs, leadership
viewed it as an important element of systems change: “It is really a balance between direct
service vs. systems change. They are there to be the resource and facilitate services for people
with disabilities. We’ve been helped by WIOA on serving people with disabilities. We would have
to tell them (Worksource centers) that they are part of the people they serve”.
“With WIOA enrollment, we have integrated service delivery here. So, everyone who use the
resource room will get WIOA services”
Data suggests that the presence of DRCs at the Worksource pilot sites has a positive,
direct impact among those they serve. AJC Managers were pleased to have DRCs on their team
and readily acknowledged that they were contributing to a more disability-friendly environment.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
159
In that regard, the DRCs have much in common with the Disability Navigators (DPNs) who
came before them. Like the DPNs, the DRCs are raising the profile of the AJCs as resources
where JSWDs can get the support and assistance that they need. However, it is less clear that the
consistency and individual attention that they get from DRCs could be provided by AJC staff,
absent the presence of the DRC. This is not to suggest that the DRCs are siloed individuals
within the Center, or that their service methods and approaches are not recognized, endorsed, or
even used by AJC employment specialists. It’s that the multi-functional aspects of a DRC’s job
are significantly broad undertakings for one individual. It is unclear to us that the range of
services provided by DRCs, from generating referrals; to hands-on employment services; to
creating partnerships from which flow IRTs; to working with employers on behalf of individual
job-seekers, and; to helping to disseminate what they do and have accomplished both within the
AJC and to external audiences, will be assumed in whole or in part by AJC employment
specialists after the project ends. Research supports that systems and services change take time,
and we heard as much from many DEI staff. It was unsurprising that when we asked participants
what should be done to improve the DEI and employment services that they receive, the
unanimous response was to give their DRCs more time and more help.
“Nobody really helped me before meeting the DRC. No one really wanted to cooperate. I’ve
always felt that he understands my needs generally. He absolutely does not use the cookie-cutter
approach”.
Initially access to benefits planning was provided by a sub-contracted partner. But later,
the 3 DRCs were trained as CWICs and their presence at the AJCs and ability to directly provide
benefits counseling a s they were working with JSWDs on job development became a strong
feature of the program. According to DEI leadership: “For me the Benefits Planning was a
standout. We hadn’t been able to do that before. Having the expertise really helped people move
past the belief that once they had benefits, they couldn’t go to work because they would lose
them”.
Ticket to Work: The two pilot sites approached TTW enrollment (as well as enrollment
in WIOA) differently. Seattle-King made TTW enrollment the first option of participants when
they enroll in the DEI although an AJC partner told us that TTW was not required. “TTW isn’t
required. We always co-enroll with WIOA, so trying to maximize what we have, like
transportation, food stamps…etc. Try to co-enroll them”. However, some of the DEI customers
who participated in the focus group termed the DEI as the ‘Ticket to Work Program’.
Meanwhile, the Snohomish site considered TTW enrollment more strategically and offered it
only when a customer appeared motivated to work and willing to follow through. “Job readiness
defines the ticket assignment. You cannot assign ticket to everyone who comes into the door
because it’s unmanageable. They are really focused on utilizing tickets. 30 tickets are assigned
in this round and about 20 or so have been employed, and generated about $60,000 of revenue”.
Their approach to WIOA enrollment used similar criteria. TTW was a viable program revenue
source and according to leadership the DEI generated almost $400,000 and served approximately
250 SSA beneficiaries were served in TTW over the life of the grant.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
160
vi. Partnerships and Collaborations
The success of the Washington Round 6 DEI owes much to the value ascribed and the
efforts made in developing partnerships. As described below in the ‘Best and Promising
Practices’ section, their CP focus depends upon partnerships with credential-issuing institutions
and with employers. Their September 2018 self-report that 109 people received CP services and
139 people were employed through DEI over the life of the grant, would suggest that these two
vital links in the CP continuum were being established. As described in the ‘Best and Promising
Practices’ section below, partnerships between the DEI and the Northwest Access Fund (NWAF)
are proving fruitful in leveraging the resources needed to add financial literacy and asset building
training and support to the DEI’s mix of services. While these are important examples of
Washington’s efforts to build partnerships, data on the emergence of a mutually beneficial
partnership between the DEI and DVR after that entity implemented the Order of Selection
(OOS), suggests that positive outcomes may be developed in challenging situations if
communication is present.
DVR began accepting only clients with severe mobility disabilities and/or people with
Social Security Medicaid who are on the employment waiver when they went to OOS in 2018.
As an alternative to wait-listing, customers who didn’t meet these new DVR service
requirements were referred to either the Seattle- King or Snohomish County DEIs. Collaborating
this way on program referrals benefits both DVR and the DEI. When a person is ineligible for
DVR OOS services and meet the DEI eligibility criteria, DVR can refer them to DEI. And, even
although there is no Partnership Plus arrangement, DVR and DEI have agreed that the DEI
Round 6 can sign their Ticket to Work. And that also means that a person can benefit from DEI-
initiated wrap-around services like training or certification and blending and braiding services
like WIOA, TTW, (Supplemental Nutrition Assistance Program (SNAP), paratransit assistance,
Section 8 housing, or other programs. In essence, DEI utilized is being used as DVR’s “Plan B”
for people who would otherwise not be served in those the counties. The result is an increase in
the number of new DEI participants and another example of braiding of services. The
Washington DEI Round 6 project self-reported in their July-September 2018 Quarterly Report
that over 200 DEI participants entered into employment, as of that date
54
.
vii. Customer Focus
Customers were generally enthusiastic in their praise of the services they get directly
from their DRC, but less so from their prior experiences at Worksource Centers. When analyzing
the presence or lack of a strong customer focus in DEI, our evaluation protocol probes not only
for customer satisfaction, but also for corroboration from customers, DEI staff, and partners that
customers are at the center of the process and driving the types and range of employment
services they prefer. An AJC partner described how they view providing services to JSWDs,
suggesting that while the customer is the focus throughout the process, the orientation of the DEI
and the AJC were parallel, distinct, and mutually reinforcing:
“If we focus on DEI, what (the DRC) could do differently is to help us to be reminded that
54
DEI Quarterly Report July-Sept, 2018 (pg.1)
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
161
employers have a hard time and don’t want to hire PWDs. Because it comes up every time. The
DRC is very customer-focused, we are business-focused”.
We obtained a significant amount of data from the participant focus groups that confirms
that the Washington DEI Round 6 has a strong focus on customer needs. While most talked
about the help that they were receiving from their DRC to develop resumes and search for a job,
the responses suggest that the DRCs are providing valuable assistance as knowledgeable
navigators within and across systems. One participant underscored how important it was that the
DRCs have systems as well as services knowledge and that they helped job seekers not only to
get a job, but also to get a job with a future.
“As far as, I am not sure what other services are available. I am still continuing my education.
But the DRC really wanted me to go through WIOA and that is the reason why I did. WIOA
denied me, so I talked to him, he talked to them, then WIOA called me back and I was approved.
He has the magic wand. I’ve been trying for six months”. JSWDs
“I was trying to figure out TTW, the DRC led me through it. It’s to the point that they would give
me a minimum wage and I would work. Sure, but how can I support myself and my son? I work
hard for what I’ve done. They would try to get me jobs that are just above the minimum wage.
And the DRC is trying to get me a job that is more solid work. I don’t have schooling. He helps
you and he has a positive attitude, so he’s been the true beam of light in this whole process.”
JSWDs
It appears that both pilot sites are doing well in focusing on customer needs, in ways that
participants generally did not experience in prior involvement at their AJCs. DRCs themselves
provided examples of how they helped JSWDs overcome obstacles to employment when in the
past, those challenges were unable to be surmounted. From the customer standpoint, DRCs
appear to be regarded as invaluable resources for navigating systems and getting jobs, although
they say that the “Worksource Centers need more staff like them.”
viii. Employer Partnerships
The DEI Round 6 employer development strategy was well-conceived, and if it had
succeeded, could have been an example of a DEI Promising Practice. The DEI contracted with
Washington Business Association (WaBA) and their consultant who is also a well-regarded
businessman who can speak the language of business when he meets with human resource staff
of some of the area’s major employers to better secure career pathways opportunities for DEI
customers. “WaBA contracted to partner with DEI to achieve three outcomes: recruit and secure
commitments from 10 businesses and two industry associations; facilitate and coordinate with
business employment by creating curricula, credentials, and certification; and secure and manage
business commitments to participate in training and provide work-based training opportunities
like internships
55
”. This employer-responsive strategy was intended to be augmented by other,
concurrent employer engagement practices, including the Worksource business development
teams; the DRCs’ own job development efforts; and the business development efforts
55
DEI Quarterly Report April-June, 2018 (pg.4)
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
162
implemented through the State VR agency. By relying on a business owner/consultant as the
liaison between the DEI and employers, who speaks the language of business through direct
experience; is familiar with the business and economic forces that affect hiring; and is welcomed
into the meetings and forums that businesses typically attend, the business case for hiring people
with disabilities can be made more strongly than through traditional employer partnership
arrangements. The WaBA’s consultant also uses a wheelchair and the DEI team agreed that
having a mobility-related disability might also be a powerful portrayal to employers on how a
disability need not be a barrier to job success. Unfortunately, two factors combined that
undermined the potential success of this proposed innovation. First, the DEI pilot sites did not
come up to speed with serving customers and preparing them for career pathways jobs at the
pace that the consultant was outreaching and engaging potential employers. So, even if some
employers were initially receptive, the pipeline of job candidates was not built. The consultant
also ran up against employer reluctance to hire persons with disabilities.
“I misjudged the companies. I thought they would have open arms for PWDs. But that was not
the case. It’s getting them on board and recapitalizing it. I was targeting mid-level companies,
even when I knew someone there. I got stalemated”.
The second challenge proved to be insurmountable for the DEI, when the sub-contract
was discontinued before the end of the project due to staff cuts and the unavailability of the
consultant due to prolonged medical leave. Consequently, rather than having a statewide
employer partnership resource pointed towards securing work experiences and placements in
career pathways jobs for JSWDs, each pilot site relied upon their more traditional methods of
DRC-initiated employer engagement including Worksource business development teams and
AJC Job Fairs to match participants with their preferred jobs. Nonetheless, we suggest that the
concept of developing CP jobs through the WaBA partnership with a business-business
orientation was potentially promising and could have had significant job development outcomes
if circumstances were different.
ix. Career Pathways
Career Pathways are described in detail within the Partnerships and Collaborations and
Promising Practices sections. Their July-September Quarterly Report indicates that 109 DEI
participants during the life of the grant in Snohomish County entered training of some kind and
in King County, 186 DEI participants entered a vocational training program and that 56 DEI
customers would earn a credential during the life of the grant
56
. We know from available
Participant Tracking System (PTS) data that participants are being entered into Career Pathways.
However, the PTS Report for the period ending February 26, 2019 only indicated that 72 of 402
participants received CP services. Consequently, we cannot determine whether or not the DEI
met their targeted completion outcomes, from among participants who were enrolled in CP-
focused services.
56
DEI Quarterly Report July-September, 2018 (pg. 1)
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
163
x. Dissemination
The WA Round 6 project dissemination progress was analyzed according to three
criteria: 1) How well is the project making its services known to jobseekers with disabilities, and
promoting their referral and enrollment? 2) is the project effectively disseminating its mission
and services to partners, including employers to increase their awareness of the DEI? 3) Is the
project disseminating its successes through various media, including presentations, papers, etc.?
It appears from interview data that the project could do more to make its services known to
customers and partners and to convey their progress and successes. One DRC offered that
dissemination was primarily being done informally and that: “We don’t really do much
community-facing marketing. We don’t have resource fairs. We’ll talk to housing group or
downtown emergency center, and the homeless shelter”. From AJC leadership we obtained
disparate responses:
Referrals also from community groups. I get invited to different meeting doing presentations and
meeting housing partners. I talk about what DEI does and what we can do for their CWDs. AJC
partner Outreach for DEI? Word of mouth.”
Participants in the Renton area focus group tended to equate their DEI as the TTW
program. Hearing about, and enrolling in TTW was their first exposure to the DEI. “I think
someone verbally mentioned TTW and I wasn’t sure what it was. The DRC was great, she
followed up, was there to help, and I am very thankful for her.”, and a DRC offered that hearing
about TTW and how it may help a person to get a job often led to the connection to the DEI if
the need for its services was established. “No one really comes to get enrolled in DEI, but come
here to get a job. Unless they come for DEI services and need some kind of financial support.
Then, those will come specifically for DEI”. Outreach to employers was a planned strategy with
WaBA as the broker for those relationships although that initiative ended during the course of
the grant. A member of WorkSource leadership reported that they regularly reach out to
employers, but that they were unsure about how well the DEI as doing that on their own:
For this grant, I haven’t heard of any employers. I think I heard there were a couple job fairs,
encouraging people to hire PWDs. I haven’t gotten a list of employers or anything.”
While the DEI itself, at both locations appeared to struggle with designing and
implementing a dissemination strategy, it must be noted that strong DEI state leadership
regularly promoted the initiative at the state level, including its accomplishments and challenges
as a way to enhance its profile and keep the issue of addressing the needs of jobseekers with
disabilities high on the state leadership agenda. This is a significant accomplishment that
potentially creates a platform for the development of related disability and employment efforts
linked to the WorkSource system in the future.
xi. Promising Practices Blending and Braiding Resources
Description
From the outset of their project, the Washington DEI Round 6 recognized the necessity of
building strong partnerships that would result in resource leveraging in order to address the
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
164
complexity of needs of their participants. Their strategy was comprehensive and they understood
that an employment initiative focused on Career Pathways required participation by academic
and credentialing institutions at ‘one end of the pipeline’ and employers at the other end of the
pipeline’ who were engaged as partners who would be incentivized to hire qualified JSWDs. In
both King and Snohomish Counties, resource leveraging with a number of career preparation
entities was implemented. According to their July-September 2018 Quarterly Report, 295 DEI
participants entered a vocational or educational program in their career pathway
57
, appearing to
substantiate their success in engaging these entities in their project. The diversity of career
preparation tracks is also impressive and speaks well for the person-centered career development
focus of their project. They reported in their April-June 2018 report that “Clients were enrolled
or received accreditation or certification from programs like iCATCH, Edmonds Community
College hospitality certification program, substitute teacher refresher classes, Everett
Community College Western Washington University classes and certification workshops,
Masters esthetics, El Centro de la Raza Latino’s Finance training program
(http://www.elcentrodelaraza.org/what-we-do/education-and-asset-building/), South Seattle
College’s medical office, technical trades and welding programs, YWCA Career Work$
(https://www.ywcaworks.org/programs), Lake Washington Technical College
(https://www.lwtech.edu/academics/), and Seattle Central Community College’s Business
Technology program”
58
.
A second successful blending and braiding strategy is demonstrated by their partnership
with the Northwest Access Fund (NWAF) to provide financial literacy and income stability
training and support to participants. Washington recognized that getting a job is only one part
(although a necessary part) of the income self-sufficiency process. People seeking to escape
poverty also need help in managing their assets, credit, and financial planning. Although we find
that a number of DEI states include blending and braiding resources as one of their services
delivery strategies and many offer Social Security benefits planning, Washington State augments
these services by leveraging partner resources to provide financial literacy and asset development
consultations for job-seekers with disabilities. Through individual interviews and document
review, we find that the Washington DEI financial literacy partnership is a potentially model
practice whose elements might be demonstrated in other locations that serve individuals with
barriers to employment. They report that their pilot sites referred over 90 DEI clients to their
asset development partner NWAF for financial wellness training sessions and the total number of
asset development and financial wellness training sessions is 187 over the three-year life of the
grant
59
.
Rationale for its Implementation
Resource leveraging is a critical factor in sustaining DEI services after the grant ends.
The Washington DEI Round 6 focused efforts not only on income generation through Ticket to
Work, but tackled the problem of the difficulties that JSWDs often have in connecting to
accredited and business and industry-relevant education and training programs and an often-
57
DEI Quarterly Report July-September, 2018 (pg. 1)
58
DEI Quarterly Report April-June, 2018. (pp. 1-2)
59
DEI Quarterly Report October-December, 2018. (pg. 2)
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
165
overlooked concurrent problem when JSWDs start earning money, but lack the skills needed to
manage their finances and grow their assets.
Why the Practice Could be Considered Promising
Two factors were apparent in assessing the importance of Washington’s DEI resource
leveraging strategies that other DEIs could consider. The first is addressing multiple converging
interests. This includes recognizing that Washington State’s growth industries in a low-
unemployment economy are interested in hiring qualified job candidates. It also includes
recognizing that the interests of JSWDs and the DEI is to help secure credentials and access to
jobs with a future, rather than entry-level, low-wage employment. By leveraging CP training
partnerships, it would appear that the interests of all parties can be met. The second factor are the
DEI’s congruent approaches to address poverty among JSWDs. Their DEI leadership
recognized that benefits planning and re-entry into employment is not enough to help people
break the cycle of poverty and that helping JSWDs manage their finances and accumulate assets
through their NWAF partnership is needed as well for their participants’
long-term financial stability. And, with NAWF’s role as both a service provider, and as an initial
point of entry for customers that can later be referred to DEI, job-seekers with disabilities have
the opportunity to learn about, and plan for financial wellness even before they start their job
search.
xii. Challenges
We suggest that the Washington DEI Round 6 has encountered, and may continue to face
a number of significant challenges. One of these is how to better embed the knowledge, skills
and participant trust that DRCs appear to have created into the Worksource sites so that
‘everyone becomes a DRC’ when it comes to serving JSWDs. We also note that great impetus
for the development of the DEI, particularly for the level of integration with the state’s other
disability-focused efforts is greatly attributable to the leadership of the DEI State Lead, who
passed away in 2018. Consequently, the DEI during its final year, and for sustainability of its
services and integration of those services into the state’s fabric of disability-related services must
address ways to seamlessly continue strong, systems change-oriented leadership. Revenue
generation is also a challenge and at this point, and we do not know how much TTW or
leveraged revenue has been acquired by the end of the project and how it will be used to continue
DEI’s progress after the grant ends.
xiii. Sustainability
One measure of sustainability is the success of a DEI to establish employment networks
and generate sufficient TTW revenue to support employment of DRCs after the grant ends.
Another factor in determining sustainability is to validate that partners, including AJCs, have
learned DEI practices and have embedded those within their own systems so that, even in the
absence of having specific DRC-designated staff, DEI practices are continued and JSWDs have
improved access and use of mainstream as well as other disability employment services
resources. All DRCs and project leadership staff were incorporated into the state’s RETAIN
grant, ensuring that DEI perspectives, partnerships and skills would continue in both AJCs.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
166
According to DEI leadership: “The benefits planning will be used in Snohomish. The Career
Pathways as well. It was excellent. I think we’re talking about sustainable. Ticket to Work will
be kept. We used Ticket to Work to bring in an AmeriCorps worker and train him in Benefits
Planning. Co-planning that is using IRTs has become much more the norm in the system then
when the program first began”. According to information provided at the Sustainability
Interview, TTW revenue was saved until the DEI grant was over and will be used to get a staff
member to work on Ticket to Work on a continued basis and also develop services for youth.
And, TTW revenue is braided with RETAIN to support 2 of the DRCs that are now RETAIN
Return To Work Coordinators who are funded 50/50 on Return to Work and Ticket to Work and
therefore at least 50% of their time is directly working with Social Security beneficiaries.
xiv. Summary
Our data analysis suggests that there is much to commend regarding the progress that the
Washington Round 6 DEI made over the life of its grant. There is little doubt, from our
perspective that DEI customers feel that they are receiving better services now that they have
DRCs to assist them. DVR, despite its Order of Selection limitations, appears to regard the DEI
as one alternative to serving customers who would otherwise languish on a waiting list. And, we
found ample evidence that DRCs and DEI leadership are both skilled and committed to
sustaining the intent of the program and will continue to incorporate its strategies into the two
WorkSource Centers with their RETAIN grant and in their WIOA services when serving
JSWDs. We offer that the Washington Round 6 DEI has succeeded in achieving significant
systems and services change.
VIII. EVALUATION
The DEI Round 5 and Round 6 impact evaluation included three distinct research
designs. The first was a matched comparison group design, with the treatment and comparison
groups designated at the WDA level. Treatment WDAs were selected by the grantees as having
the capacity to implement the Rounds 5–6 DEI intervention. The research team selected the
comparison group WDAs to align with the demographic and economic characteristics of the
treatment WDAs. Discussions with each grantee about their possible comparison WDAs were
used to finalize the list of comparison WDAs.
60
In this primary analysis, a matched comparison
group of individuals with disabilities provided the counterfactual so that there were no
systematic differences between the two groups of participants that may have influenced program
outcomes. The three outcomes were: (1) employment in the first quarter after AJC exit,
(2) employment retention within the three quarters after AJC exit, and (3) total wages in the
second and third quarters after AJC exit, per participant.
61
60
See Appendices 2 and 3 for the full list of treatment and comparison WDAs for each state that the research team
included in the QED analysis.
61
The first two outcomes use employment information and have the same number of observations. The third
outcome uses wage information and has fewer observations with non-missing information.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
167
The secondary impact analysis featured a short interrupted time series (SITS), and the
sample consisted of only participants in treatment WDAs. We used this as a baseline impact
measure on how outcomes changed before and after the appearance of interventions for each
round. This analysis was particularly important in instances in which states implemented DEI-
funded interventions across the entire state, which resulted in the contamination of these
interventions for evaluation purposes. This analysis was also necessary when the grantee state
was comprised of a single WDA, in which case there was no comparison group within the state
to match and compare the treatment group with.
A tertiary analysis featured an additional quasi-experimental design (QED) that measured
the impact of the specific Career Pathways component that was part of the Rounds 5–6
intervention. The sample consisted of only participants who were enrolled in treatment WDAs.
In this design, we matched participants enrolled in a CP program with similar participants who
did not. These participants who did not enroll in a CP program may have enrolled in other
programs and services, such as staff-assisted core, intensive, or training services.
A. Matched Comparison Group Analysis When Comparison Groups Exist
The first QED examined the overall impact of the Rounds 5–6 DEI interventions on
individual-level outcomes. To measure the overall impact of these interventions, we created a
treatment group that consisted of the grantee WDAs, and a comparison group of
demographically similar WDAs that were within the same state. We defined treatment
customers as those who self-disclosed a disability and participated in the Rounds 5–6
interventions in the treatment WDAs. Observations were at the individual level, and by using
demographic information from WIA administrative data,
62
we matched individuals from the
treatment WDAs with similar individuals in the comparison WDAs.
Any QED should minimize systematic observable or unobservable differences between
WDAs in the treatment and comparison groups except for the availability of DEI services. We
used a multilevel model regression analysis to determine the impact of the DEI services in order
to account for the fact that the level of inference was at the WDA level (i.e., the differentiation of
DEI interventions was at the WDA level) and to control WDA-level and participant-level
characteristics.
Each set of regressions also accounted for variations in inherent demographic and
economic characteristics across WDAs and states using multilevel mixed effects. This type of
fixed effects analysis accounted for the nested nature of WDAs within states.
The analysis used the model below:
(1) y
ijk
= α + β
1
T
jk
+ β
2
Baseline
ijk
+ β
3
Participant
ijk
+ β
4
WDA
jk
+ β
5
State
k
+ d
jk
+
e
k
+ ε
ijk
62
The administrative data used in the impact evaluation changed from WIA to PIRL during the time of the
evaluation. The research team cross-walked all relevant variables from WIA to PIRL, thereby creating one dataset
that had a uniform set of variables for analysis.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
168
Where
y
ijk
= the outcome for participant i in WDA j in state k
α = covariate-adjusted mean participant outcome for comparison WDAs
β
1
= impact of DEI (i.e., the difference between the mean outcome for treatment WDAs and
the mean outcome for comparison WDAs)
T
jk
= 1 for treatment WDA and 0 for comparison WDA
β
2
= parameter estimate for the contribution of the participant-level baseline measure
Baseline
ijk
= pre-intervention measure for each participant i in WDA j in state k
63
β
3
= a vector of parameter estimates for the contributions of participant-level covariates, in
WDA j in state k
Participant
ijk
= a vector of baseline covariates for each participant i in WDA j in state k
β
4
= parameter estimate for the contribution of WDA-level covariates
WDA
jk
= WDA-level baseline covariates for each WDA j in state k
β
5
= parameter estimate for the contribution of state-level covariates
State
k
= state-level baseline covariates for each state k
d
jk
= a random intercept for WDA j in state k
e
k
= a random intercept for state k
ε
ijk
= a random error term for participant i in WDA j in state k
We estimated impacts using a three-level model with the treatment impact estimated at
the WDA level.
64
We accounted for inherent differences between the WDAs and states by
comparing their characteristics through a multilevel fixed and random effects model. The impact
of the DEI intervention is represented by the WDA-level parameter estimate, β
1
. The parameter
estimate quantified the difference in the participant outcome for treatment WDAs compared to
the outcome for “business-as-usual” comparison WDAs. If the p-value for the parameter
estimate was less than 0.05, we concluded that there was a statistically significant impact of the
DEI intervention on the given participant outcome.
B. Matching Using Treatment and Comparison Group Individuals’ Characteristics
To take full advantage of the individual-level data characteristics that we had obtained
through the WIA data system, we used demographic and employment characteristics to help
create the match between participants in the treatment and comparison sites. We anticipated that
there would be enough variation at the individual level for the evaluation to create fine-grained
propensity scores, which was confirmed by the resulting balance across the treatment and
63
These are parameters that were captured at the time of entry, which could have changed during the time that
individuals exited the AJC (e.g., education level).
64
The Stata command “mixed” was used for all regression analyses to account for WDA and state fixed effects, as
well as random effects.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
169
comparison groups. Matching participants across different states did not occur during the
propensity score matching process.
The propensity score matching analysis involved multiple iterations of the matching
process to ensure that our final match across the treatment and comparison groups was well-
aligned. As we matched all eligible individuals from the treatment WDAs to all eligible
individuals from comparison WDAs, we estimated the average treatment effect of DEI on
outcomes.
After each cycle of the matching process, we determined if the treatment and comparison
groups were equivalent at baseline; that is, equivalent in their characteristics prior to the start of
their enrollment services at the AJC, within 0.25 standard deviation of each other. If equivalence
was not demonstrated, then the propensity score model was modified to make equivalence more
likely at the next iteration.
As the services were provided through an AJC (as opposed to a community or technical
college), we acknowledged that participants enrolled in services at different points in time. To
help determine whether participants were roughly starting at the same time, we matched
treatment and comparison participants using their date of first enrollment at the AJC. This helped
offset any time-dependent biases that may have been present if we matched participants who
received services at very different points in time.
C. Short Interrupted Time Series (SITS) Analysis When No Comparison Groups Exist
In the evaluation of the impact of a program using a matched comparison design, the
methods necessitate the use of a comparison group to match the treatment group based on
characteristics that are available in the data. However, there are some instances in which a viable
comparison group is unavailable or simply does not exist. In the case of the DEI evaluation, there
were several states in which there were no viable comparison WDAs (e.g., South Dakota in
Round 5 and Alaska in Round 6). In these instances, we looked toward another methodological
approach that does not require a comparison group — a SITS approach.
In our evaluation, we used SITS for all grantee sites to measure a baseline impact estimate
of DEI by comparing the pre-intervention time trend on participants’ outcomes to post-intervention
outcomes.
The SITS design measured the intervention impact as a departure from the expected
levels of the outcome measure (e.g., employment, employment retention, and earnings) when
projected forward in time as an estimate of what would likely happen were the treatment not
introduced at all (see Shadish, Cook, & Campbell, 2002; Bloom, 2003). The SITS design
entailed: (1) generating a counterfactual for the outcome measure (which was the expected level
of the outcome in the post-intervention period in the absence of the treatment as the projected
trend) by using pre-intervention observations of the outcome measure, and (2) modeling the
treatment impact as a deviation of actual post-intervention outcomes from this counterfactual.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
170
SITS necessitates the use of multiple pre-intervention data points to form a linear
counterfactual trend of the outcome variables by extrapolating the pre-intervention trend
information into the post-intervention period. If we found a departure from this counterfactual,
the multiple post-intervention data points represented the impact of the intervention on
outcomes.
Exhibit 4 demonstrates a scenario in which there are three time points prior to the start of
the intervention and three time points after the start of the intervention. For SITS to be viable in
the DEI evaluation for the states that did not have a comparison group, we needed at least two
time points prior to the start of the intervention and two time points after the start of the
intervention. However, we ultimately used five time points both before and after the start of the
intervention for greater accuracy in measuring time trends.
Prior to the intervention, a trend line (solid blue line) is created using the three pre-
intervention time points. The dotted blue line represents the counterfactual measure as the
projected trend using pre-intervention observations. The average impact is measured by Δ
T
,
which is the deviation between the counterfactual measure and the observed post-intervention
measure (solid gray line).
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
171
Exhibit 4: Pre- and Post-Intervention Trend Lines to Measure the Impact of an
Intervention
The analysis used the model below:
(2) y
tij
= α + β
1
t + β
2
Post Intervention
t
+ β
3
Participant
ijk
+ β
4
WDA
jk
+ β
5
State
k
+
d
jk
+ e
k
+ ε
tijk
Where
y
tij
= the outcome for participant i in WDA j at time t
α = covariate-adjusted mean participant outcome for the counterfactual at t = 0
β
1
= the time trend over all time periods
t = the counter for observations, and t = 1, 2, and 3 denote the three pre-intervention time
periods; t = 4, 5, and 6 denote the three post-intervention time periods
β
2
= covariate-adjusted impact of DEI (i.e., the difference between the outcome for the
observed post-intervention trend versus the counterfactual trend)
Post Intervention
t
= 1 if t = 4, 5, or 6; otherwise 0
β
3
= a vector of parameter estimates for the contributions of participant-level covariates
Participant
ij
= a vector of covariates for each participant i in WDA j in state k
β
4
= parameter estimate for the contribution of WDA-level covariates
WDA
j
= WDA-level covariate for each WDA j in state k
β
5
= parameter estimate for the contribution of state-level covariates
State
k
= state-level baseline covariates for each state k
d
jk
= a random intercept for WDA j in state k
e
k
= a random intercept for state k
ε
tijk
= a random error term for participant i in WDA j in state k at time t
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
172
D. Measuring the Impact of the Career Pathways Component
An additional QED examined the impact of the specific CP interventions on participant-
level outcomes. We implemented this QED on all grantee sites since all grantees had a CP
component as part of their DEI interventions. While the primary analysis determined the impact
of the overall Rounds 5–6 interventions on customer outcomes, we believed that it was also
important to isolate the impact of the CP training on outcomes. Career Pathways comprise an
important bundle of interventions being used by USDOL and other federal agencies and is
expressly different from the previous Rounds 1–4 DEI interventions.
Much like the primary analysis, we used our available demographic information to match
customers according to their propensity scores in this QED. Although there is a self-selection
issue here in that customers who were more motivated to learn and get a job in a particular field
may have opted into a CP program, this is not uncommon among QEDs in postsecondary
research. As long as the treatment and comparison groups could demonstrate baseline
equivalence, measurements were taken at the same points in time across both groups, and
outcomes were consistently defined and collected across both groups, this type of QED would
receive a moderate causal evidence rating according to federal guidelines (e.g., CLEAR, What
Works Clearinghouse).
The analysis used the model below:
(3) y
ijk
= α + β
1
T
ik
+ β
2
Baseline
ijk
+ β
3
Participant
ijk
+ β
4
WDA
jk
+ β
5
State
k
+ d
jk
+
e
k
+ ε
ijk
Where
y
ijk
= the outcome for participant i in WDA j in state k
α = covariate-adjusted mean participant outcome for the comparison group
β
1
= impact of Career Pathways (i.e., the difference between the mean outcome for
participants and the mean outcome for non-participants)
T
ik
= 1 for Career Pathways enrollment and 0 if not
β
2
= parameter estimate for the contribution of the participant-level baseline measure
Baseline
ijk
= pre-intervention measure for each participant i in WDA j in state k
β
3
= a vector of parameter estimates for the contributions of participant-level covariates, in
WDA j in state k
Participant
ijk
= a vector of covariates for each participant i in WDA j in state k
β
4
= parameter estimate for the average contribution of WDA-level covariates
WDA
jk
= WDA-level covariate for each WDA j in state k
β
5
= parameter estimate for the average impact of state-level covariates
State
k
= state-level covariate for each state k
d
jk
= a random intercept for WDA j in state k
e
k
= a random intercept for state k
ε
ijk
= a random error term for participant i in WDA j in state k
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
173
We estimated impacts using a three-level model with the treatment impact estimated at
the participant level. The impact of the CP intervention is represented by 𝛽
1
. The parameter
estimate quantified the difference in the participant outcome for CP participants compared to the
outcome for those who do not participate in CP services. If the p-value for the parameter
estimate was less than 0.05, we concluded that there was a statistically significant impact of the
CP services on the given participant outcome.
E. Treatment and Comparison Group Individuals in the Career Pathways Analysis
Enrollment in CP was determined by the AJC DRC or employment counselor. AJCs
conducted assessments of the training needs of each DEI participant to determine if they were
candidates for CP or other services (e.g., core, staff-assisted core, and intensive). As such, the
treatment group consisted of individuals in treatment WDAs who elected to enroll in one of the
four conditions: (1) core services + CP, (2) staff-assisted core + CP, (3) intensive services + CP, or
(4) AJC training + CP. The comparison group consisted of individuals in the same WDA as
treatment individuals who did not enroll in CP services.
Much like the primary analysis, we used demographic and employment characteristics in
this QED. The matches took place within the grantee WDA sites, and as we demonstrate, there
was enough individual-level variation to make accurate matches across the treatment and
comparison groups.
IX. IMPACT ANALYSIS
The main impact analysis determined the effect of the DEI intervention on outcomes by
using two quasi-experimental approaches. A tertiary analysis determined the effect of CP
programs on outcomes, using only individuals who received services in the treatment group
AJCs. To provide context for these findings, the study also examined descriptive characteristics
of all the individuals in the WIA data sample, as well as findings on activities of daily living and
disability type by individuals in only the comparison group. The overall findings suggest that
although DEI as a program did not have whole-scale impacts on outcomes, CP enrollment did
have impacts on outcomes even after accounting for selection bias.
A. Descriptive Statistics
Each of the regression analyses used the same set of variables as control variables in an
effort to control for individuals’ varying characteristics as captured in the WIA data. These
variables represented the most complete information captured in the intake process at AJCs and
provided a representation of demographic characteristics, disability status, and previous
employment characteristics. The characteristics were not mutually exclusive and were meant to
be comprehensive descriptors of the participants in our sample.
The control variables included gender, using a binary indicator if the individual was
female. Binary variables also indicated race and veteran status at the time of intake. There are
several mutually-exclusive categories for race including Hispanic as opposed to a separate
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
174
indicator for ethnicity. Disability status was self-disclosed at the individual level, with indicators
for physical, mental, or both physical and mental disabilities, though most persons did not
indicate their specific disability. Control variables also included previous employment, college
attainment, and two measures of need: (1) low-income status (as indicated by a series of
measures of federal, state, or local support) and (2) receipt of TANF, SSI, SSDI, or any other
type of cash assistance. Lastly, a series of binary variables indicated limited English proficiency,
whether the individual was ever homeless, was ever a criminal offender, or was a single parent.
Prior to controlling for individuals’ characteristics in a regression analysis, we found that
there were statistically significant differences between the treatment and comparison groups for
employment and wage outcomes. Exhibit 5 shows the average outcomes and descriptive
characteristics of the analysis sample for the comparison group QED for both the treatment and
comparison groups. Almost all characteristics were different across the two groups, with some of
the largest differences in race, disability type, college attendance, low-income status, and public
assistance receipt.
Exhibit 5: DEI Comparison Group QED Raw Sample Characteristics — Unmatched
Treatment
Group Mean
Comparison
Group Mean
Difference
65
CHARACTERISTICS
Female
44.2%
43.1%
1.1pp
Hispanic
12.5%
10.4%
2.1pp***
Black
19.7%
21.2%
-1.5pp***
Asian
2.1%
3.4%
-1.2pp***
White
62.3%
60.6%
1.7pp**
Veteran
11.9%
14.6%
-2.7pp***
Physical disability
11.9%
15.6%
-3.7pp***
Mental disability
17.4%
12.1%
5.3pp***
Physical and mental disability
2.3%
3.1%
-0.8pp***
Previously employed
14.8%
15.6%
-0.8pp
Attended college
35.3%
40.2%
-4.9pp***
Low-income status
34.2%
25.6%
8.6pp***
Received any public assistance
15.2%
12.5%
2.7pp***
Homeless
1.6%
1.6%
0.0pp
Criminal offender
1.9%
2.6%
-0.7pp***
Limited English proficiency
0.3%
0.9%
-0.6pp***
Single parent
3.0%
4.4%
-1.4pp***
Number of observations
12,776
8,193
Source: Rounds 5 and 6 WIA data.
Two-tailed t-tests used for statistical significance.
*** p<0.01, ** p<0.05, * p<0.10
65
Differences in percentage points are listed as “pp” in this and subsequent exhibits.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
175
B. Employment/Retention/Earnings Outcomes
Exhibit 6 shows the outcomes of the Round 5 and Round 6 individuals across treatment
and comparison groups. Approximately one-quarter of the individuals were employed in the first
quarter after AJC exit, with the treatment group having a 3.6 percentage-point advantage over the
comparison group. Employment retention over the first three quarters after AJC exit dropped to
below 15 percent, with the treatment group again having a higher level of sustained employment.
There were, however, no noticeable differences in wages across the second and third quarters
after AJC exit between the two groups.
Exhibit 6: DEI Comparison Group QED Raw Sample Outcomes — Unmatched
Treatment
Group Mean
Comparison
Group Mean
Difference
OUTCOMES
Employed first quarter after AJC exit
27.1%
23.5%
3.6pp***
Retained employment three quarters after
AJC
14.9%
12.3%
2.6pp***
Total wages in the second and third quarters
after AJC, per participant ($)
2,429.25
2,312.77
116.48
Number of observations
12,776
8,193
Source: Rounds 5 and 6 WIA data.
Two-tailed t-tests used for statistical significance.
*** p<0.01, ** p<0.05, * p<0.10
C. Survey to Collect Information on Activities of Daily Living and Disability Type in
Comparison Group Sites
Our ability to provide context surrounding individuals with disabilities and how they
progress through their day depended on the information collected on activities of daily living and
disability type. The treatment WDAs collected this information through the Participant Tracking
System, and the comparison WDAs collected the same information through a novel comparison
group survey. Both data collection instruments were administered upon entry at the AJC for any
customer who disclosed a disability to the AJC representative.
For the comparison group survey, individuals who shared that they had a disability and
were 18 years of age or older were surveyed at the AJCs that: (1) did not receive DEI funding,
(2) were selected by the research team to be part of the comparison group, and (3) agreed to
implement the survey. The survey itself was conducted from June 9, 2017, through March 29,
2019, in English and Spanish. Staff members from these AJCs determined study eligibility and
asked eligible individuals if they were interested in completing a short survey. After signing up
online at the AJC, respondents had the option to complete the survey over the web or over the
phone with an interviewer at a later time. A gift card of $15 was provided to each respondent
who completed the survey.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
176
A total of 176 individuals completed the survey during this time frame, and their
responses are shown in Exhibit 7.
66
Over a quarter of the respondents indicated that they were
currently employed, while just under 49 percent of respondents indicated that their disability kept
them from finding a job. Nearly a quarter indicated that their disability affected their ability to go
shopping, which appeared to be the activity most affected by a disability. A smaller portion of
respondents indicated that their disability affected their ability to prepare their own meals, drive,
or dress themselves. Over 44 percent of respondents indicated that they took care of another
family member, while just under 30 percent indicated that they needed special reminders to
attend to their daily activities or chores.
Exhibit 7: Survey Characteristics of Individuals in Comparison Sites
Replied “Yes”
Questions
Current Employment
Are you currently employed?
27.8%
Did your disability keep you from finding a job?
48.8%
Activities of Daily Living
Does your disability affect your ability to prepare your own meals?
17.0%
Does your disability affect your ability to go shopping?
23.3%
Does your disability affect your ability to drive a car?
14.8%
Does your disability affect your ability to dress?
9.1%
Do you take care of anyone else such as a spouse, children, or grandchildren?
44.3%
Do you need any special reminders to attend to your daily activities or chores?
29.6%
Disabilities
67
Attention deficit or hyperactivity disorders
10.2%
Blindness or low vision
4.0%
Brain injuries
10.2%
Deaf or hard-of-hearing
8.0%
Learning disabilities
16.5%
Medical disabilities
25.6%
Physical disabilities
36.9%
Psychiatric disabilities
25.6%
Speech and language disabilities
2.8%
Developmental and intellectual disabilities
2.3%
Autism
1.1%
Number of observations
176
Source: Abt Associates survey on activities of daily living and disability type.
When asked about disability type, the most common among the respondents was some
form of physical disability. This was followed by medical and psychiatric disabilities, with over
a quarter of the respondents indicating that they had either disability. This was followed by
66
Activities of daily living and disability type response categories were not mutually exclusive.
67
As shared at some point with the point of contact when receiving services at the AJC.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
177
learning disabilities, brain injuries, and attention deficit or hyperactivity disorders, in which at
least 10 percent of the respondents indicated that they had some form of these disabilities.
D. Impact Estimates for Matched Comparison Group Analysis
The comparison group QED examined the overall impact of the Round 5 DEI
interventions on individual-level outcomes. To measure the overall impact of DEI, we used a
treatment group that consisted of the grantee WDAs and a comparison group of demographically
similar WDAs that were within the same state. We defined treatment individuals as those who
self-disclosed a disability and enrolled in one of the treatment WDAs. The sample in this
analysis only consisted of individuals with a disability who enrolled at an AJC after April 1,
2015, for Round 5 or after April 1, 2016, for Round 6.
Individuals from Alaska and South Dakota were dropped from this analysis, as each state
was comprised of a single WDA. Individuals from Illinois and California were also excluded in
this analysis due to issues with setting up appropriate comparison conditions in these states. Data
from Massachusetts, Hawaii, and Washington State were unavailable for both the QED and SITS
analyses.
The results indicate that there was no statistically significant impact of the DEI on the
three outcomes once we control for individuals’ characteristics. The coefficients in the top row
of Exhibit 8 show the regression results from the comparison group QED analysis. Impact
estimates across each of the three outcomes are displayed on the top row of results.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
178
Exhibit 8: DEI Comparison Group QED Impact Results and Control Variable Coefficients
— Unmatched
Employed first
quarter after AJC
exit
Retained
employment three
quarters after AJC
exit
Wages in the second
and third quarters
after AJC exit ($)
VARIABLES
Treatment WDA Enrollment (Impact)
-0.003
-0.009
-36.014
(0.009)
(0.009)
(191.327)
Female
0.006
0.001
-205.533**
(0.005)
(0.004)
(98.351)
Hispanic
0.027***
0.009
-122.609
(0.008)
(0.007)
(159.565)
Black
-0.001
-0.011
-544.673***
(0.008)
(0.007)
(158.464)
Asian
-0.012
-0.002
132.503
(0.016)
(0.014)
(321.853)
White
0.008
0.013**
19.672
(0.007)
(0.006)
(134.840)
Veteran
0.001
-0.003
816.528***
(0.008)
(0.007)
(150.130)
Physical disability
0.013
0.001
19.427
(0.009)
(0.008)
(163.922)
Mental disability
0.069***
0.040***
355.730**
(0.009)
(0.008)
(170.366)
Physical and mental disability
0.006
0.004
-209.271
(0.016)
(0.014)
(298.214)
Previously employed
0.158***
0.103***
2,078.546***
(0.007)
(0.006)
(131.769)
Attended college
0.036***
0.017***
1,429.792***
(0.006)
(0.005)
(112.775)
Low-income status
0.077***
0.043***
-191.087
(0.009)
(0.007)
(152.356)
Received any public assistance
-0.021**
-0.011
386.583**
(0.009)
(0.008)
(158.686)
Homeless
-0.084***
-0.036**
-1,241.746***
(0.020)
(0.017)
(367.438)
Criminal offender
-0.002
-0.007
36.483
(0.018)
(0.015)
(310.997)
Limited English proficiency
0.027
0.030
279.511
(0.034)
(0.029)
(585.998)
Single parent
-0.047***
-0.020*
-155.579
(0.014)
(0.012)
(241.753)
Number of observations
20,969
20,969
17,118
Source: Rounds 5 and 6 WIA data.
Standard errors in parentheses.
Two-tailed t-tests used for statistical significance.
*** p<0.01, ** p<0.05, * p<0.10
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
179
Veteran status was associated with positive gains in wages, and previous employment at
the time of AJC enrollment had a strong positive association with all three outcomes. Some time
spent in college was also positively associated with all three outcomes. Low-income status was
positively associated with both employment and retention. However, having received any public
assistance was negatively associated with employment, while homelessness was negatively
associated with all three outcomes, with statistical significance at either the 5 or 1 percent levels.
Being a single parent had a negative association with employment at the 1 percent level.
E. Overall Impact of DEI, with Matching
In the next set of regressions, we used the same regression framework to determine the
overall impact of the DEI on outcomes, but we incorporated a matching strategy between the
treatment and comparison groups. For every individual in the treatment group, one individual in
the comparison group who matched most closely according to the information in WIA was kept
in the new matched sample. Because there were more treatment individuals than comparison
individuals in the sample, approximately 15 percent of the treatment group was dropped from the
original sample, leaving behind only individuals who were matched well with participants in
their comparison group counterpart. The matching strategy used a logit regression and performed
a 1-to-1 nearest neighbor match without replacement, matching on the logarithm of the odds
ratio of the propensity score.
68
Exhibit 9 shows that after the matching process, there were statistically significant
differences in the first two outcomes, with the treatment group having higher employment and
retention outcomes than the comparison group. This indicates that prior to controlling for
individuals’ characteristics by using a regression analysis, the raw outcomes indicate that the
treatment group fared better in all three outcomes.
Only two variables out of 17 total variables were now statistically significantly different
across the treatment and comparison groups, indicating appropriate balance as a result of the
matching process. This imbalance is within the realm of random chance, and we are confident
that the matched treatment and comparison groups were similar to each other.
68
Guidance on 1-to-1 matching is provided in Ho, D., Imai, K., King, G., & Stuart, E. A. (2011). MatchIt:
Nonparametric Preprocessing for Parametric Causal Inference. Journal of Statistical Software, 42(8). For guidance
on the addition of covariates using a theory-driven approach, see Thoemmes, F. J., & Kim, E. S. (2011). A
systematic review of propensity score methods in the social sciences. Multivariate Behavioral Research, 46(1).
Lastly, technical advice on matching is provided in Caliendo, M., & Kopeinig, S. (2008). Some practical guidance
for the implementation of propensity score matching. Journal of Economic Surveys, 22(1).
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
180
Exhibit 9: DEI Comparison Group QED Raw Sample Characteristics — Matched
Treatment
Group Mean
Comparison
Group Mean
Difference
OUTCOMES AND CHARACTERISTICS
Outcomes
Employed first quarter after AJC exit
26.5%
23.5%
3.0pp***
Retained employment three quarters after AJC
14.0%
12.3%
1.7pp***
Wages in the second and third quarters after
AJC ($)
2,762.41
2,313.46
448.95***
Control Variables
Female
43.3%
43.1%
0.1pp
Hispanic
10.1%
10.4%
0.3pp
Black
20.8%
21.2%
-0.4pp
Asian
2.9%
3.4%
-0.4pp
White
61.6%
60.6%
1.0pp
Veteran
13.7%
14.6%
-0.9pp*
Physical disability
15.1%
15.6%
-0.5pp
Mental disability
11.9%
12.1%
-0.2pp
Physical and mental disability
2.6%
3.1%
-0.5pp*
Previously employed
15.0%
15.6%
-0.6pp
Attended college
39.9%
40.2%
-0.3pp
Low-income status
25.5%
25.6%
-0.1pp
Received any public assistance
12.5%
12.5%
0.0pp
Homeless
1.3%
1.6%
-0.3pp*
Criminal offender
2.2%
2.6%
-0.4pp*
Limited English proficiency
0.4%
0.9%
-0.5pp***
Single parent
3.7%
4.4%
-0.7pp**
Number of observations
7,948
8,193
Source: Rounds 5 and 6 WIA data.
Two-tailed t-tests used for significance.
*** p<0.01, ** p<0.05, * p<0.10
The impact of the DEI intervention was again represented by the WDA-level parameter
estimate, “being enrolled in a treatment WDA.” The parameter estimate quantified the difference
in the participant outcome for treatment WDAs compared to the outcome for “business-as-usual”
comparison WDAs.
We found that there was no impact of being in a treatment WDA on all three outcomes.
The matched results in the top row of Exhibit 10 largely mirror the unmatched findings from
Exhibit 8, with all three coefficients being close to zero and not statistically significant.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
181
Exhibit 10: DEI Comparison Group QED Impact Results and Control Variable
Coefficients — Matched
Employed first
quarter after AJC
exit
Retained
employment three
quarters after AJC
exit
Wages in the second
and third quarters
after AJC exit ($)
VARIABLES
Treatment WDA Enrollment (Impact)
-0.002
-0.007
33.038
(0.010)
(0.009)
(225.707)
Female
0.010*
0.002
-115.334
(0.006)
(0.005)
(121.965)
Hispanic
0.028***
0.000
-327.788
(0.010)
(0.008)
(210.912)
Black
-0.004
-0.016**
-743.863***
(0.009)
(0.008)
(193.134)
Asian
-0.015
-0.004
83.109
(0.017)
(0.014)
(364.056)
White
0.006
0.009
-79.461
(0.008)
(0.007)
(166.620)
Veteran
0.003
0.004
905.153***
(0.009)
(0.007)
(177.452)
Physical disability
0.018*
0.004
52.112
(0.010)
(0.009)
(198.249)
Mental disability
0.074***
0.050***
402.742*
(0.011)
(0.010)
(221.737)
Physical and mental disability
0.004
0.007
-93.127
(0.018)
(0.015)
(348.568)
Previously employed
0.175***
0.107***
2,430.757***
(0.008)
(0.007)
(161.012)
Attended college
0.030***
0.015***
1,440.799***
(0.007)
(0.006)
(135.318)
Low-income status
0.095***
0.051***
-6.060
(0.011)
(0.009)
(200.441)
Received any public assistance
-0.021*
-0.014
313.355
(0.011)
(0.009)
(207.198)
Homeless
-0.091***
-0.043**
-1,671.314***
(0.025)
(0.021)
(483.158)
Criminal offender
-0.007
-0.003
97.598
(0.020)
(0.017)
(365.780)
Limited English proficiency
0.032
0.034
417.043
(0.036)
(0.031)
(657.577)
Single parent
-0.049***
-0.020
-124.348
(0.015)
(0.013)
(279.105)
Number of observations
16,141
16,141
12,879
Source: Rounds 5 and 6 WIA data.
Standard errors in parentheses.
Two-tailed t-tests used for significance.
*** p<0.01, ** p<0.05, * p<0.10
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
182
The coefficients on the control variables paint a similar story to the findings from the
non-matched analysis, with positive outcome associations for individuals who were previously
employed and went to college. Homelessness was again negatively associated with all three
outcomes.
69
The impact estimates from the regression results will differ from the raw differences in
means between the treatment and comparison groups, as shown in Exhibits 9 and 10. This occurs
even when the matched groups are similar in demographic characteristics, as the regression
estimates incorporate fixed effects at the state and WDA levels. The differences across states and
WDAs are accounted for in the regression-adjusted fixed effects models, which are not in the
raw differences in means. As a result, when the differences in means are relatively small, it is
possible for this perceived impact to disappear altogether after regression adjustments are made.
Finally, we note that in addition to measuring outcomes using information from the WIA
and PIRL administrative data, we also analyzed outcomes using the National Directory of New
Hires (NDNH).
70
This served as a robustness check for the regression results in Exhibit 10, as
the NDNH information provides another view of employment and wage information. By using
NDNH, individuals who are living and working across state lines are more accurately captured
using these data, as compared to other administrative data sources. The mean differences on all
three outcomes between treatment and comparison groups were not statistically significant,
providing more evidence against a consistent and detectable impact of DEI on outcomes.
F. Impact Estimates for Short Interrupted Time Series (SITS) Analysis
In our evaluation, we used a SITS for all grantee sites to measure a baseline impact
estimate of DEI on employment outcomes. We accomplished this by comparing the pre-
intervention time trend on participants’ outcomes to post-intervention levels. For states with a
comparison group, we used both SITS and a QED. SITS was used to measure the longitudinal
impact of DEI before and after the treatment sites, while the QED was used as a comparison
measure between the treatment and comparison sites. For states without a comparison group,
SITS was the sole DEI impact analysis. In this analysis, only three states were excluded due to
the unavailability of data from each state: Massachusetts, Washington, and Hawaii.
Prior to conducting the regression analysis, we completed a descriptive analysis that
determined average outcomes as well as average characteristics of the analysis sample used for
SITS. In Exhibit 11, we see that about 29 percent of jobseekers were employed in the first
quarter after AJC exit, and 18 percent retained employment three quarters after exit. Average
wages in the second and third quarters after exit were about $2,775. Less than half of the sample
was female, with the majority of the sample being white. More individuals indicated that they
69
All variables that were used in the regression analysis are listed in Exhibit 10.
70
The NDNH analysis used a sample of 9.769 individuals and calculated the same variables as measured in the WIA
and PIRL data. The data only include employment covered by Unemployment Insurance, and excludes off-the-
books or informal work. All differences in outcomes between the treatment and comparison groups had p-values of
over 0.35, indicating that the differences were not close to being statistically significant at even the 10 percent level.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
183
had a mental disability as compared to a physical disability, more than a third had attended some
college, and about a third indicated that they were low income.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
184
Exhibit 11: SITS Sample Characteristics (Individuals in Treatment WDAs, Pre- and
Post-DEI)
Mean
OUTCOMES AND CHARACTERISTICS
Outcomes
Employed first quarter after AJC exit
29.2%
Retained employment three quarters after AJC exit
17.9%
Wages in the second and third quarters after AJC ($)
2,775.14
Control Variables
Female
48.2%
Hispanic
11.8%
Black
18.8%
Asian
2.2%
White
63.2%
Veteran
12.0%
Physical disability
9.0%
Mental disability
14.3%
Physical and mental disability
2.0%
Previously employed
14.4%
Attended college
34.9%
Low-income status
33.1%
Received any public assistance
18.0%
Homeless
1.8%
Criminal offender
1.6%
Limited English proficiency
0.2%
Single parent
2.3%
Number of observations
20,538
Source: Rounds 5 and 6 WIA data.
SITS necessitates the use of multiple pre-intervention data points to form a linear
counterfactual trend of the outcome variables by estimating the pre-intervention trend
information into the post-intervention period. With the abundance of information in quarters of
enrollment in the WIA data, we used five quarters of individuals’ data prior to the start of DEI on
April 1, 2015, for Round 5, and 10 quarters prior to the start of DEI on April 1, 2016, for Round
6. For information post-treatment, we used data from 10 quarters after the start of DEI for Round
5 and 5 quarters after the start of DEI for Round 6. The impact of the intervention on outcomes is
represented by a departure from this counterfactual on multiple post-intervention data points.
The parameter estimate quantified the difference in the participant outcome for
individuals in DEI-funded AJCs who enrolled after the start of DEI compared to the outcome for
individuals in those same AJCs who enrolled prior to the start of DEI. The SITS results indicate
that while there was no DEI impact on first-quarter employment, there was a positive impact on
employment retention for those who were employed. The results in the top row of Exhibit 12
show that individuals who enrolled in a DEI-funded AJC after DEI implementation were 11.4
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
185
percentage points more likely to retain their employment and earned over $900 more when
compared to individuals who enrolled in one of these same AJCs prior to that date.
71
71
The sample sizes for the first two columns are the same since all individuals in our analysis had outcome
information for three quarters after they exited their AJC.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
186
Exhibit 12: DEI SITS Impact Results and Control Variable Coefficients
VARIABLES
Employed first
quarter after AJC
exit
Retained employment
three quarters after
AJC exit
Wages in the second
and third quarters
after AJC exit ($)
3 Post-Round 5 Enrollment (Impact)
0.021*
0.114***
937.490***
(0.011)
(0.010)
(191.278)
Time (quarters)
-0.012***
-0.033***
-198.770***
(0.001)
(0.001)
(26.509)
Female
0.001
-0.009*
-358.955***
(0.006)
(0.005)
(103.259)
Hispanic
0.029***
0.014*
93.128
(0.009)
(0.008)
(161.838)
Black
-0.003
-0.023***
-513.690***
(0.009)
(0.008)
(164.768)
Asian
-0.018
-0.023
261.859
(0.019)
(0.017)
(348.607)
White
0.010
0.000
96.728
(0.008)
(0.007)
(136.637)
Veteran
-0.013
-0.009
1,059.565***
(0.009)
(0.008)
(159.975)
Physical disability
0.041***
0.019*
356.785*
(0.012)
(0.011)
(208.138)
Mental disability
0.094***
0.050***
447.886**
(0.012)
(0.010)
(205.945)
Physical and mental disability
0.026
-0.007
-269.522
(0.021)
(0.018)
(363.083)
Previously employed
0.187***
0.136***
2,185.864***
(0.008)
(0.007)
(139.735)
Attended college
0.085***
0.041***
1,871.019***
(0.006)
(0.006)
(117.126)
Low-income status
0.053***
0.046***
-464.344***
(0.009)
(0.008)
(161.049)
Received any public assistance
-0.027***
-0.030***
450.980***
(0.010)
(0.009)
(173.069)
Homeless
-0.058***
-0.048***
-873.784**
(0.021)
(0.018)
(366.486)
Criminal offender
0.068***
0.028
712.454*
(0.023)
(0.020)
(384.120)
Limited English proficiency
0.057
0.033
-66.681
(0.055)
(0.048)
(906.301)
Single parent
0.017
0.058***
703.613**
(0.019)
(0.017)
(320.045)
Number of observations
20,538
20,538
17,685
Source: Rounds 5 and 6 WIA data.
Standard errors in parentheses.
Two-tailed t-tests used for statistical significance.
*** p<0.01, ** p<0.05, * p<0.10
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
187
There was a strong positive association between all three outcomes and previous
employment at the time of AJC enrollment. College attendance was also positively associated
with all three outcomes. Low-income status was positively associated to the employment and
retention outcomes at the 1 percent level, but negatively associated with wages. Receiving any
type of public assistance was negatively associated with the employment and retention outcomes,
but positively associated with wages. Once again, homelessness was strongly negatively
associated with all three outcomes. However, in this analysis, single parents were more likely to
retain employment and earn higher wages.
Although these findings suggest positive DEI impacts on some outcomes, it is important
to keep in mind the contrasts in the SITS analysis when compared to the QED analysis. While
the previous QED analysis compared individuals in treatment AJCs to individuals in comparison
AJCs after the implementation of DEI, the SITS analysis compared individuals in treatment
AJCs after implementation to individuals in those same AJCs prior to implementation. The SITS
analysis also included states that were not in the QED analysis, such as California and Illinois.
As the comparison groups and the composition of those groups were both different, we would
expect some difference in impacts across the two analyses. The SITS findings suggest that within
DEI-funded AJCs, there may have existed impacts in employment retention and earnings that
can be attributed to the appearance of DEI programming. But, these findings are still
inconclusive given the robust QED findings that show that there were no impacts on outcomes.
G. Impact of Career Pathways
A second comparison group QED examined the impact of the specific CP interventions
on participant-level outcomes. Because all grantees had a CP component as part of their DEI
interventions, the sample consisted of individuals with a disability who enrolled in an AJC after
April 1, 2015, for Round 5 or after April 1, 2016, for Round 6. Per the information provided by
the AJCs through the WIA data, we define CP enrollment as having enrolled in one of the
following services: OJT, skill upgrading, customized training, other occupational skills training,
and apprenticeship training.
The demographic characteristics indicated that CP enrollees were much more likely to be
employed in the first quarter after AJC exit, retain employment for three quarters after exit, and
earn greater wages compared to their non-enrollee counterparts in the same DEI-funded AJCs.
Exhibit 13 shows the average outcomes and characteristics of the CP enrollees (treatment group)
and the CP non-enrollees (comparison group). Prior to matching, we found that there were large
differences across the two groups. These differences were most pronounced in disability types,
low-income status, public assistance receipt, and having been a criminal offender.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
188
Exhibit 13: Career Pathways Comparison Group QED Raw Sample Characteristics —
Unmatched
Treatment
Group Mean
Comparison
Group Mean
Difference
OUTCOMES AND CHARACTERISTICS
Outcomes
Employed first quarter after AJC exit
42.3%
26.3%
16.0pp***
Retained employment three quarters after AJC
23.8%
14.5%
9.3pp***
Wages in the second and third quarters after
AJC ($)
3,862.77
2,352.26
1,510.51***
Control Variables
Female
40.5%
44.4%
-3.9pp*
Hispanic
12.7%
12.5%
0.2pp
Black
26.1%
19.4%
6.7pp***
Asian
2.5%
2.1%
0.4pp
White
61.0%
62.3%
-1.4pp
Veteran
16.7%
11.7%
5.0pp***
Physical disability
16.0%
11.7%
4.4pp**
Mental disability
33.4%
16.6%
16.8pp***
Physical and mental disability
2.6%
2.3%
0.3pp
Previously employed
15.4%
14.8%
0.6pp
Attended college
31.2%
35.5%
-4.2pp**
Low-income status
76.9%
32.1%
44.8pp***
Received any public assistance
35.9%
14.2%
21.7pp***
Homeless
5.0%
1.5%
3.5pp***
Criminal offender
10.7%
1.5%
9.3pp***
Limited English proficiency
1.3%
0.3%
1.0pp***
Single parent
10.9%
2.6%
8.3pp***
Number of observations
605
12,171
Source: Rounds 5 and 6 WIA data.
Two-tailed t-tests used for statistical significance.
*** p<0.01, ** p<0.05, * p<0.10
Upon running the regression analysis, the results in the top row of Exhibit 14 indicate that
CP enrollment was positively associated with all three outcomes at the 1 percent level of
statistical significance. Individuals who were enrolled in CP were over 13 percentage points
more likely to be employed, 5.5 percentage points more likely to retain employment, and earned
over $1,840 more than individuals over two quarters when compared to those who did not enroll
in Career Pathways.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
189
Exhibit 14: Career Pathways Comparison Group QED Impact Results and Control
Variable Coefficients — Unmatched
Employed first
quarter after AJC
exit
Retained
employment three
quarters after AJC
exit
Wages in the second
and third quarters
after AJC exit ($)
VARIABLES
Career Pathways Enrollment (Impact)
0.134***
0.055***
1,840.777***
(0.016)
(0.014)
(281.810)
Female
-0.001
-0.004
-295.933**
(0.007)
(0.006)
(122.564)
Hispanic
0.028***
0.008
-21.885
(0.010)
(0.009)
(193.245)
Black
-0.001
-0.018*
-350.960*
(0.011)
(0.009)
(200.810)
Asian
-0.017
-0.013
379.815
(0.023)
(0.020)
(436.406)
White
0.013
0.009
231.884
(0.009)
(0.008)
(169.519)
Veteran
-0.010
-0.009
730.320***
(0.011)
(0.009)
(194.981)
Physical disability
0.018
0.001
128.877
(0.012)
(0.011)
(213.907)
Mental disability
0.061***
0.031***
136.587
(0.012)
(0.011)
(212.517)
Physical and mental disability
0.019
-0.006
-277.963
(0.022)
(0.019)
(397.807)
Previously employed
0.156***
0.101***
1,881.128***
(0.009)
(0.008)
(167.623)
Attended college
0.050***
0.018***
1,596.531***
(0.008)
(0.007)
(144.539)
Low-income status
0.047***
0.022**
-686.532***
(0.011)
(0.009)
(187.587)
Received any public assistance
-0.028**
-0.012
473.607**
(0.011)
(0.010)
(192.392)
Homeless
-0.109***
-0.060***
-1,238.887***
(0.026)
(0.022)
(453.290)
Criminal offender
0.005
-0.013
-17.413
(0.024)
(0.021)
(418.221)
Limited English proficiency
0.066
0.059
246.755
(0.057)
(0.049)
(939.703)
Single parent
-0.058***
-0.026
-324.099
(0.019)
(0.017)
(328.436)
Number of observations
12,776
12,776
Source: Rounds 5 and 6 WIA data.
Standard errors in parentheses.
Two-tailed t-tests used for statistical significance.
*** p<0.01, ** p<0.05, * p<0.10
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
190
The coefficients on the control variables indicate that veteran status was again positively
associated with wages among this sample of DEI-funded AJC enrollees. There were positive
associations with previous employment and college attendance on all three outcomes at the
1 percent level of statistical significance. Low-income status was positively associated with
employment and retention outcomes but negatively associated with wages.
Receiving any public assistance was negatively associated with employment and
retention outcomes, while homelessness was strongly negatively associated with all three
outcomes. Limited English proficiency was strongly positively associated with employment and
retention outcomes, while being a single parent was negatively associated with only
employment.
H. Impact of Career Pathways, with Matching
In the final set of regressions, we incorporated a matching strategy between the treatment
(CP enrollees) and comparison (CP non-enrollees) groups. Because there were many more
comparison group individuals for every treatment group individual within the AJCs, we
incorporated a 1:2 nearest neighbor match. For every individual in the treatment group, two
individuals in the comparison group who matched most closely according to the information in
WIA were kept in the new matched sample. The matching strategy used a logit regression and
performed a nearest neighbor match without replacement, again matching on the logarithm of the
odds ratio of the propensity score.
Exhibit 15 shows that all variables were not statistically significant at the 5 percent level
across the treatment and comparison groups, indicating a high level of balance. After matching,
we find that there are statistically significant differences in all three outcomes between the
treatment and comparison groups. This indicates that prior to controlling for individuals’
characteristics using a regression analysis, individuals who enrolled in CP programs at DEI-
funded AJCs fared better than matched individuals who did not enroll in these programs.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
191
Exhibit 15: Career Pathways Comparison Group QED Raw Sample Characteristics —
Matched
Treatment
Group Mean
Comparison
Group Mean
Difference
OUTCOMES AND CHARACTERISTICS
Outcomes
Employed first quarter after AJC exit
42.3%
29.4%
12.9pp***
Retained employment three quarters
after AJC
23.8%
17.6%
6.2pp**
Wages in the second and third quarters
after AJC ($)
3,862.77
2,438.78
1,423.99***
Control Variables
Female
40.5%
41.8%
-1.3pp
Hispanic
12.7%
10.8%
1.9pp
Black
26.1%
25.8%
0.3pp
Asian
2.5%
1.7%
0.8pp
White
61.0%
64.7%
-3.7pp
Veteran
16.7%
15.2%
1.5pp
Physical disability
16.0%
18.2%
-2.1pp
Mental disability
33.4%
36.7%
-3.3pp
Physical and mental disability
2.6%
3.1%
-0.4pp
Previously employed
15.4%
16.6%
-1.3pp
Attended college
31.2%
29.4%
1.8pp
Low-income status
76.9%
78.4%
-1.5pp
Received any public assistance
35.9%
36.4%
-0.5pp
Homeless
5.0%
3.3%
1.6pp*
Criminal offender
10.7%
9.5%
1.2pp
Limited English proficiency
1.3%
0.6%
0.7pp
Single parent
10.9%
10.5%
0.4pp
Number of observations
605
1,166
Source: Rounds 5 and 6 WIA data.
Two-tailed t-tests used for statistical
significance.
*** p<0.01, ** p<0.05, * p<0.10
The impact of the CP intervention is represented by the individual-level parameter
estimate, “being enrolled in a Career Pathways program.” The parameter estimate quantifies the
difference in the participant outcome for individuals with disabilities who entered CP training
compared to the outcome for those who did not.
The results in the top row of Exhibit 16 indicate consistently positive impacts of CP
enrollment on all three outcomes at the 1 percent level of statistical significance. An individual
with a disability who was enrolled in CP was 16 percentage points more likely to be employed,
4.7 percentage points more likely to retain employment, and earned over $1,700 more over two
quarters compared to a similar individual with a disability in a treatment WDA who did not
enroll in CP.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
192
Exhibit 16: Career Pathways Comparison Group QED Impact Results and Control
Variable Coefficients — Matched
Employed first
quarter after AJC
exit
Retained
employment three
quarters after AJC
exit
Wages in the second
and third quarters
after AJC exit
VARIABLES
Career Pathways Enrollment (Impact)
0.161***
0.047**
1,701.396***
(0.021)
(0.018)
(378.122)
Female
-0.025
-0.006
822.383**
(0.021)
(0.018)
(388.467)
Hispanic
0.032
0.027
-314.186
(0.030)
(0.026)
(556.435)
Black
-0.075**
-0.017
-856.831
(0.033)
(0.029)
(617.755)
Asian
-0.039
-0.011
-977.783
(0.070)
(0.061)
(1,319.979)
White
-0.011
-0.001
72.854
(0.030)
(0.026)
(557.350)
Veteran
-0.121***
-0.054**
1,145.127**
(0.029)
(0.025)
(527.610)
Physical disability
0.086***
0.016
830.197
(0.030)
(0.026)
(550.436)
Mental disability
0.049*
-0.018
-612.759
(0.027)
(0.023)
(509.464)
Physical and mental disability
0.063
-0.013
-799.283
(0.057)
(0.049)
(1,019.651)
Previously employed
0.094***
0.087***
665.106
(0.026)
(0.023)
(466.668)
Attended college
0.025
-0.013
272.032
(0.023)
(0.020)
(424.029)
Low-income status
0.055**
0.009
-1,655.346***
(0.028)
(0.024)
(490.050)
Received any public assistance
-0.041*
-0.052***
136.222
(0.023)
(0.020)
(430.450)
Homeless
-0.157***
-0.081*
-802.113
(0.048)
(0.042)
(919.816)
Criminal offender
-0.007
-0.009
305.039
(0.033)
(0.029)
(585.008)
Limited English proficiency
0.010
-0.050
-1,145.713
(0.101)
(0.088)
(1,710.139)
Single parent
-0.108***
-0.038
-772.110
(0.032)
(0.028)
(576.071)
Number of observations
1,771
1,771
Source: Rounds 5 and 6 WIA data.
Standard errors in parentheses.
Two-tailed t-tests used for statistical significance.
*** p<0.01, ** p<0.05, * p<0.10
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
193
Among this group of individuals who enrolled in DEI-funded WDAs, the coefficients on
the control variables tell a similar story compared to the previous regression results, with a few
exceptions. Being female was now associated with positive outcomes in employment and
retention at the 5 percent level of statistical significance. Individuals who were previously
employed were more likely to have positive outcomes, but there was no relationship between
college attendance and outcomes. Low-income status was still positively associated with
employment but had a negative impact with earnings. Having received any public assistance was
negatively associated with employment retention at the 1 percent level of statistical significance.
X. FINDINGS AND DISCUSSION
The impact estimates of both DEI as a program as well as the impact of Career Pathways
enrollment within DEI-funded AJCs were conducted using WIA information on individuals with
disabilities that the states provided on a regular basis. Using information spanning multiple years
and two rounds of the program, we were able to use two non-experimental methods, a SITS and
a comparison group QED, to determine these impacts.
The findings suggest that there were no consistently positive and statistically significant
impacts of DEI on employment, employment retention, or wage outcomes once we controlled for
individuals’ characteristics using a regression analysis and used propensity score matching to
further account for differences across the treatment and comparison groups. While the QED
analysis to determine overall DEI impacts had null findings, we did find positive impacts on
employment retention and wages using a SITS analysis. As the comparison groups differed
between the QED and SITS analyses, as did the general composition of the groups due to the
inclusion of California and Illinois in the SITS analysis, the impact estimates are not directly
comparable. However, the positive estimates using SITS indicate there may have existed impacts
in employment retention and earnings that can be attributed to the appearance of DEI
programming.
There were consistently positive and statistically significant impacts of enrolling in CP
programs within DEI-funded AJCs even after a matching strategy to keep only those individuals
who were statistically similar to each other. The measurable impacts were large enough to
warrant some further exploration in the future on what combinations of CP services and training
receipt might provide the greatest returns on outcomes. This exploration could come in the form
of a principal components analysis or a clustering analysis using machine learning techniques,
which could help uncover which parts of the DEI program were more likely to lead to successful
outcomes.
The qualitative implementation analysis included the collection of information on
systems change, assessments of SDSs, and documentation of project challenges. We used
qualitative data to assess the context in which each DEI grant was implemented, as well as how
systems changes may have affected project performance. Through the collection of qualitative
data (e.g., site visits, interviews, and focus groups) and research on promising practices in
disability employment, we developed domains and indicators to measure systems changes in
WDAs. This process began with creating a USDOL-approved DEI protocol, which we used to
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
194
collect information from state officials, DEI State Leads, WDAs, DRCs, and DEI participants.
Site visits included the collection of information on grantee start-up issues, implementation of
grant requirements, and utilization of SDSs.
Analyses of site visits, telephone interviews, and focus groups provided descriptions of
participating WDAs, including partnerships and collaborative initiatives for JSWDs, the status of
existing cross-agency/organizational partnerships, and the leaders who influenced disability
employment in each WDA. In addition, data captured the social and political context of each
WDA, such as system members, existing interagency agreements, MOUs, the degree of
inclusiveness of JSWDs, and the current level of service integration. We also collected
information on the structure and organization of the DEI in each state.
The study’s qualitative research questions (Exhibit 17) investigated the staffing of DEI
grantees as well as adoption of SDSs, perceived challenges to implementation, DEI grant
requirements, TTW implementation, Partnerships and Collaborations, and the sustainability of
DEI practices. The quantitative research questions analyzed the number of customers with
disabilities served, the extent to which DEI grants affected the number and types of customers
with disabilities, and the outcomes and impacts of the DEI project.
Exhibit 17: Qualitative Research Questions
Qualitative Research Questions
WHAT STAFFING INFRASTRUCTURE DID GRANTEES DEVELOP FOR THE DEI?
HOW DID GRANTEES IMPLEMENT OPTIONAL DEI SERVICE DELIVERY STRATEGIES?
WHAT WERE THE PERCEIVED CHALLENGES AND SUCCESSES OF IMPLEMENTING TTW ACTIVITIES?
W
HAT WERE THE REQUIREMENTS OF THE
DEI?
TO WHAT EXTENT DID DEI GRANTEES COORDINATE WITH STATE VR AND OTHER PARTNERS?
HOW DID GRANTEES USE TTW REVENUES AND OTHER RESOURCES TO FUND DRCS?
T
O WHAT EXTENT WERE
DEI
ACTIVITIES SUSTAINED WHEN GRANT FUNDING EXPIRED
?
A. Potential Areas for Future Research
The goal of the DEI was to improve the employment outcomes of JSWDs through the
application of specialized SDSs and the facilitation of systems change. The impact evaluation
measured and assessed the outcomes and impact of the DEI. To provide context on the
individuals with disabilities who received services outside of DEI-funded AJCs, a survey on
employment, activities of daily living, and disability type was analyzed and reported in this
evaluation. The source of information from this analysis came in the form of a novel survey that
was administered via web and phone to individuals in comparison group AJCs who elected to
answer questions in exchange for a gift card. The comparison site survey included newly formed
data elements that were not part of WIA/PIRL data collection and, as a result, garnered a
relatively small sample of comparison and treatment group individuals. Due to the challenges of
on-the-ground data collection and limited support from WDA personnel due to their own
workloads, the comparison group survey provided useful information that was limited by a small
subset of comparison group individuals across participating WDAs. Future research that
compares treatment to comparison groups should include specific PIRL addenda that allow for
the inclusion of additional data elements that are required for certain groups such as DEI
participants, WDA staff, and/or youth with disabilities.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
195
B. DEI Evaluation Logic Model
The DEI Round 5 through Round 6 logic model (Exhibit 18) conveys an understanding of
the relationships among the inputs, outputs, and expected outcomes of the DEI evaluation. It was
developed in five phases. As we collected information through site visits, interviews, and focus
groups; program documentation; and WIOA data, changes to the logic model were made to
illustrate the program’s design and its evolution for the DEI Round 5 and Round 6 grantees. As
the DEI project matured, project developers provided detailed explanations of the design,
purpose, and utilization of each SDS, as well as training designed to help DRCs and other DEI
stakeholders understand the relationships among various DEI resources and the systems changes
that were expected to occur during the project period.
The DEI logic model specifies relationships among situations/priorities, inputs, outputs,
and outcomes at the JSWD and systems levels as well as the challenges and facilitators of
program implementation. The logic model has five components:
1. Inputs: Investments in the DEI. These include staff knowledge, experience, and skills;
T/TA; DEI grant resources, including grant-funded positions; SDSs; partnerships;
providers; TTW and EN activities; and WIOA services and data.
2. Outputs: Products and services provided to JSWDs. Individual-level outputs include
SDSs, grant-funded positions (e.g., DEI State Leads and DRCs), TTW implementation,
and internal staff trainings. Systems-level outputs include WDA engagement, outreach to
JSWDs and employers, and partnerships, both internal and external, including WIOA-
mandated partners, child care, transportation, housing, WIPA services, and CP.
3. Outcomes: Changes in the circumstances (e.g., employment, wages, and job retention) of
JSWDs that can be attributed to the DEI. Individual-level outcomes include increases in
the number of JSWDs served by AJCs, the employment rate of JSWDs, the disability
self-disclosure rate, SSI/SSDI beneficiary enrollment rate, SSI/SSDI beneficiary public
assistance termination rate that is replaced by employment, CP training completion, and
wages. Systems-level outcomes include increases in partner, provider, and employer
engagement, as well as systems change, including integration of resources through
Blending and Braiding Funds and Partnerships and Collaborations.
4. Program Impact: A primary goal of the DEI evaluation is to identify the impact of DEI
activities on employment-related outcomes. To accomplish this, we used a clustered
randomized selection procedure to assign WDAs to serve as treatment sites or as pilot
sites.
72
72
Social Dynamics collected the same information that ETA received on a quarterly basis from each DEI grantee. It
was decided in 2010 by USDOL that the DEI evaluation would focus only on customers who received WIOA
intensive and/or training services and self-disclosed a disability.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
196
5. Challenges and Facilitators: These are either exogenous or endogenous factors that affect
program inputs, outputs, and outcomes. Exogenous factors are external to the DEI. For
example, a state hiring freeze that delays the hiring of DEI State Leads and/or DRCs and
the discontinuation of the WIPA program are exogenous factors that may have affected
program outcomes. Another exogenous factor is whether a VR agency is under Order of
Selection in a particular state, which may result in constraints on engaging VR as a DEI
partner. Other exogenous factors include WDA unemployment rates, types of industries
and jobs available in each WDA, quality of local transportation systems and availability
of other support services, population density, local politics, and institutions.
Conversely, endogenous factors are dependent variables that change due to a relationship
within the model. One of the most important endogenous factors is the capacity of
participating WDAs to implement all of the DEI requirements. Capacity is defined as
having the appropriate staff expertise, coordination, and management systems in place at
the time of the commencement of the grant. Another critical endogenous factor is DEI
State Lead and/or DRC expertise and turnover of these positions. These factors may
generate ripple effects throughout the DEI that lead to staff retraining, reestablishing
partnerships, dealing with the new staff “learning curve,” and JSWD attrition during the
final two quarters of the DEI.
The program theory for the DEI is based on the assumption that each grantee utilized two
grant-funded positions (e.g., DEI State Lead and DRC) and that selected SDSs were implemented
according to the way they were designed by the program developers with some variation to adapt
SDSs for JSWDs. For example, IRTs are designed to provide “diversified service systems,
coordinating services and leveraging funding in order to meet the needs of an individual jobseeker
with a disability.”
73
Likewise, WBL Opportunities were implemented “in order to develop”
jobseeker “aspirations, make informed choices about careers,” and engage in “training designed to
improve job-seeking and workplace skills.”
74
The expectation was that access to DEI State Leads
and DRCs, the two positions that were funded by the DEI, combined with the implementation of
SDSs, would improve the employment, job retention, and earnings of DEI participants.
73
WorkforceGPS. (2017). Integrated resource team FAQs. Retrieved from
https://disability.workforcegps.org/resources/2016/04/13/14/02/Integrated_Resource_Team_FAQs
74
U.S. Department of Labor, Office of Disability Employment Policy. (n.d.). Career preparation and work-based
learning experiences. Retrieved from https://www.dol.gov/odep/categories/youth/career.htm
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
197
Exhibit 18: DEI Logic Model
Within the context of the DEI, systems change was an adjustment in the way WDAs
made decisions about their policies, programs, and allocation of resources. The purpose of
systems change was to maximize each WDA’s ability to offer JSWD services that improved the
access and availability of job training and employment services by resolving systemic
inefficiencies, which may have included identifying JSWDs and enrolling them in training that
led to employment. The DEI Round 5 through Round 6 systems change analysis focused on the
implementation of DEI grant requirements in addition to the structure of the system (e.g.,
WDAs), including changes in its individual and organizational members and leadership.
Social Dynamics also observed the daily operations of AJCs and collected relevant
documentation on intake activities, service coordination activities, and the availability of and
access to employment-related services and training. Observations of state and local meetings,
WDA, and AJC levels also informed the systems in each WDA rated on a four-point ordinal
scale that includes eight domains. The objectives of the Systems Change Coding Scheme were to
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
198
provide a system for coding interviews and focus groups, beginning with a review of the
indicator descriptions associated with each change component of the DEI evaluation.
75
The analysis of DEI implementation included qualitative data analysis designed to
expand the breadth and nuance of enquiry. The qualitative research component focused on how
WDAs affected systems change.
76
Qualitative data was coded by researchers and aggregated
across stakeholder categories. Systems change was determined by the eight sub-domains by
searching for information in interview and focus group transcripts.
Approach:
• Create a system for coding DEI interviews by hand based on the attached SCCS;
• Develop a process that can be easily translated into MAXQDA for increased productivity;
and
• Ensure that the process has inter-rater/-coder reliability.
Data:
Systems Change Coding Scheme (SCCS)
Final interview protocols
Interview transcripts
SCCS
Protocol
Transcripts
SCCS Transcript Coding Instructions
1) Review all indicator descriptions associated with each sub-domain; indicator descriptions
are to the right of each sub-domain. You will find that rows 1–4 and columns A–B are
frozen to improve navigation of the SCCS. The indicators should be attached in columnar
format by domain for easier reading and scrolling.
2) Search for and find information in the transcripts. Highlight and color-code the
information as indicated for the domain. Only highlight portions with fewer than 10
words such that it can be coded as a search term for MAXQDA.
75
Previous USDOL programs designed to improve the employment outcomes of individuals with disabilities
through systems change included ETA’s Work Incentive Grants, the Disability Program Navigator initiative, and
employment service models, such as ODEP’s Customized Employment, Workforce Action (Olmstead), START-
UP USA (Self-Employment), and State Intermediary Youth grants.
76
Kaufman, N. J., Castrucci, B. C., Pearsol, J., Leider, J. P., Sellers, K., Kaufman, I. R., … Sprague, J. B. (2014).
Thinking beyond the silos: Emerging priorities in workforce development for state and local government public
health agencies. Journal of Public Health Management & Practice, 20(6): 557–565.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
199
3) Use the color scheme below to color-code text that addresses each of the eight domains:
Domain Color Code
1. Capacity
2. Coordination/Integration
3. Customer Choice
4. Employer Support
5. Service Delivery Strategies
6. Dissemination
7. Universal Design
8. Sustainability
Select color set from the Home Tab in MS Word using the paint bucket as presented in
the image below.
Simply mark the text with you cursor, go to Home, right click paint bucket on the button,
and choose the appropriate color for the domain.
4) Insert comments to identify the sub-domains that you are coding after highlighting the
color associated with the broader domain. For example, if you have highlighted in
orange the text associated with sustainability, and those highlighted portions refer to sub-
domain “8A,” then insert a comment on that portion of text with the alphanumeric “8A”
only. If, however, the segment of the interview refers to more than one domain, please
identify all relevant sub domains in the same comment section.
5) Highlight a segment of text and code as an indicator, copy the text, and place it into the
associated row of the SCCS spreadsheet in column “Z”. This step allows you to compile
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
200
quoted language from interview transcripts that can be used to design a coding system for
MAXQDA that is informed by the data.
6) Do not insert comments other than naming the sub-domain in the color-coded transcript.
If you have comments on the transcript itself, compile all of your concerns in a separate
document starting with “On page #, lines #–# of [Interviewee’s] transcript” and provide
the appropriate information.
7) Numerous instances of the same DEI element may be distracting as they often pile up on
the document. If you do not want to see them while reading a transcript, go to the review
tab above MS Word, select the drop-down menu for Show Markup, and deselect
“Comments.”
8) After you have exhausted your search and coding of the information in the transcript that
aligns with the indicator descriptions for a particular sub-domain based on the
information from the transcripts, make a decision as to the level of implementation of the
sub-domain. You should review the coded portions for the sub-domain you identify based
on the excerpt from the SCCS below and rate what level of implementation is suggested
by the coded content.
9) Please enter your implementation ratings in merged cells in columns D–G of the SCCS.
The rating scale includes the following four levels: Not Implemented (1), Active
Planning for Implementation (2), Partially Implemented (3), and Fully Implemented (4).
The SCCS provides further guidance and criteria for your rating decisions. As shown in
the graphic above, the SCCS provides an explanation for each rating score. For example,
“Not Implemented” with a rating of 1 is described as “There is little to no evidence that
this sub-domain is being met. No effort is being made to implement this sub-domain.”
Cells D4–G4 in the SCCS provide similar guidance and criteria for each implementation
rating decision.
10) If the interview transcript suggests there is clearly “No Intention to Implement,” please
note that information in column C with “NI.” If you do not have enough information to
make an” NI” or a scored implementation rating, please note that with an “X” in column
H.
11) Start with Domain 1, “Capacity to achieve positive employment outcomes for customers
with disabilities (PWD),” and its associated sub-domains. A tab was created in Excel that
lists the domains, sub-domains, and indicators in each column, as shown below, for those
who prefer to read them in that layout.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
201
12) Code one of the eight domains at a time while working across transcripts; you will finish
in eight cycles and will likely become more adept at coding a particular domain as you
work from one transcript to another.
13) In the end you will have:
a. A series of marked-up transcripts that are color-coded to describe domains and are
noted with comments that detail the alphanumeric identifier of the sub-domain for
the portion of text you have color-coded;
b. Use an Excel workbook of completed rating sheets that report your
implementation score for each sub-domain for a given interview — one interview
per tab; and
c. An edited version of the SCCS where you have copied text that is 10 words in
length and is descriptive of each sub-domain in column Z of the SCCS [Please
save and name this file with your initials].
1. Capacity to achieve positive employment outcomes for
customers with disabilities (PWD).
1C
WDA communications are inclusive and sufficient. 4.2, 39, 40, 42
1. WDA and/or its AJC policy requires that all communication in all
media (print, audio, visual, web-based, etc.) is universally
accessible.
2. The AJC's communications, and information including Internet
websites, are accessible (e.g. use of section 508 requirements).
3. Communications from the WDA and One Stop staff use
appropriate language when discussing individuals with disabilities
/ disability issues.
4. One Stops staff in this WDA have received training on effective
strategies for communicating with PWD.
5. Materials are available in formats that account for a variety of
learning styles, and are also accessible to people who have limited
or no reading skills (e.g., pictures, videos, audio-tapes)
6. As part of its efforts to provide universal accessibility, the AJC
Center does outreach to people with disabilities, as required by
WIA regulations. For example: the AJC holds Job Fairs, school-
based events for youth, and/or disseminates information via
brochures and social media.
7. Customers perceive the AJC communications as accessible. Data
for this indicator will come from the Focus Groups.
8. AJC communications are accessible for PWD with different kinds
of disabilities.
9. AJC communications directed towards employers use common
language/methods that are approopriate to employer needs
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
202
14) The next step in the process is to bring the group of coders together to discuss your
scores. We will begin with an overall inter-rater reliability score. If the score is
determined to be reliable, no additional information is needed. If the score is determined
to be unreliable, coders will discuss the issues that led to their scores and make decisions
on how to resolve differences across coders.
15) The SCCS Rating Sheet is excerpted from the SCCS and is designed for you to record
your various implementation ratings in one workbook; indicator information is not listed in
the rating sheets. In each tab, use cell B3 to identify the interviewee(s) and their position
title (e.g., DRC, State Lead, Employment Counselor, Partner, etc.), and use cell B4 to
identify the WDA by adding the WDA name, number, and location. As you make your
ratings, please add notes and comments to describe your thinking in the column next to
your ratings labeled “Comments and rationale for ratings.” Keep in mind that one of the
next steps is to resolve inconsistency among raters. Having a shared understanding of each
coder’s reasoning for a particular rating is essential to that task.
16) Start with Domain 1, “Capacity to achieve positive employment outcomes for customers
with disabilities (PWD),” and its associated sub-domains. An MS Word document was
created that lists the domains, sub-domains, and indicators in columns.
17) After you have completed coding of an interview transcript, save the file with your initials
added and place it in the “SCCS Coding Team” folder named “Coded Interviews.” Put your
implementation ratings workbook in the folder named “Implementation Ratings (with rater
ID)” in the same location. Please update the saved workbook on the drive as you add
completed implementation ratings worksheets to the workbook.
18) In the end, for each state you will have the following: (1) A series of marked-up transcripts
that are color-coded to indicate reference to domains and noted with comments that detail
the alphanumeric identifier of the sub-domain for the portion of text you have coded.
(2) An Excel workbook of completed rating sheets that report your implementation scores
for each sub-domain for each given state interviewee. This workbook should be updated as
you progress through the individual interview ratings, which will be aggregated to describe
the WDA. (3) An edited version of the SCCS where you have copied phrases from
interviews into the SCCS (column Z); the phrases should be 10 or fewer words and
descriptive of the sub-domain next to the inserted text. Please save and name this file with
your initials. This compilation may, alternatively, be created through a shared file that will
be placed on the G-drive rather than documents created by each individual coder.
19) The next step in the process is to bring the group of coders together to discuss scores. We
will begin with an overall inter-rater reliability score on the implementation scale, fidelity
scales, and checklist. If the scores are determined to be reliable, no additional information
is needed. If the scores are determined to be unreliable, coders will discuss the issues that
led to their scores and make decisions on how to resolve differences across coders. Our
goal is 80 percent or higher reliability/agreement across three coders.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
203
Researchers collected information 3–6 months after the expiration of each grant period to
determine the extent to which grantees sustained their DEI practices; sustainability findings are
subject to change and should be revised annually. Each synopsis includes information from
grantee leaderships, AJC staff, WDA leaders, WIOA-mandated partners, staff from community-
based agencies, and individuals who enrolled in DEI.
Systems Change Coding Scheme Results
A. Systems Change Coding Scheme Results
To what extent did DEI implemented their Round 5 and Round 6 projects?
California Round 5
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
204
Kansas Round 5
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
205
Massachusetts Round 5
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
206
Minnesota Round 5
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
207
South Dakota Round 5
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
208
Alaska Round 6
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
209
Georgia Round 6
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
210
Hawaii Round 6
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
211
Iowa Round 6
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
212
Washington Round 6
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
213
B. Sustainability Findings
To what extent were DEI activities sustained when grant funding expired?
DEI grantees were asked to discuss the sustainability of their DEI project to determine
the extent to which DEI State Leads and DRCs coordinated and continued TTW and EN
implementation and training, use of SDSs, continued availability of DRCs, and the overall
sustainability of the project. We conducted 31 interviews within 6 months of the end of the grant
period with DEI State Leads and DRCs to determine if DEI Round 1 through Round 4 grantees
sustained their selected DEI practices. At the beginning of each call, we asked DEI State Leads
to confirm their selected SDSs, treatment and control WDAs, and service populations (i.e., adults
or youth). We then proceeded to discuss the sustainability of the DEI, including the continued
use of SDSs, DRCs or similar support role, TTW, and, finally, challenges and recommendations
for the project. Each DEI State Lead discussed their programs freely. Probing questions were
guided by background information collected from grantee abstracts, site visit reports, quarterly
narrative reports, and DEI quarterly report summaries.
DEI Grantees by Round
California Round 5
Service Delivery Strategies
Sustained Service Delivery Strategies
Service Delivery Strategies
Asset Development
Sustained in certain locations
Blending and Braiding Funds
Sustained
Career Pathways Training
Sustained
Customized Employment
Selected components used
Integrated Resource Teams
Sustained
Partnerships and Collaborations
77
Sustained in certain locations
DEI Services
CWIC Services
Sustained in certain locations
DRC and Traveling DRC Availability
Sustained in certain locations
Employment Networks
Sustained
Sustainability Score: 4 out of 5
Confirmed by Grantee
Illinois Round 5
Service Delivery Strategies
Sustained Service Delivery Strategies
Service Delivery Strategies
Asset Development
Sustained in certain locations
Blending and Braiding Funds
Sustained in certain locations
Career Pathways Training
Sustained
Customized Employment
Selected components used
Integrated Resource Teams
Sustained in certain locations
DEI Services
CWIC Services
Not sustained
DRC Availability
Sustained in certain locations
77
California Round 5 continued to operate both the DEI and DEA. The DEA will continue after the DEI grant
period.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
214
Employment Networks
Sustained
Partnerships and Collaboration
Sustained in certain locations
Sustainability Score: 4 out of 5
Confirmed by grantee
Kansas Round 5
Service Delivery Strategies
Sustained Service Delivery Strategies
Service Delivery Strategies
Blending and Braiding Funds
Sustained
Career Pathways Training
Sustained in certain locations
Cultural Change in WDAs
Sustained in certain locations
Customized Employment
Selected components used
DEI Services
CWIC Services
Sustained
DRC Availability
Not sustained
Employment Networks
Sustained
Integrated Resource Teams
Sustained in certain locations
Partnerships and Collaboration
Sustained in certain locations
Sustainability Score: 3 out of 5
Confirmed by grantee
Massachusetts Round 5
Service Delivery Strategies
Sustained Service Delivery Strategies
Service Delivery Strategies
Blending and Braiding Funds
Not sustained
Customized Employment
Selected components used
CWIC Services
Not sustained
DRC Availability
Not sustained
Employment Networks
Not sustained
DEI Services
Integrated Resource Teams
Sustained in Certain Locations
Health Care pathways
Sustained
Partnerships and Collaboration
Sustained in Certain Locations
Sustainability Score: 2 out of 5
No response from grantee
Minnesota Round 5
Service Delivery Strategies
Sustained Service Delivery Strategies
Service Delivery Strategies
Blending and Braiding Funds
Sustained in certain locations
Career Pathways Training
Sustained in certain locations
Cultural Change in WDAs
Sustained in certain locations
Customized Employment
Selected components used
DEI Services
CWIC
Sustained in certain locations
DRC Availability
Sustained in certain locations
Employment Networks
Sustained
Integrated Resource Teams
Sustained in certain locations
Partnerships and Collaboration
Sustained in certain locations
Universal design
Sustained in certain locations
Sustainability Score: 4 out of 5
Confirmed by grantee
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
215
South Dakota Round 5
Service Delivery Strategies
Sustained Service Delivery Strategies
Service Delivery Strategies
Blending and Braiding Funds
Sustained in certain locations
Career Pathways Training/Academic
Employment Transitions
Sustained
78
Cultural Change in WDAs
Sustained in certain locations
Customized Employment
Selected components used
DEI Services
CWIC Services
Sustained
DRC Availability
Sustained in certain locations
Employment Networks
Sustained
Integrated Resource Teams
Sustained in certain locations
Partnerships and Collaboration
Sustained in certain locations
Universal designed
Sustained in certain locations
Sustainability Score: 3 out of 5
Confirmed by Grantee
Alaska Round 6
Service Delivery Strategies
Sustained Service Delivery Strategies
Service Delivery Strategies
Blending and Braiding Funds
Sustained
Business/Employer Partnerships
Sustained
Career Pathways Training/Academic
Employment Transitions, OJT-WBL
Sustained
Customized Employment
Selected components used
Cultural Change in WDAs
Sustained
DEI Services
CWIC Services
Sustained
DRC Availability
Sustained
79
Integrated Resource Teams
Sustained in certain locations
Partnerships and Collaboration
Sustained
Universal Design
Sustained
Sustainability Score: 5 out of 5
Confirmed by grantee
Georgia Round 6
Service Delivery Strategies
Sustained Service Delivery Strategies
Service Delivery Strategies
Blending and Braiding Funds
Sustained in certain locations
Business/Employer Partnerships
Sustained in certain locations
Career Pathways Training
(agriculture, hospitality, logistics
Sustained
Customized Employment, OJT,
Customized Service Delivery
80
Selected components used
DEI Services
78
JSWDs expressed concern that some employment opportunities did not provide appropriate CP training that
would lead to increases in wages and employment status.
79
Includes individuals with mental illness, substance abuse, and intellectual and developmental disabilities.
80
Georgia Round 6 includes Discovery, customized apprenticeships, and evidence-based practices.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
216
CWIC Services
Sustained
DRC Availability
Sustained in certain locations
Employment Networks
Sustained
Integrated Resource Teams
Sustained in certain locations
Partnerships and Collaboration
81
Sustained in certain locations
Universal designed
Sustained in certain locations
Sustainability Score: 4 out of 5
Confirmed by grantee
Hawaii Round 6
Service Delivery Strategies
Sustained Service Delivery Strategies
Service Delivery Strategies
Blending and Braiding Funds
Limited in certain locations
Business/Employer Partnerships
Limited in certain locations
Career Pathways Training
(agriculture, hospitality, logistics)
Available sustained, limited use.
82
Cultural Change in WDAs
Sustained in certain locations
Customized Employment
Selected components used
DEI Services
CWIC Services
Not sustained
DRC Availability
83
Not sustained
Employment Networks
Sustained with limited use
Integrated Resource Teams
Limited use in certain locations
Partnerships and Collaboration
Limited use in certain locations
Universal designed SDS
Sustainability Score: 2 out of 5
Confirmed by grantee
Iowa Round 6
Service Delivery Strategies
Sustained Service Delivery Strategies
Service Delivery Strategies
Blending and Braiding Funds
Sustained
Business/Employer Partnerships
Sustained
Career Pathways Training/
Apprenticeships (agriculture,
hospitality, logistics)
Sustained
Cultural Change in WDAs
Sustained in certain locations
Customized Employment
Selected components used
DEI Services
CWIC Services
Sustained
DRC Availability
Sustained in certain locations
Employment Networks
Sustained
Integrated Resource Teams
Sustained
Partnerships and Collaboration
84
Sustained
81
Grantees accessed community and technical college services and support systems.
82
JSWDs expressed concern that some employment opportunities did not provide appropriate CP training that
would lead to increases in wages and employment status.
83
Hawaii Round 6 co-tasked DRCs with both Round 8 and Round 6 responsibilities. It also sustained the Round 6
DRCs when they transitioned to Round 8.
84
Community and technical college services and support systems were included in the DEI.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
217
Universal designed SDS
Sustained in certain locations
Sustainability Score: 5 out of 5
Confirmed by grantee
New York Round 6
Service Delivery Strategies
Sustained Service Delivery Strategies
Service Delivery Strategies
Blending and Braiding Funds
Sustained
Business/Employer Partnerships
Sustained in certain locations
Career Pathways
Training/Apprenticeships/WBL
(agriculture, hospitality, logistics)
Sustained
Cultural Change in WDAs
Sustained in certain locations
Customized Employment
Selected components used
DEI Strategies
CWIC Services
Sustained in certain areas
Cultural Change in WDAs
Sustained in certain locations
Employment Networks
Sustained
Integrated Resource Teams
Sustained in certain locations/Certain
components used
Partnerships and Collaboration
85
Sustained in certain locations
Universally designed SDS
Sustained in certain locations
Sustainability Score: 5 out of 5
Confirmed by grantee
Washington Round 6
Service Delivery Strategies
Sustained Service Delivery Strategies
Service Delivery Strategies
Blending and Braiding Funds
Sustained in certain locations
Business/Employer Partnerships
Sustained in certain locations
Career Pathways
Training/Apprenticeships/WBL
(agriculture, hospitality, logistics)
Sustained in certain locations
Customized Employment
Sustained in certain locations
Cultural Change in WDAs
Sustained in certain locations
DEI Services
CWIC Services
Sustained in certain locations
DRC Availability
Sustained in certain locations
Employment Networks
Sustained
Integrated Resource Teams
Sustained in certain locations
Partnerships and Collaboration
Sustained in certain locations
Universally designed SDS
Not sustained
Sustainability Score: 4 out of 5
Confirmed by grantee
85
Community and technical college services and support systems were available to youth with disabilities.
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
218
Appendix 1: American Job Center Survey
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
219
Appendix 2: DEI Round 5 and Round 6 WDAs
Grantee
DEI Round 5 - California Treatment and Comparison LWIAs
# of LWIAs
3 Treatment Sites
6170 Sacramento City/County
6090 Merced County
6280 SELACO-Southeast Los
Angeles Consortium
3 Comparison Sites
6115 Solano County
6175 San Joaquin County
6015 Long Beach (city)
Grantee
DEI Round 5 - Illinois Treatment and Comparison LWIAs
# of LWIAs
2 Treatment Sites
17035 Cook County, Balance of -
Area 7
17030 Dupage County
2 Comparison Sites
17025 Kane/Dekalb/Kendall Counties
17050 Will County
Grantee
DEI Round 5 - Kansas Treatment and Comparison LWIAs
# of LWIAs
2 Treatment Sites
20005 Area 1
20015 Area 3
2 Comparison Sites
20010 Area 2
20025 Area 5
Grantee
DEI Round 5 - Massachusetts Treatment and Comparison LWIAs
# of LWIAs
3 Treatment Sites
25020 North Central Mass
25030 Metro North
25025 Central Mass
3 Comparison Sites
25035 Brockton
25070 Metro South/West
25055 Merrimack Valley
Grantee
DEI Round 5 - Minnesota Treatment and Comparison LWIAs
# of LWIAs
3 Treatment Sites
27085 ANOKA County
Council
27105 Central Minnesota
27055 Southwest Minnesota
2 Comparison Sites
27030 South Central Minnesota
27115 Ramsey
Grantee
DEI Round 6 - Georgia Treatment and Comparison LWIAs
# of LWIAs
2 Treatment Sites
13275 Northeast Georgia
13195 Richmond-Burke
2 Comparison Sites
13225 West Central Georgia
13110 Lower Chattahoochee
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
220
DEI Rounds 5-6 LWIAs
Grantee
DEI Round 6 - Iowa Treatment and Comparison LWIAs
# of LWIAs
5 Treatment Sites
19160 Spencer (Regions 3 and 4)
19120 Waterloo (Region 7)
19135 Des Moines (Region 11)
19140 Sioux City (Region 12)
19080 Burlington (Region 16)
5 Comparison Sites
19155 Carroll (Region 8)
19095 Mason City (Region 2)
19130 Iowa City (Region 10)
19030 Marshalltown (Region 6)
19075 Ottumwa (Region 15)
Grantee
DEI Round 6 - New York Treatment and Comparison LWIAs
# of LWIAs
3 Treatment Sites
36090 HMO: Herkimer, Madison,
and Oneida
36005 Capital Region: Albany,
Rensselaer, and Schenectady
36155 CDO: Chenango, Delaware,
and Otsego
5 Comparison Sites
36185 Onondaga
36220 Broome/Tioga
36150Clinton/Essex/Franklin/
Hamilton
36195 Cayuga/Cortland
36100 Oswego County
Grantee
DEI Round 6 - Washington Treatment and Comparison LWIAs
# of LWIAs
2 Treatment Sites
53025 Seattle/King County
53030 Snohomish County
4 Comparison Sites
53015 Pacific Mountain Consortium
53005 Southwest Washington
53040 Tacoma/Pierce Consortium
53045 North Central Washington
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
221
Appendix 3: Round 5 Treatment and Comparison Sites for QED
Analysis
Kansas
WDA Name Treatment/Comparison
WDA Identifier
20005
Area 1
Treatment
20015
Area 3
Treatment
20010
Area 2
Comparison
20025
Area 5
Comparison
Minnesota
WDA Name Treatment/Comparison
WDA Identifier
27085
ANOKA County Council
Treatment
27105
Central Minnesota
Treatment
27055
Southwest Minnesota
Treatment
27030
South Central Minnesota
Comparison
27115
Ramsey
Comparison
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
222
Appendix 4: Round 6 Treatment and Comparison Sites for QED
Analysis
Georgia
WDA Name Treatment/Comparison
WDA Identifier
13275
Northeast Georgia
Treatment
13195
Richmond-Burke
Treatment
13225
West Central Georgia
Comparison
13110
Lower Chattahoochee
Comparison
Iowa
WDA Name Treatment/Comparison
WDA Identifier
19160
Spencer (Regions 3 and 4)
Treatment
19120
Waterloo (Region 7)
Treatment
19135
Des Moines (Region 11)
Treatment
19140
Sioux City (Region 12)
Treatment
19080
Burlington (Region 16)
Treatment
19155
Carroll (Region 8)
Comparison
19095
Mason City (Region 2)
Comparison
19130
Iowa City (Region 10)
Comparison
19030
Marshalltown (Region 6)
Comparison
19075
Ottumwa (Region 15)
Comparison
New York
WDA Name Treatment/Comparison
WDA Identifier
36090
HMO: Herkimer, Madison, and Oneida
Treatment
36005
Capital Region: Albany, Rensselaer, and
Schenectady Treatment
36155
CDO: Chenango, Delaware, and Otsego
Comparison
36185
Onondaga
Comparison
36220
Broome/Tioga
Comparison
36150
Clinton/Essex/Franklin/Hamilton
Comparison
36195
Cayuga/Cortland
Comparison
36100
Oswego County
Comparison
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
223
APPENDIX 5: Navigation Instructions
Navigate to the information you want:
1. Type Alt-E-F on your computer to navigate to the Find and Replace dialog box.
2. Press Find in.
3. Press Main Document.
4. Highlight text1 >: Executive Summary
5. Highlight text2 >: Description of the Program
6. Highlight text3>: Exhibit 1: Complete List of Service Delivery Strategies
7. Add Highlights as necessary.
8. www.socialdynamicsllc.com for technical assistance
1
2
3
Disability Employment Initiative Evaluation
Social Dynamics, LLC/Abt Associates
224
Acknowledgements
Contracting Officer’s Representative: David Rosenblum, M.S.
CONTACT INFORMATION
Douglas Klayman, Ph.D., President
Social Dynamics, LLC
dklayman@socialdynamicsllc.com
240-426-5823
Download
Save PDF to desktop
Email
Email completed PDF
Send for Signing
Email for others to sign
Print
Print document
Choose a Field
Click on a field and
select the person that should complete it
Want to add a signature?
You can add a signature field
and assign it to someone else
to sign
Choose a Field
Click on a field and
select the person that should complete it

Disability Employment Initiative Project Implementation and Impact Evaluation (Department of Labor)

This document is locked as it has been sent for signing.

You have successfully completed this document.

Other parties need to complete fields in the document. You will recieve an email notification when the document has been completed by all parties.

This document has been signed by all parties.

Completed 19 July 2020

Message

View Audit Log
Print With Audit Log
Duplicate Document