APHSA Symposium 2019

HEALTH AND HUMAN SERVICES FOR ACTION ANALYTICS SYMPOSIUM SUMMARY REPORT

U N D E R W R I T T E N B Y

JULY 24-26, 2019

H Y A T T R E G E N C Y C H I C A G O U N I V E R S I T Y O F C H I C A G O - G L E A C H E R C E N T E R

@APHSA1

WWW. APHSA .ORG

SUMMARY REPORT: APHSA’S SECOND HEALTH AND HUMAN SERVICES ANALYTICS SYMPOSIUM FOR ACTION

BUILDING A CULTURE OF ANALYTICS TO CREATE A HEALTHIER WORLD JULY 24-26, 2019 | CHICAGO, ILLINOIS

TABLE OF CONTENTS I. Summary ......................................................................................... 4 II. Attendees ....................................................................................... 4 III. Highlights of Panel Presentations .................................................. 5 IV. Highlights of Breakout Sessions ..................................................... 9 V. Reflections onWhat Lies Ahead .................................................... 10 VI. Appendices .................................................................................. 11 A. Full SymposiumAgenda .......................................................... 12 B. Pre-symposium Survey Results ............................................... 14 C. Panel Presentation Slides ........................................................ 15 D. Breakout Questions for Team Exercises ................................... 29 E. Attendee Roster ...................................................................... 30

This report is a compilation of activities and presentations which took place at the DataAnalyticsSymposium inChicago,July 2019,Chicago, IL.The materials included are done so with the permission of the authors.Should errors or omission exist in the event summary, they rest with the staff, and not the symposiumpresenters.

I. Summary APHSA’s second “Health and Human Services Analytics Symposium for Action”, underwritten by Optum andThe Lewin Group, from July 24-26, 2019. The symposium focused on building state and local capacity in data analytics in the health and human services sector. The agenda, informed by a pre-symposium survey of participants as well as calls with a group of participants, covered how to develop a cross-program data analytics strategy that: • Advances program priorities and answers key policy and operational questions that will support program improvement; • Drives culture change focused on improving outcomes for individuals and families who may be served by multiple programs and to create a “culture of analytics” throughout an agency; • Builds staff capacity to use data to improve decision-making; • Shares and manages data across programs and organizations to achieve shared goals; and • Uses data ethically and transparently to ensure public trust and avoid bias. The symposium at the University of Chicago’s Gleacher Center included presentations by state and local officials overseeing innovative data analytics initiatives and by academic and industry thought leaders. Break-out sessions focused on specialized topics and included time for state and local teams to brainstorm about specific strategies they could employ to strengthen their capacity to use data analytics to address their communities’ priorities. The welcome reception onWednesday evening and networking breaks and meals during theThursday and Friday symposium provided valuable opportunities for participants to mingle and learn from colleagues from other jurisdictions and organizations. Please see Appendix A for the full symposium agenda.

II. Attendees Fifty-four individuals attended the invitation-only symposium. Thirty-six were from state and county governments fromAlaska, California, Florida, Illinois, Indiana, Ohio, Maryland, Michigan, Minnesota, Nevada, Virginia, West Virginia, and Fairfax County, Virginia. Each jurisdiction sent teams of two to five people representing different offices that were committed to building analytics capacity to achieve shared goals. There were eighteen experts and facilitators from the Annie E. Casey Foundation, Optum, The Lewin Group, Chapin Hall at the University of Chicago, and APHSA.

4

APHSA H/HS Analytics Symposium for Action 2019

III. Highlights of Presentations Below is a summary of each presentation at the symposium. Please see Appendix C for some of the slides that speakers used during their presentations. 1 Welcome and Introduction. Opening remarks were made by Scott Dunn (Director of Health and Human Services Programs, Optum), FredWulczyn (Senior Fellow, Chapin Hall at University of Chicago), and Phil Basso (Director, Organizational Effectiveness, APHSA). They discussed the value of analytics; new partnership models that involve government practitioners, academics, and industry thought leaders; the critical importance of building infrastructure – systems, software, and human capital – that create the enabling conditions for analytics; and the results of the pre- symposium survey of attendees. The primary goals for the symposium were to share practical tips and best practices on how to advance the use of analytics and make analytics an integral part of the health and human services culture. (Please see Appendix B for the results of the pre-symposium survey.) Panel 1: State Showcase. Patricia Babcock (Deputy Secretary, Florida Department of Children and Families) and KimMcCoyWade (Acting Director, California Department of Aging, and formerly the Director of California’s CalFresh (SNAP) program) described innovative analytics projects their states are pursuing to enable data-informed decision-making for state priorities. The session was moderated by Yvonne Powell (Senior VP, The Lewin Group). Patricia Babcock presented the strategies that new leadership of Florida’s Department of Children and Families are using to integrate child welfare, substance abuse and mental health, and economic self-sufficiency data to achieve two key goals: (1) reducing the number of people in crisis and (2) reducing re-entry. These included: • An Office of Innovation that solicits improvement ideas from staff and strengthens enterprise-wide data analytics capacity. • Establishing an Advanced Statistical Analysis Laboratory for ChildWelfare, which will bring in university students. • Creating a Child Safety Data Hub tasked with identifying and integrating all data sources, including education, that can extend child safety when incorporated into analytics, research, and operational processes.

• Creating visualization tools to help front-line workers and supervisors use data for case management. For example, FL created a visualization tool for perpetrator chronicity in child welfare by merging data frommultiple sources. • Making a Chief Quality Officer responsible for data quality in a central data repository that can be used to create the right metrics, at the right time. This office draws on academics and university students and provides training for staff on how to use data. Babcock stressed the importance of involving front-line workers and supervisors as new visualization tools are developed to ensure they see the value of the new tools and have the necessary training to use them. Future projects will focus on using integrated data to refine metrics, creating data visualizations to inform decision- making, and pursuing opportunities to incorporate artificial intelligence and bots into the agency’s data strategy. (Please see Appendix C-1 for slides .) KimMcCoy-Wade described the strategies California’s HHS is using to deliver people-centered, data-driven services across 58 counties and to answer these key questions: (1) how well are we serving the people of California? (2) who are we missing? (3) how can we reach new people? (4) what is the impact of our services on people? and (5) how do we develop data leaders to keep asking and answering these questions with and for the people we serve? She presented specific examples CA is using including: • The CalFresh Data Dashboard 2 , which analyzes and visualizes SNAP trends, by county, in demographics, participation rates, timelines, benefit accuracy, and churn rates. • A participation analysis that links Medicaid, Temporary Assistance for Needy Families (TANF), Supplemental Nutrition Assistance Program (SNAP) and other program data to understand where enrollment gaps exist (e.g., individuals who are likely eligible for multiple programs that are only enrolled in one). • A data hub , under development, that will link state and county data on SNAP and SSI recipients, including languages spoken, to help counties conduct outreach and enroll SSI recipients in SNAP. • A research study by the University of California

San Francisco, funded by philanthropy, to examine the impact of SNAP benefits on health outcomes of older adults on SSI.

1 Slides in Appendix C only include highlights & select slides.You may find all full presentations online here: https://aphsa.org/NC/NC/events_and_ engagement.aspx 2 CalFresh Data Dashboard: http://www.cdss.ca.gov/inforesources/Data-Portal/Research-and-Data/CalFresh-Data-Dashboard

5

Building a culture of analytics to create a healthier world!

• Analyses for the legislature and stakeholders, on enrollment trends, approvals and denials (e.g., missed interview), and benefit amounts. • Creating “Data Leads” within program offices who work on joint projects across programs, in partnership with a central data and research unit, to build data capacity and drive culture change. (Please see Appendix C-2 for slides.) Issues highlighted during the Q&A with panelists: • Master data-sharing agreements: in place of “Master Agreement” that covers sub-projects. • Data-sharing “Messenger”: staff don’t want to hear from “SiliconValley types”; they need to hear from their peers about the value of data-sharing. Panel 2: Change Management. The second panel discussed how to encourage data driven decision making in health and human services agencies and how to make the process stick. Presenters were Connor Norwood (Chief Data Officer, Indiana Family and Social Services Administration) and three officials from Fairfax County, Virginia (Ron Chavarro, Deputy Director, Department of Family Services; Kevin Filbey, Professional and Organizational Development Director; and John Ruthinoski, Senior Project Manager). Phil Basso fromAPHSA moderated the panel. Connor Norwood described the importance of agencies having a clear vision statement that ties the creation of an “insight driven culture” to improved health and well-being of citizens. Three key mantras are: (1) get people access to services as quickly as possible; (2) figure out what else they need; and (3) measure and report progress using data. To assess the current state of analytics and the obstacles to establishing a data-driven culture, Norwood engaged staff from both the “data” and “business” teams. Key obstacles identified by staff were lack of understanding about how to use analytics to improve services, lack of data analytics skills, difficulty in accessing data, concerns with the data, and a culture that does not encourage data sharing. His organization devised strategies to address these obstacles, including: • A data governance framework to ensure secure availability of high quality data for decision-making. A legislative change to assist inmates to apply for Medicaid prior to release provided a powerful use case for linking Corrections and Medicaid data. one-off data-sharing agreements that are slow and time-consuming to implement, CA has created a

• A data literacy strategy, modeled on the Federal Data Strategy 3 , which trains staff to use “data as a strategic asset.”

• Defining clear value propositions that align the business strategy. For example, based on data analysis, Indiana instituted a “carve-out” for Hepatitis-C virus medications fromMCO plans, saving $10K per member receiving medication each month. (Please see Appendix C-3 for slides.) The Fairfax County team described how their strategic vision and goals have driven progress. The vision states: “By 2020 we will address the root cause of our customers’ needs by integrating service at an optimal level to generate positive changes in the community.” Strategic goals are: (1) work together in partnership with the community ; (2) provide customized, responsive, and top-quality services to our diverse community; (3) cultivate an environment supportive of critical thinking, happiness, and opportunities for innovation ; and (4) encourage and support one another in our workforce to develop and maximize our talents and leadership skills. To address strategic goal number four, Fairfax County began hosting a “Data Analytics Fellowship Academy” (DAFA) in 2016 (based on a similar program in New Jersey’s child welfare agency). It teaches data analysis, business process, diagnostic and presentation skills, and has been a culture change catalyst. Classes of up to 25 staff (mostly practitioners) work in teams on real-time data analytics projects that inform agency decisions. Over ten months, Fellows attend workshops two days per month and have additional time dedicated to work on class projects in teams. They are trained to conduct quantitative analysis of client-level data in Excel spreadsheets, conduct literature reviews and qualitative analysis, perform business-process mapping (i.e., the steps to get from point A to point B), and integrate information to develop solutions. Teams present their findings and recommended solutions to agency leaders, staff and stakeholders for feedback and decisions. Each class generates 15 to 20 data-informed solutions that are tracked to measure the success of the program. The classes have wide-ranging benefits to the HHS system, the county agency, and clients. Trained fellows mentor other staff in problem-solving and data analysis. The presenters cited examples of policy and operational changes that have resulted from DAFA, including targeted outreach to reach individuals in need of additional services. (Please see Appendix C-4 for slides.)

3 Federal Data Strategy: Leveraging Data as a Strategic Asset: https://strategy.data.gov

6

APHSA H/HS Analytics Symposium for Action 2019

Panel 3: Analytics to Answer Policy Questions. Fred Wulczyn (Chapin Hall) and Matt Stearmer (Chief Data Officer, Ohio Department of Medicaid) presented examples of how data can be used to answer policy-relevant questions that demonstrate the value of building analytics capacity. Tracy Wareing Evans (APHSA) moderated the session. FredWulczyn’s presentation emphasized the importance of starting with a policy question and asking more granular questions to understand variation so that effective, targeted solutions can be developed. As a use case, he started with this state-level policy question “To reduce over- representation of black children in foster care, what resources are needed and how should they be allocated?” To answer this question, decision-makers need to dive more deeply into the data to answer questions about variation, including: (1) where are the racial disparities occurring? (2) what are the characteristics of those places compared to other places? (3) are the disparities attributable to how long a child stays in care? and (4) what are the disparities in guardianship vs. adoption? His slides (Appendix C-5) , included graphs showing quantitative variations that could be used by decision-makers to improve targeting of resources to achieve their policy goal. Matt Stearmer described lessons from Ohio’s experiences using data to improve healthcare delivery. For example: • It is critical to anticipate and model unintended consequences of policy decisions. Ohio’s well-intended policy of closing down “pill mills” to stem the opioid epidemic drove people with addiction into the illicit market, unexpectedly driving death rates even higher. • An effective way to improve data quality is to engage people in developing solutions. Five years ago, 30 percent of Ohio’s Medicaid race and ethnicity data was incomplete, stemming in part from people’s distrust of the medical system. After Ohio invited people to participate on a working group to develop solutions, they achieved a 95 percent completion rate. • Real-time data analytics projects that generate useful information for decision-makers demonstrate the value of analytics. When Ohio was redesigning its behavioral health system, the data analytics team joined data to answer policy questions in real time, making it a vital resource.

• A strong data governance framework is critical to ensuring data can be linked to answer all key policy questions.

Panel 4: Thought Leader Panel: This panel of industry experts described strategies for accelerating the use of analytics. Presenters were Erika Robbins (Vice President, The Lewin Group), Jeanne McNeil (Director, Delivery Operations, Optum), and Don Johnson (Chief Technology Officer, Optum). The panel was moderated by AndrewCone (Senior VP, Optum). Erika Robbins used LongTerm Services and Supports (LTSS) as a use case for how to improve data literacy across the workforce and accelerate the use of analytics for improvement. In many organizations, a gap exists between data and analytics leaders and organizational units responsible for program policy and operations. Closing the gap requires understanding the knowledge base and learning styles of people in the organization, which she classified into four groups: Data Aristocrat, Data Knight, Data Dreamer, and Data Doubter. 4 She offered strategies for engaging each type of individual to strengthen their data literacy, and for using an analytical advisory group to identify priorities and develop a knowledge growth plan. She described several projects The Lewin Group is conducting with the U.S. Administration on Community Living and in Georgia to improve data dissemination by tailoring information and visualizations to the end-user. (Please see slides in Appendix C-6 .) Jeanne McNeil provided examples of Michigan projects that use analytics to drive program improvement . These included: • matching data fromMedicaid and the RyanWhite HIV program to identify individuals enrolled in both whose services could be covered by Medicaid, thus freeing up more slots for RyanWhite coverage; • matching beneficiary data for Medicaid, SNAP, andWIC to identify individuals who could potentially be enrolled in additional programs to help them achieve better outcomes; • evaluating Drug and Mental Health Court

recidivism programs by comparing the outcomes of participants to an equivalent population that did not participate; and

• identifying high risk individuals for the Medicaid Beneficiary Monitoring Program who appear to be overutilizing or misusing their Medicaid benefits, and identifying whether beneficiaries should be candidates for the Opioid Health Home.

4 https://www.qlik.com/us/bi/-/media/635E0FBF5EC54CC49793DC3FDB5F5DA2.ashx

7

Building a culture of analytics to create a healthier world!

She described multiple examples of how Michigan is using its Enterprise DataWarehouse – which includes health and non-health data linked through a master-client index – to help providers and beneficiaries access integrated information that can improve health outcomes. Projects include: • CareConnect360, a statewide care management web portal providing a comprehensive view of individuals’ and populations’ participation

• Make your AI system-agnostic so that it can be used with multiple systems, including new systems that replace legacy systems. • Use integrated data to train AI. (Don’t use AI to automate existing rules and process steps since AI can inform improvements to those.) (Please slides in Appendix C-6. ) Presentation: Getting “Big Data” to the Good Guys. Chris Kingsley, Senior Associate for Data Initiatives at the Annie E. Casey Foundation, discussed trends in how state and local governments are using integrated data to better understand population needs and predict which individuals and communities are at greatest risk (e.g., children at risk of abuse or juveniles at risk of engaging in crime.) Key ideas and examples (please see slides in Appendix C-7 ) included: • Several high-profile predictive analytics initiatives recently failed because of flawed algorithms, botched implementation, and lack of transparency with community members. The following articles highlight the factors that led to abandoned predictive analytics projects: • Data mining program designed to predict child abuse proves unreliable 5 • Palantir Has Secretly Been Using NewOrleans toTest its Predictive PolicingTechnology 6 • Well-intended, well-designed analytics that highlight overrepresentation of minorities in a risk group or system can be viewed as inherently racist and lead to community backlash. For example, when data from schools and local law enforcement are used to predict which youth are most likely to engage in crime, communities may fear that the predictions will magnify disparities and essentially operate like racial profiling. • The MetroLab Network has published four principles and a useful checklist 7 for ethically applying predictive tools within human services . The principles are: (1) Engage (internally within the organization and with the community); (2) Pre-validate the model; (3) Review (by evaluating and adjusting the model) and (4) Open Up (proactively sharing information about the model). Kingsley facilitated an interactive discussion with symposium participants about practical strategies for ensuring appropriate and ethical use of data and

in programs, demographic information, health conditions, and services received.

• MI Bridges, an integrated service delivery portal to enable individuals to apply for benefits, explore and connect with resources based on the information

they enter, and manage their own cases. • Predictive analytics to improve targeting of outreach and services. (Please see slides in Appendix C-6 .)

Don Johnson described how artificial intelligence (AI) can support more proactive, predictive solutions to improve customer-centric care . AI dramatically increases the amount of information that can be collected and analyzed frommultiple sources (e.g., legacy systems, distributed systems, cloud-based systems), enabling employees to focus more attention on tailoring services to client needs. “Machine learning” (which uses structured/labeled data) and “deep learning” (which can work with vast amounts of unstructured data) are subsets of AI that use new data to continually improve how a system classifies information and performs tasks. In the future, the same AI techniques that Amazon now uses to predict what customers will want to purchase could be used in human services to reduce or eliminate application forms, intervene before a situation becomes critical, or identify preventive actions that address social determinants of health. AI can make predictions that help case workers know how to engage clients and what questions to ask. (It represents a tool for case worker decision making.) Some key principles for building capacity to use AI include:

• Do NOT build AI into legacy systems. • Externalize your data (i.e., store it in an external repository outside your operational systems) – in real time.

5 ChicagoTribune (December 2017), “Data Mining Program Designed to Predict Child Abuse Proves Unreliable”: https://www.chicagotribune.com/ investigations/ct-dcfs-eckerd-met-20171206-story.html 6 TheVerge (February, 2018), “Palintir Has Been Secretly Using NewOrleans toTest its Predictive PolicingTechnology”: https://www.theverge. com/2018/2/27/17054740/palantir-predictive-policing-tool-new-orleans-nopd 7 MetroLab Network’s Ethical Guidelines for Applying Predictive Tools in Human Services: https://metrolabnetwork.org/wp-content/uploads/2017/09/ Ethical-Guidelines-for-Applying-Predictive-Tools-within-Human-Services_Sept-2017.pdf

8

APHSA H/HS Analytics Symposium for Action 2019

IV. Highlights of Breakout Sessions Breakout Topics. There were four breakout sessions on specific topics, each attended by several state or local teams. Below are highlights of each session. Session 1: Making your desired state for data and analytics clear. This session was facilitated by Fred Wulcyzn (Chapin Hall) and Phil Basso (APHSA). The group discussed how to build an understanding of purpose for data and analysis throughout an agency and arrived at a common definition: “To understand the root causes of barriers to desired outcomes so that effective innovations can be advanced.” Key discussion points: • Initial steps are needed to improve the quality and scope of data collected and the ability to use data across programs. Agencies can then build up a basic skillset for using data and analysis throughout the agency including the local office level, such as through a learning program that embeds needed critical thinking and continuous improvement methods. • It’s helpful to then focus on tangible challenges like child trauma prevention, economic mobility for TANF recipients, establishing an ROI for long- term support services, or reducing dependency on opioids. Aligning culture to agency vision and values is best accomplished by changing decision-making, casework practices and resource allocation versus in the abstract. • Phasing the effort and being patient with small steps and small wins is important. Linking progress in data and analysis with support from functions like HR, Legal, Finance, and especially Communications is also critical. Session 2: Specific analytics applications, especially matching resources to needs, preventing trauma, and making the ROI case. This session was facilitated by JenniferWalthall (Secretary, Indiana Family and Social Services Administration) and Scott Dunn (Optum). The group discussed examples of using data analytics to understand the needs of clients, prevent and mitigate trauma, and achieve high ROI. Examples included: • Indiana’s Office of Healthy Opportunities 8 is driving a cross-agency, data-driven focus on social determinants of health that helps match resources to needs. IN now collects voluntary information on social needs from applicants for TANF, SNAP and Medicaid and uses the information to create a state-wide heat map of unmet needs. Staff receive training on how to ask clients sensitive questions

authentically engaging community members to build trust and increase transparency. The group offered strategies including: (1) increasing racial diversity among decision-makers in policy roles; (2) forging alliances between government and authentic voices in advocacy organizations to build consensus on how to use government levers appropriately; (3) writing into RFPs agreed-upon terms for data use; (4) involving community members in the early planning stages of analytics initiatives to get public buy-in; (5) improving staff training; and (6) making improved access to services (rather than “scrutiny” of individual behavior) the focus of using integrated data.

8 Indiana Office of Healthy Opportunities, Driving a Cross-Agency Focus on Equity and Access: https://www.chcs.org/media/IN-Office-of-Healthy- Opportunities-Case-Study_120518.pdf

9

Building a culture of analytics to create a healthier world!

V. Reflections on What Lies Ahead During the two-day meeting, the state and county participants lifted up several subjects that require further reflection to advance the field. For example, the session discussing ethical application of predictive tools prompted a very an important debate amongst participants, and participants requested that APHSA move forward with promoting discussions of more authentic engagement and trust engagement and trust building with community, how to increase racial diversity within their own agencies, and so on. Other feedback included a need for more concrete examples of how organizations are applying data analytics and are using AI to improve outcomes. Finally, some IT staff would like to dig into specific data literacy and other resources that would help them engage with their business/program side to understand how better to support them and their needs. The planning team for the event has been engaged in follow up calls conducted with over half the participating teams. The calls have shown that plans and next steps for advancing data and analysis practices are in place and moving forward without exception, covering foundation basics, change management, and targeted applications. Call participant needs for ongoing support in making changes are varied but the essential capacity for following through appears to be in place. Participant suggestions for future conversations in this space include networking time amongst systems that share either an urban or a rural orientation, or that are either state or locally administered.

in order to be more responsive to their needs. • Reducing infant mortality and infant morbidity has extremely high ROI due to the enormous costs of neo-natal health care and of long-term supports for children with lasting disabilities. Strategies for identifying and engaging at-risk mothers in effective pre-natal care to address their health and social needs (e.g., evidence- based home visiting programs) hold potential for improving maternal and child outcomes while significantly reducing health care costs. • Data analytics strategies must try to identify all factors relevant to understanding population needs and developing effective solutions (e.g., places with large numbers of undocumented immigrants) and data collection must not inflict trauma (e.g., create fear that the government will share data with law enforcement agencies.) Session 3: “Knowing what we know” about inequities by race and place. This session was facilitated by Kevin Filbey (Fairfax County, VA) and Christina Becker (APHSA). The group discussed the importance of collecting race/ethnicity data, and the importance of including the community to create system change. Examples included: • Creating a policy, such as One Fairfax 9 , that commits the agency to intentionally consider equity when making policies or delivering programs and services. The group noted the importance of an operational mechanism behind the policy, so it has a chance to make an impact. • Acknowledging the fact that human services agencies are part of a larger system that has caused some inequities to continue and grow. The community needs to lead the conversation about what data should be collected about them, and should be a driver of solutioning for them, rather than one agency trying to drive this process on behalf of its customers. Session 4:Workforce staffing challenges and building up the data literacy of your staff. This session was facilitated by Cynthia Green Edwards (Michigan Chief Compliance Officer) andYvonne Powell (Lewin Group). Team Exercises. On each day, teams from each jurisdiction met to brainstorm about how to apply what they learned to their own operations and improvement plans. See Appendix D .

9 Fairfax CountyVirginia’s One Fairfax Policy: https://www.fairfaxcounty.gov/topics/one-fairfax

APHSA H/HS Analytics Symposium for Action 2019 10

VI. Appendices A. Full SymposiumAgenda B. Pre-symposium survey results c. Panel presentation slides

C-1: Innovation, Analytics, and Data-informed Decision Making (Patricia Babcock, FL) C-2: Delivering People-Centered, Data-Driven Services (KimMcCoyWade, CA) C-3: Becoming Data-Driven & Connecting to Purpose (Connor Norwood, IN) C-4: Building Our Analytics Culture (Ron Chavarro, Kevin Filbey, John Ruthinoski, Fairfax Co.) C-5: The Deeper Dive: Why Taking the Plunge Matters (FredWulczyn, Chapin Hall) C-6: Accelerating the Use of Analytics: • Advancing Analytical Decision-Making in Long-Term Supports (Erika Robbins, The Lewin Group) • Analytics in Action (Jeanne McNeil, Optum) • Beyond Analytics: Foundations of AI to Achieve Predictive Decisions (Don Johnson, Optum) C-7: Getting Big Data to the Good Guys (Chris Kingsley, A.E. Casey Foundation) D. Breakout Template for State and Local Teams E. Attendee Roster

Building a culture of analytics to create a healthier world! 11

Appendix A. Full SymposiumAgenda

Health and Human Services Analytics Symposium for Action Building a culture of analytics to create a healthier world!

Wednesday, July 24, 2019

6:00 p.m. to 8:00 p.m.

Welcome Reception Hyatt Regency Chicago 151 EastWacker Dr. Chicago, Illinois

TracyWareing Evans President and CEO, American Public Human Services Association

Thursday, July 25, 2019

8:00 am to 8:30 am Breakfast at Gleacher Center 450 Cityfront Plaza Dr. Chicago, Illinois 8:30 am to 9:30am Welcome and Introduction to Symposium Goals for This Symposium Review of LastYear’s Symposium Review of “Findings” from Pre-SymposiumTA

Scott Dunn , Director of Health and Human Services Programs, OptumGovernment Solutions FredWulczyn , Senior Fellow, Chapin Hall at the University of Chicago Phil Basso , Director of Organizational Effectiveness, APHSA Patricia Babcock , Deputy Secretary, Florida Department of Children and Families KimMcCoyWade , Acting Director,

9:30 am to 10:30 am

State Showcase A round of innovative state analytics projects.

California Department of Aging Yvonne Powell , Sr. Vice President, The Lewin Group, Moderator

10:30 am to 11:00 am Networking Break 11:00 am to 12:00 pm

Change Management How to make analytics happen in your agency and how to make the process stick. What do you need? How do you prepare staff?

Connor Norwood, Chief Data Officer, Indiana Family and Social Services Administration Ronald Chavarro, Deputy Director, Fairfax County (VA) Department of Family Services Kevin Filbey, Professional and Organizational Development Director, Fairfax County John Ruthinoski, Senior Project

Cultural Transformation during large initiatives

Manager, Fairfax County Phil Basso, Moderator

12:00 pm to 1:00 p.m. 1:00 pm to 2:00 pm

Lunch

Analytics to Answer Policy Questions Understanding the nature of data. Administrative data versus coordinate case-level data. Projects using de-identified data.

Steven Matthew Stearmer, Chief Data Officer, Ohio Department of Medicaid FredWulczyn, Chapin Hall TracyWareing Evans, Moderator

2:00 pm to 2:30 pm Short Break andTransition to Breakout Sessions 2:30 pm to 4:00 pm Breakout Sessions

Facilitated by Analytics Symposium PlanningTeam Faculty: Secretary JenniferWalthall Kevin Filbey FredWulcyzn Cynthia Green Edwards

4:00 pm to 4:15 pm Break 4:15 pm to 5:00 pm

Debrief Breakout Sessions, and Next Steps for Day Two

Christina Becker, Health Policy & ProgramAssociate, APHSA

Dinner onYour Own

APHSA H/HS Analytics Symposium for Action 2019 12

Friday, July 26, 2019 7:45 am to 8:15 am Continental Breakfast at the Gleacher Center 8:15 am to 8:30 am Insights from Day One

Christina Becker, APHSA

8:30 am to 9:45 am

Thought Leader Panel How to accelerate the use of analytics

AndrewCone, Sr. Vice President - OptumGovernment Solutions, Moderator Jeanne McNeil, Director, Delivery Operations at Optum

Don Johnson, CTO andVP of Product, Optum Erika Robbins, Vice President, The Lewin Group

9:45 am to 10:00 am Break 10:00 am to 10:45 am

Getting “Big Data” to the Good Guys* This session invites participants to address some of the new challenges that analytics and risk assessment raise related to trust, transparency, and the purpose of systems innovation.

Chris Kingsley, Senior Associate for Data Initiatives, Annie E. Casey Foundation

10:45 am to 11:30 am

Breakout Sessions

Facilitated by Analytics Symposium PlanningTeam Faculty: SecretaryWalthall Kevin Filbey FredWulcyzn Cynthia Green Edwa rds

11:30 am to Noon

Reflections onWhat Lies Ahead We will debrief the breakout session, discuss key reflections of the Symposium, and plan for post- SymposiumTA to build capacity in analytics use in the field.

Phil Basso, APHSA

Noon

Box Lunch and Departures

Building a culture of analytics to create a healthier world! 13

Appendix B. Pre-symposium survey results

Symposium for Action Pre-symposium Findings Presentation by Phil Basso, APHSA Director of Organizational Effectiveness The purpose of the pre-symposium survey and calls was to understand what’s foremost on the minds of participants in order to:

• Strengthen the planned presentations • Determine the breakout spotlight topics • Set the stage for next steps after the symposium Consensus emerged around seven topics: 1. Making your desired state for data and analytics clear • Linking your data and analysis priorities to overall strategy and program priorities. • Putting programs/practitioner in the driver’s seat on data and analysis • Answering the question, “why do we need analytics?” 2. Building up the data literacy of your staff  • Defining terms: who is doing this well/no need to reinvent in every system • Using broad internal programs like a data academy • Linking analytics to case-level workflows and practice models • Making decisions using (imperfect) data and analysis 3. Workforce staffing challenges

• Staffing and staff retention strategies • Using both internal and contracted talent

• Having the right mix of skillsets and developing them as needed 4. Sharing and managing data across programs and entities • Partnership development around shared goals

• Different approaches to governance • Having strong legal counsel in place • Understanding what political will is in place • Outlook on federal-level supports  

5. “Knowing what we know” about inequities by race and place • Ethical considerations: what are we obliged to study and act upon? • Practical considerations: how do we impact disparities we know about?

6. Using a Theory of Impact to drive factor modeling, ROI analysis and continuous learning • Understanding what it takes to establish algorithms connecting data to predictive power • Creating a culture of learning and of generating evidence 7. Specific analytics applications, especially matching resources to needs, preventing trauma, and making the ROI case • Long-term support services • Child welfare

• Social and economic mobility for TANF recipients Opioids/substance abuse prevention and treatment

APHSA H/HS Analytics Symposium for Action 2019 14

Appendix C. Panel Presentation Slides Select slides from our panel presentations are below. To see each of these presentations in full, please visit https://aphsa.org/NC/NC/events_and_engagement.aspx Appendix C-1: Select Slides from Innovation, Analytics and Data-Informed Decision Making, Patricia Babcock and Rodney Moore, Florida Department of Children and Families

Building a culture of analytics to create a healthier world! 15

Appendix C-2: Select Slides from Delivering People-Centered, Data-Driven Services: Case Study from CA SNAP/Cal Fresh , KimMcCoyWade, Acting Director, California Department of Aging

APHSA H/HS Analytics Symposium for Action 2019 16

Appendix C-3: Select Slides from Becoming Data-Driven & Connecting to Purpose, Connor Norwood, Chief Data Officer, Indiana Family & Social Services Administration

Building a culture of analytics to create a healthier world! 17

Appendix C-4: Select Slides from Building Our Analytics Culture; Ron Chavarro, Deputy Director; J. Kevin Filbey, Professional and Organizational Development Director; John Ruthinoski, Data Analytics Senior Project Manager, Fairfax County (VA) Department of Family Services

APHSA H/HS Analytics Symposium for Action 2019 18

Appendix C-5: Select Slides from The Deeper Dive: WhyTaking the Plunge Matters, FredWulczyn, Center for State ChildWelfare Data, Chapin Hall, University of Chicago

Building a culture of analytics to create a healthier world! 19

APHSA H/HS Analytics Symposium for Action 2019 20

Appendix C-6: Select Slides from three presentations about Accelerating the Use of Analytics

1. Advancing Analytical Decision-Making in LongTermServices and Supports, Erika Robbins, Vice President, The Lewin Group

Building a culture of analytics to create a healthier world! 21

2. Analytics in Action, Jeanne McNeil, Optum Delivery Director, State Government Solutions

APHSA H/HS Analytics Symposium for Action 2019 22

Building a culture of analytics to create a healthier world! 23

3. B eyond Analytics: Foundations of AI to Achieve Predictive Decisions, Don Johnson, Chief Technology Officer &Vice President of Product, Optum

APHSA H/HS Analytics Symposium for Action 2019 24

Building a culture of analytics to create a healthier world! 25

APHSA H/HS Analytics Symposium for Action 2019 26

Appendix C-7: Select Slides from Getting Big Data to the Good Guys, Chris Kingsley, Senior Associate for Data Initiatives at Annie E. Casey Foundation

Building a culture of analytics to create a healthier world! 27

APHSA H/HS Analytics Symposium for Action 2019 28

Appendix D. Breakout Template for State and Local Teams

APHSA’s Second Health and Human Services Analytics Symposium for Action: BreakoutTemplate for State and LocalTeams

State or Locality: ____________________________ DAY ONE

This is an opportunity for your team to discuss what you’d like to take back to your system as lessons and improvements. On Day Two you’ll have an opportunity to further shape these ideas into next steps. Questions to Consider: 1. What basic data and analysis work does your system need to prioritize as foundational?

2. What analytics-related topic or application do you want to prioritize? 3. What change readiness do you feel you have in place to make related plans? 4. What help or advice would you like from others who are here at the symposium?

Your facilitators will wrap up this breakout by asking each team to share highlights from their discussion, inviting other teams to provide input, and capturing answers to question #4 for a full-group discussion after the breakout. DAYTWO This is an opportunity for your team to shape your ideas from Day One into next steps and initial plans. Questions to Consider: 1. What basic and analytics-related work do you want to prioritize for your next steps back home? 2. How do your change readiness strengths and barriers play into your next steps and plans? 3. What help and advice do you need after the symposium to support your next steps? From partners? Staff? Peers? The SymposiumTA team? Your facilitators will wrap up this breakout by asking each team to share highlights from their discussion, inviting other teams to provide input, and capturing answers to question #3 for a full-group discussion after the breakout.

Building a culture of analytics to create a healthier world! 29

Appendix E. Attendee Roster

First Name Last Name:

Organization/Agency:

Title:

Yousif Patty

Al-Hajiby Babcock

Minnesota Department of Human Services Florida Department of Children and Families American Public Human Services Association American Public Human Services Association

Research Analyst Deputy Secretary

Phil

Basso

Director

Christina Becker Suzanne Bierman

Health Policy & ProgramAssociate

Nevada DHCFP

Administrator

Matthew Buck

State of Michigan DHHS

State Assistant Administrator

Ron Hari

Chavarro

Fairfax County (VA) Department of Family Services Deputy Director

Dulal Dunn

VA Department of Social Services

Deputy Comissioner-Strategy & Engagement Director, Health and Human Service Programs

Scott Sarah Kevin

Optum

Esty

State of Michigan DHHS

Senior Deputy Director

Filbey Flagg Gaines

Fairfax County (VA) Department of Family Services Professional and Organizational Development Director

Ann

American Public Human Services Association

Director, Collaborative Centers Medicaid Program Specialist III Deputy Secretary & Chief of Staff

Luana

State of Alaska

Michael

Gargano

IN Family and Social Services Administration

Amy

Gilbert

Indiana Family and Social Services Administration Chief Science Officer

Cynthia

Green-Edwards MI Department of Health and Human Services

Chief Compliance Officer

Drea

Hall

Illinois Department of Human Services Minnesota Department of Human Services American Public Human Services Association

Director of Strategy, Equity &Transformation Continuous Quality Improvement Supervisor

Danielle Hegseth

Donna

Jarvis-Miller

Director, Membership and Events

Don

Johnson

Optum Optum

CTO andVP of Product

Rebecca Kane

Sr Director

Chris David Heidi Jared

Kingsley

Annie E Casey Foundation

Senior Associate

Lee

Baltimore (MD) City Department of Social Services AK Department of Health and Social Services IN Family and Social Services Administration California Department of Aging | Acting Director Department of Children and Families Illinois Department of Human Services Maryland Department of Human Services Indiana Family Social Services Administration WV Department of Health and Human Resources OH Department of Job and Family Services

Deputy Director for Local General Operations

Lengdorfer

State HIT Coordinator Chief Information Officer Chief, CalFresh and Nutrition

Linder

Kim McCoyWade

Michael Rodney

McCreight

Chief Operating Officer

Moore

Assistant Secretary for Substance Abuse and Mental Health

Gabriela Moroney

Senior Public Service Administrator

Subi

Muniasamy Norwood Nowviskie

Chief Technology Officer

Connor

Chief Data Officer

Kent

Assistant Deputy Secretary state administrative assistant

Kathy

Pabst

State of Michigan DHHS

Lourdes

Padilla

Maryland Department of Human Services

Secretary

Tanuj

Patel

Optum

Analytics Director Research Analyst

Jon

Pedersen

Minnesota Department of Human Services

Julia

Peek

Nevada Department of Health and Human Services Deputy Director

Yvonne

Powell

The Lewin Group

Sr. VP

Erika John

Robbins

Optum/Lewin

Vice President

Ruthinoski

Fairfax County (VA) Department of Family Services Senior Project Manager

Sandie

Ruybalid

NV Division of Health Care Financing & Policy

Chief IT Manager

Kathy

Stack

KB Stack Consulting

CEO

Matthew Stearmer

Ohio Department of Medicaid

Chief Data Officer

Jen

Walthall

IN Family and Social Services Administration

Secretary

Tracy

Wareing Evans American Public Human Services Association

President and CEO

OH Department of Job and Services, Office of Family Assistance

Kara

Wente

Deputy Director 6

Deanne Wertin

CA Health and Human Services Agency

Deputy Agency Information Officer Assistant to the Deputy Secretary

Jeff

Wiseman

WV Department of Health and Human Resources

Tom Woods

Annie E. Casey Foundation

Senior Associate

Fred

Wulczyn

Chapin Hall at the University of Chicago

Senior Research Fellow

APHSA H/HS Analytics Symposium for Action 2019 30

THANK YOU TO OUR UNDERWRITING SPONSORS!

The American Public Human Services Association (APHSA) is a bipartisan, nonprofit membership organization representing state and local health and human service agencies through their top-level leadership. Our mission is to advance the well-being of all people by influencing modern approaches to sound policy, building the capacity of public agencies to enable healthy families and communities, and connecting leaders to accelerate learning and generate practical solutions together. We envision a world with thriving communities built on human potential. Through our member network and three national Collaborative Centers, APHSA seeks to influence modern policies and practices that support the health and well-being of all children and families and that lead to stronger communities. Through our National Collaborative for the Integration of Health and Human Services, APHSA has particular and extensive experience on the integration and alignment of health and human service programs and on the organizational dynamics that support or inhibit that integration. The Lewin Group is a premier national health and human services consulting firm. We have more than 45 years of experience providing government clients with strategic and analytical services to improve policy and expand knowledge of health care and human services systems; enact, run, and evaluate programs to enhance delivery and financing of health care and human services; deal with shifts in health care practice, technology, and regulation; optimize performance, quality, coverage, and health outcomes; and create strategies for, governments, institutions, communities and people to make health care and human services systems more eŠective. We are committed to independence and integrity in our work. We combine professional expertise with extensive knowledge and a rigorous approach to analyzing and solving problems to deliver value to each of our clients and to the larger community.

Optum is a leading health and human services (HHS) and innovation company dedicated to helping make the HHS system work better for everyone. With more than 145,000 people worldwide, Optum combines technology, data and expertise to improve the delivery, quality and eŽciency of HHS services. Optum uniquely collaborates with all HHS participants, connecting them with a shared focus on creating a healthier world. Hospitals, doctors, pharmacies, employers, health plans, government agencies and life sciences companies rely on Optum services and solutions to solve their most complex challenges and meet the growing needs of the people and communities they serve. Our services and solutions are infused with our core competency of data and analytics. We call this OptumIQ, which is the blend of curated data, leading analytics, and applied expertise to build a common language, innovate with purpose, and guide action for success. Chapin Hall at the University of Chicago is one of the nation’s foremost research centers focused on the well-being of vulnerable children. Although the portfolio of work being done at Chapin Hall is broad, the research program, developed over the past 35 years, is organized around the use of electronic records for applied research with special emphasis on policy analysis, program planning, and evaluation. The Center for State Child Welfare Data, housed at Chapin Hall for the past 15 years, is a long-standing partnership between state and local child welfare agencies, the American Public Human Services Association, and Chapin Hall. The Center combines a diverse research agenda with a robust technical assistance arm devoted to supporting public and private child welfare agencies and their eŠorts to both generate and use research evidence.

The Lewin Group and Optum are proud sponsors of the second Health and Human Services Analytics Symposium for Action. We believe that through the power of analytics, we can help create a healthier world. The Lewin Group and Optum provide insight-driven solutions in 49 Medicaid and Human Services agencies, together oŠering clients policy and analytic consulting, data analytics and enterprise data warehousing, claims processing and eligibility systems, population health management, as well as behavioral health services, long term services and supports and programs. To find out more, go to Lewin.com or Optum.com.