Prospective Validation of the New Ontario Decision Support System: Phase III Report (2009-2010)
Principal Investigator: Aron Shlonsky, Factor-Inwentash Faculty of Social Work, University of Toronto
Funded by: Ontario Ministry of Child and Youth Services (MCYS)
Project Overview
As a part of the design and implementation of the Transformation Agenda for child welfare services in Ontario, the Ministry of Children and Youth Services (MCYS) contracted with Professor Aron Shlonsky (PI) of the University of Toronto, Factor-Inwentash Faculty of Social Work to conduct a prospective validation of the investigative portion of Ontario’s new decision-making system. The original contract was divided into four phases that were to be conducted in two stages.
Stage I, Phase I: Utility Study
Stage one has been completed (see report ‘The Perceived Utility of Child Maltreatment Risk Assessment and Clinical Assessment Tools) and was comprised of a single phase (Phase I) that occurred in fiscal year 2005-2006. Phase I was an exploration of several decision-making tools, otherwise known as the ‘test drive’, whereby Ontario child protection workers completed different decision-making tools on closed cases and then participated in a series of focus groups to discuss their relative merits. This information was used by the Child Welfare Secretariat to choose and modify the current decision-making package.
Stage II, Phases II, III, and IV: Validation Study
Stage two, consisting of phases II, III, and IV will validate and, if necessary, improve the current risk assessment tool. The original contract called for the principal investigator to access and extract data from the Province’s new Single Information System (SIS). This SIS was being piloted in three sites (Simcoe, Timiskaming, and Renfrew) offering a unique opportunity to test not only the instruments but the capacity of the new system to deliver meaningful information for analysis. During the process of contract negotiation for phases II, III, and IV, the SIS pilot ran into some serious implementation issues. In anticipation of continuing difficulties, the PI asked the Children’s Aid Society of Toronto (CAST) and Toronto Catholic Children’s Aid Society (CCAS) to participate in the study (at no extra charge to them or MCYS) as part of these agencies’ ongoing involvement with the Children’s Services Database (CSD). The CSD contains the investigative and placement histories of children receiving child protection services in the province, and the new decision-making tools were simply added to new extract requests. Their inclusion insured that if SIS was unable to deliver data, the tools could still be analyzed.
The validation study uses risk assessment and administrative data to model the likelihood of subsequent investigation, confirmed abuse, child injury or death, and placement in foster care. Specifically, the MCYS contract investigates the statistical properties of the safety assessment and risk assessment tools by following a cohort (group of children and families investigated for maltreatment within a 6 month period) for up 24 months to ascertain whether children experience the following outcomes:
1. Subsequent report of child maltreatment;
2. Subsequent substantiated maltreatment;
3. Subsequent placement in foster or kinship care