90 FR 95 pgs. 21310-21311 - Protocol for the Pathways to Work Evidence Clearinghouse: Methods and Standards, Version 2.0
Type: NOTICEVolume: 90Number: 95Pages: 21310 - 21311
Pages: 21310, 21311FR document: [FR Doc. 2025-08842 Filed 5-16-25; 8:45 am]
Agency: Health and Human Services Department
Sub Agency: Children and Families Administration
Official PDF Version: PDF Version
[top]
DEPARTMENT OF HEALTH AND HUMAN SERVICES
Administration for Children and Families
Protocol for the Pathways to Work Evidence Clearinghouse: Methods and Standards, Version 2.0
AGENCY:
Administration for Children and Families, Department of Health and Human Services.
ACTION:
Request for public comments.
SUMMARY:
The Administration for Children and Families (ACF), within the U.S. Department of Health and Human Services (HHS), oversees the Pathways to Work Evidence Clearinghouse (Pathways to Work). ACF seeks comments on proposed revisions to existing standards and on new standards for including and rating the quality of program cost studies in the "Protocol for the Pathways to Work Evidence Clearinghouse: Methods and Standards, Version 2.0".
DATES:
The deadline for comments on this notice is June 18, 2025.
ADDRESSES:
Interested parties may submit written questions, comments, and supplementary documents by email to PathwaysClearinghouse@abtglobal.com with "Pathways to Work Evidence Clearinghouse 2025 FRN comment" in the subject line. To ensure that your comments have maximum effect, please identify clearly the section of the draft Protocol for the Pathways to Work Evidence Clearinghouse: Methods and Standards, Version 2.0 that your comments address.
Readers are referred to the full version of the draft Protocol for the Pathways to Work Evidence Clearinghouse: Methods and Standards, Version 2.0 (Protocol Version 2.0) on the Clearinghouse website ( https://pathwaystowork.acf.gov/sites/default/files/2025-05/Pathways_to_Work_Methods_and_Standards_Report_V2.0.pdf ).
SUPPLEMENTARY INFORMATION:
1.0 Background and Legislative Context
In response to the Consolidated Appropriations Act of 2017 (Pub. L. 115-31), the ACF Office of Planning, Research, and Evaluation (OPRE) established the Pathways to Work Evidence Clearinghouse (Pathways to Work). Pathways to Work seeks to be a comprehensive resource that a range of audiences, including State and local Temporary Assistance for Needy Families administrators, can use to identify the services that will best help people with low incomes succeed in the labor market. Pathways to Work carries out a systematic process to identify, review, and rate the quality of eligible studies of programs and policies that have the primary aim of improving the employment and earnings of people with low incomes. The process is implemented by trained reviewers using consistent and transparent standards and procedures. The current "Protocol for the Pathways to Work Evidence Clearinghouse: Methods and Standards Report (Protocol Version 1.0)" ( https://pathwaystowork.acf.gov/publication/ProtocolPathways ) provides a detailed description of the standards currently used to identify, review, and rate the quality of eligible studies and the procedures followed by Pathways to Work staff. The Protocol Version 1.0 was informed by the scope and protocols used in a prior OPRE-sponsored evidence review project, the Employment Strategies for Low-Income Adults Evidence Review, consultations with research and practice experts, and the review processes developed and used by other prominent federally funded evidence clearinghouses. The Protocol Version 1.0 was further informed by public comments submitted in response to Federal Register Notice 83 FR 26290 (June 6, 2018), here: https://www.federalregister.gov/d/2018-12160.
2.0 Revisions and Clarifications to the Pathways to Work Methods and Standards in the Draft Protocol Version 2.0
The draft Protocol Version 2.0 aims to revise existing standards and to add new standards for including and rating the quality of program cost studies. The draft Protocol Version 2.0 was informed by consultations with research and practice experts. Below is a comprehensive list of specific revisions proposed to Pathways to Work's existing methods and standards, and the new standards for reviewing program cost studies. Subsequent chapter and section numbers all refer to the chapter and section numbering for the draft Protocol Version 2.0, unless the text explicitly indicates a reference to Protocol Version 1.0 chapter and section numbering.
2.1 Chapter 1. Introduction
The revised introduction includes an update of the key terms used by Pathways to Work, reflecting engagement activities with key Pathways to Work audiences to enhance plain language terminology. The content of the section on updates to the report was moved to a revised Appendix A describing the evolution of the protocol and methods and standards.
2.2 Chapter 2. Identifying Eligible Studies
Revisions to the electronic citation database section (Section 2.1.3) include dropping the set of intent and geographic terms and enhancing the set of outcomes, sample, and design keywords to ensure greater comprehensiveness, informed by testing the sensitivity of results to these terms and expert consultations. The ProQuest Dissertations and Theses database was removed from the protocol based on very few eligible studies being identified through this database, though dissertations and theses remain eligible for review. The protocol for searching organizational websites is now in its own section (Section 2.1.4), and the list of organizational websites searched was updated (see Box 2.1). The Prevention Services Clearinghouse was added to the list of federal evidence review databases consulted (Section 2.1.5). The Protocol Version 2.0 indicates an intent to move the list of literature searches identified from an appendix in the protocol to the Pathways to Work website.
2.3 Chapter 3. Assessing a Study's Strength of Evidence
[top] Two key revisions were made to the standards for reviewing comparison-group, quasi-experimental designs (Section 3.2.5). First, in determining whether the study controlled for any potential differences in the outcome before the program, current standards require that the control for employment or earnings outcomes should be measured at least 1 year before the intervention. The revised standards allow for an exemption on this timing requirement in two specific cases: (1) studies where participants have been incarcerated for 1 year or longer prior to the program beginning; and (2) studies of minor youth who would not have had sufficient time to establish 1 year or more of work history ( e.g., a program designed for 16-year-olds with no prior work history). The timing requirement for the control to be measured at least 1 year before the program began may also potentially be waived in cases where authors provide explicit, credible evidence that participants in both the intervention and comparison conditions in the study could not have had a change in earnings in the year prior to the program beginning. Waiving the timing requirement requires the approval of the Pathways to Work principal investigator. Note that in all cases, this exemption applies only to the
Second, baseline equivalence standards were revised to permit use of effect sizes to establish baseline equivalence if there is a statistically significant difference on one of the required characteristics. If the baseline difference is statistically significant, reviewers then compute the effect size of the baseline difference, following the procedures specified in Appendix D of Protocol Version 2.0. If the effect size is less than 0.05 standard deviations, the baseline equivalence requirement is met. If the effect size is between 0.05 and 0.25 standard deviations, the study must control for the characteristic in their statistical model. If the effect size is greater than 0.25 standard deviations (or if an effect size cannot be computed), then the baseline equivalence requirement is not met. This revision is intended to address studies with large sample sizes where even very small differences in the magnitude of the baseline difference may be statistically significant.
2.4 Chapter 4. Assessing the Evidence of Effectiveness for a Program
Revisions were made to the program effectiveness ratings, where the criteria for the Insufficient Evidence to Assess Support rating were revised to include only cases with a single study where none of the findings are statistically significant. The criteria for Not Supported did not change but now include any cases with two or more studies that have a pattern of only null or unfavorable findings that do not meet the criteria for any other rating category. Lastly, the name of the rating category for cases with no high or moderate-quality studies identified that was formerly titled No Evidence to Assess Support has been changed to Cannot Assess Support for clarity.
2.6 Chapter 5. Assessing Cost Study Information for a Program
As the evidence base on employment programs for job seekers with low incomes continues to grow, so has the need for information about the costs of those programs and practices. Without additional information about the personnel/non-personnel resources used and the associated costs, it is impossible to provide guidance on the resources necessary for implementation and how best to allocate funding towards these efforts. Pathways to Work developed new standards for reviewing cost studies on employment and training programs designed for individuals with low incomes, with input from research and practice experts.
The draft standards cover three types of cost studies. Cost analysis provides an analysis of the comprehensive effort involved in program implementation, answering questions such as how much the program costs to implement, the per-participant cost of the program, and the feasibility of implementation given existing budget constraints and available resource inputs. Cost-effectiveness analysis compares the estimated cost of a program with an estimate of its impact on a given outcome of interest. Cost-benefit analysis compares the cost of a program with the monetized outcomes associated with that program.
The draft cost standards include two types of standards: (1) threshold standards that identify basic characteristics that cost studies must meet in order to rate the quality of the study's cost information, and (2) quality rating standards that are applied to studies that meet the threshold standards and provide additional information about how the study calculated program costs. Chapter 5 provides the threshold and quality standards for cost analyses. Studies that include a cost-effectiveness analysis and/or cost-benefit analysis will be reviewed and scored separately using the standards for each type of analysis, respectively, presented in Appendix E. A cost study rating will be assigned based on the threshold and quality rating standards. Cost studies that do not meet all the threshold standards receive a rating of Interpret cost findings with caution. Cost studies that meet all threshold standards are scored on quality rating standards on a 1 to 3 scale. A summative quality score is then generated by averaging the quality rating scores from each individual quality standard, which also ranges from 1 to 3, and rounding to the nearest tenth. A rating of Cost study meets standards with low quality will be assigned when the average score is between 1.0 and 1.5. A rating of Cost study meets standards with moderate quality will be assigned when the average score is between 1.6 and 2.5. A rating of Cost study meets standards with distinction will be assigned when the average score is between 2.6 and 3.0.
Detailed information about the proposed threshold standards for cost analyses can be found in Section 5.1 of the draft Protocol Version 2.0. Detailed information about the proposed cost analysis quality rating standards can be found in Section 5.2 of the draft Protocol Version 2.0. Detailed information about cost-effectiveness analysis and cost-benefit analysis threshold and quality standards can be found in Appendix E.
3.0 Timeline for the Pathways to Work To Apply New Methods and Standards
Pathways to Work proposes to apply the standards and procedures upon publication of a final Protocol Version 2.0. The public will be clearly notified on the Pathways to Work website and via appropriate dissemination channels when the final published Protocol Version 2.0 will go into effect.
4.0 Request for Information (RFI)
ACF invites comments regarding this notice on the draft "Protocol for the Pathways to Work Evidence Clearinghouse: Methods and Standards, Version 2.0" ( https://pathwaystowork.acf.gov/sites/default/files/2025-05/Pathways_to_Work_Methods_and_Standards_Report_V2.0.pdf ). To facilitate the review of comments submitted, please identify the chapter, section, and/or page number of the draft that your comments address. This RFI is for information and planning purposes only and should not be construed as a solicitation or as an obligation on the part of ACF or HHS. For more information about the Pathways to Work, visit: https://pathwaystowork.acf.gov/.
Lauren Supplee,
Deputy Assistant Secretary for Planning, Research, and Evaluation.
[FR Doc. 2025-08842 Filed 5-16-25; 8:45 am]
BILLING CODE 4184-09-P