Organizational level indicators to address health equity work in local public health agencies: a scoping review.

Salter, Katherine ; Salvaterra, Rosana ; Antonello, Deborah 等

The need to address growing health inequities - those unjust and avoidable differences in health between groups who have varied levels of social privilege related to factors such as wealth, power, education, gender or ethnicity - is an increasingly articulated global health priority. (1-3) Reducing these inequities will require strengthening the social, economic and environmental factors that influence health - commonly referred to as social determinants of health (SDoH) (4) - and eliminating their unequal distribution. (5) While root causes of health inequities will need to be tackled in broad policy arenas, essential areas of equity action for the health sector have been identified, and addressing the SDoH is viewed as fundamental to the design and provision of effective public health programs and activities. (1-3,5-8) However, addressing the SDoH, particularly as they pertain to issues of inequity, can be challenging as health inequities are often multi-factorial, values-oriented, and the result of complex relationships between social hierarchy, economic and financial restraint, and political ideology. (9) Given the complexities involved in addressing inequities in public health, it has become important to not only evaluate pertinent health outcomes, but also the strategies, processes and related activities of the local public health agencies responsible for implementing public health programs.

Assessing the performance of a health system, including the public health system, is imperative to ensure its functionality and success. Steps to reduce inequities in health have been identified through three principles: improve the conditions of daily life; tackle inequitable distribution of power; and measure the problem and evaluate action. (1,10) Most public health systems have tools in place to monitor and evaluate the functioning of the health system; (11) however, performance indicators, as they relate to how local public health agencies address issues of inequity, are far less common. Health equity performance indicators would help local public health agencies better understand their work in recognizing and addressing inequities that exist within their local populations.

Background

Ontarians are served by a system of 36 local public health agencies, each governed by a Board of Health that is responsible for the population within its geographic borders. These Boards are mandated to protect and promote the health of their communities under the Health Protection and Promotion Act and are led by a medical officer of health. (12) The release of the Ontario Public Health Standards (OPHS) in 2008 represented a step forward in the management of public health programs and services through the introduction of a revitalized performance management framework. (7) The OPHS established mandatory requirements for fundamental public health programs as well as provided a framework of foundational principles and standards intended to provide guidance toward the achievement of articulated program standards and requirements in support of improved accountability, equitable access, and the evaluation of public health programs and services. (7,13) Within the OPHS, addressing the SDoH to reduce health inequities is considered fundamental to the work of each local public health agency. Work to address health equity issues should thus be included in strategic organizational planning and addressed in the delivery and outcome evaluation across all programs and services. (7,14) However, recent research has reported a lack of consensus in practice regarding the way in which local public health agencies could and should address the social determinants and tackle health inequities at the local level. (15,16) While analysis and action on the SDoH "should be an integral part of normal practice of public health organizations and staff' (p. 24) in Canada, reported activity around health equity and SDoH has not been widespread. (8) Broad requirements for action have not been accompanied by guidance regarding implementation or assessment of whether efforts of local public health agencies are appropriate or sufficient to meet Board of Health responsibilities to address health inequities.

In 2013, a joint workgroup of the Association of Local Public Health Agencies (alPHa) and the Ontario Public Health Association (OPHA) developed an initial set of indicators derived from the requirements of the OPHS and the Ontario Public Health Organizational Standards. (17) However, the process of the workgroup was informal, and did not include a systematic literature review. In 2014, the local public health agency representatives engaged in the project reported here expressed a need for a set of evidence-based indicators, developed and tested in the context of public health practice in Ontario.

Research objective

The purpose of this literature review was to determine what organizational level indicators exist that could be used to monitor and guide the progress of local public health agencies in addressing the SDoH and reducing health inequities, as required by the OPHS and Ontario Public Health Organizational Standards.

METHOD

Our method was based on the six-stage standard scoping study framework proposed by Arksey and O'Malley, within which we a) identified a research objective (see above), b) identified relevant studies, c) selected studies, d) charted data, e) collated, summarized and reported results, and f) carried out a consultation exercise. (18)

Study identification

Relevant search terms were defined by a working group in consultation with a Health Equity Specialist working within a local public health agency. The search strategy featured key terms built around "public health" as well as measurement terms (indicators, evaluation, assessment, markers, and other variants), and terms focused on health inequity (e.g., health inequality, social determinants of health, social isolation, disparity). The complete strategy is provided in Supplementary File A (see the Article Tools section of the journal site for all supplementary files). Three electronic databases (i.e., PubMed, CINAHL and Scopus) were searched for relevant, English language, peer-reviewed literature published between January 2002 and May 2014. Systematic searches for relevant grey literature were conducted in the Canadian Health Policy Research Collection and desLibris databases. A research librarian affiliated with the project conducted Google-based custom searches for US State Government information and of Ontario Public Health Unit websites. In addition, documents were contributed from the files of the public health and academic expert members of the research team. The same time frame (January 2002-May 2014) was used to search grey literature sources as peer-reviewed. This period was chosen as it coincides with the increase in interest around health equity.

Study selection

Reports, both from peer-reviewed or grey sources, were considered for inclusion in the current review if a) they stated a research objective, aim or purpose within the areas of health equity, health inequality, health disparities, priority populations, vulnerable groups, and/or SDoH and b) the report addressed assessment or evaluation of health equity or SDoH policy, programs or organizational level activities or c) they identified/developed indicators intended for the evaluation of health equity or SDoH impacts. Reports examining population-level indicators only were excluded as they were deemed to lack sensitivity and specificity for actions limited to organizations within the public health sector. Reports with insufficient reporting details to support understanding of method or facilitate data abstraction were also excluded (e.g., editorials, commentary, conference proceedings, and abstracts) as were non-English language publications; reports were not assessed for quality. Identified articles were assessed for inclusion first by title and abstract review (round 1) and then by screening the full text of the articles (round 2). Assessment for inclusion was performed by two members of the research team on each round, and disagreements were resolved through discussion with another team member.

Charting the data

The research team collaborated to create a single, standard abstraction form to be used by team members to facilitate consistent data collection practices. In addition to the author, source and context, any identified indicators (description/ definition), development method and testing of indicators (e.g., feasibility, reliability, validity), suggestions for evaluation of reported indicators as well as additional information relevant to future indicator development or testing were recorded. For the purposes of this review, indicators and factors for future indicator development or testing recorded during data extraction will be referred to simply as indicators.

Collating, summarizing, reporting

Indicators, either developed or recommended for future development, were recorded. These indicators were reviewed and then grouped together by theme. In all, 12 broad thematic categories were identified as follows: 1) cultural competency; 2) quality of care and health services; 3) mental health; 4) elder health; 5) length of life; 6) illness, disease, injury or wellness; 7) health risks and behaviours; 8) gender equity and women's health; 9) assessment (of public health programs and services); 10) physical environment; 11) income, social status and education; and 12) civic engagement and areas for collaboration.

The list of indicators and identified factors for development were examined by a working group from within the research team to select the items perceived as most relevant for examination of health equity activity at an organizational level. If consensus could not be achieved among working group members regarding inclusion of an indicator, it was discussed within the larger research group. The selected indicators were then categorized according to four key organizational roles for public health action to reduce health inequities. The roles were identified and validated in a national consultation with the Canadian public health community by the National Collaborating Centre for Determinants of Health (NCCDH):8 1) assess and report on the health of populations describing health inequalities and inequities; 2) modify and orient public health interventions in consideration of the unique needs and capacities of priority populations; 3) engage in community and multisectoral collaboration; and 4) lead/ participate and support other stakeholders in policy analysis, development and advocacy.

Consultation

To gather additional information, and seek out other perspectives regarding the meaning and applicability of the preliminary results of the review and summary process described above, we elected to conduct a consultation process. (18,19) Provincial, national and international health equity and indicator development experts were identified using existing contacts within the research team to supplement key authors highlighted during the literature review process. Potential participants were invited to engage in individual, one-hour long, semi-structured interviews, to examine items extracted from the literature for issues of face validity within the public health role framework and to offer opinions regarding relative importance, possible assessment gaps and recommended areas for indicator development. Invitations were extended by e-mail to a total of 18 possible participants worldwide and 13 telephone interviews were conducted. Prior to each interview, participants received an information package containing a description of the shortlisted items and were asked to consider the importance, usefulness and NCCDH role classification of each item. All interviews were recorded and transcripts produced verbatim. Analysis of transcript data was performed using NVivo [R] software (version 10) by two members of the research team. A summary document presenting results of this analysis for each indicator and role, noting general role comments, specific indicator concerns and potential gaps in assessment, was generated to support a team discussion of the review results and facilitate the revision of the indicator short list.

Ethics approval

This research received formal ethics approval from Western University (ref: HSREB 105503). All interview participants provided informed consent prior to participation.

RESULTS

A total of 10 254 records were identified through the search for peer-reviewed articles, after the removal of all duplicates. After all records were screened and assessed for eligibility, a total of 18 peer-reviewed studies were included in the synthesis process. The process of identifying peer-reviewed articles for inclusion is documented in Figure 1. Searches of the Canadian Health Policy Research Collection database and the desLibris database identified a total of 228 possible items for inclusion. Custom searches conducted by the research librarian and document references provided by public health team members provided another 46 possible items. All items were screened over 2 rounds, as with the peer-reviewed articles. In the first round, 164 articles were excluded based on a review of titles, and abstracts or executive summaries where these were available. In round 2, 8 additional articles were identified from handsearching; however, based on a review of complete texts, an additional 88 articles were excluded (see Figure 2). The remaining 30 articles were identified for inclusion in the review. A complete list of all articles and reports included is provided in Supplementary File B.

One hundred and seventy-two indicators were collected from the peer-reviewed literature and an additional 302 indicators were collected from grey literature sources and classified according to the categories described previously. A description of the indicators recorded from the identified grey and peer-reviewed literature sources is provided in Table 1.

In selecting a short list of indicators, the working group determined that many of the recorded indicators were restricted to health outcomes often measured at the community or population level (e.g., infectious disease rates, preventable hospitalization rates, birth outcomes, maternal mortality, etc.) and did not necessarily capture organizational or externally-focused public health activities. All such indicators were excluded. Team discussion of the remaining indicators included issues around a) how a given concept could be translated into a measurable indicator of health equity process at the level of the organization, b) whether a given indicator could be altered to highlight equity-seeking populations, and c) whether a proposed indicator was within public health's capacity to measure or monitor, particularly at the level of the local public health agency. Selected indicators, including those developed by the joint working group of alPHa-OPHA, were also examined, and classified, in terms of their applicability within the role framework proposed by the NCCDH. (8)

Consultation feedback

From the written feedback provided and the recorded transcripts of the interviews, summaries were created to highlight feedback related to indicator importance, recommendations for development, indicator retention, and role classification. The short list of indicators (n = 28) and brief key informant feedback are provided in Table 2.

The most common reasons offered for removal of an indicator by the key informants were that 1) it was not an indicator of the performance of equity-related activity at the level of an organization, or 2) it was redundant, i.e., being assessed as part of another indicator. Indicators that were considered for retention were all noted as requiring significant clarification, definition and development. Additionally, a number of indicators (see Table 2) were identified as having a focus on organization systems, capacity and support for equity activities. Expert informants urged consideration of an additional category or role to accommodate these and any other similar indicators that might be developed. The process of consultation and review prompted the creation of a revised list of possible indicators, suggestions for development, as well as a list of assessment gaps to be addressed within each role (Table 3). In response to the expert feedback, an additional category, entitled "Organizational and System Development" was added to reflect the approaches to and means through which local health agencies may strengthen and fulfill activities within each of the other public health roles. (8,20)

DISCUSSION

The current project spans several phases of research. In this first phase, we built upon recent reports from the NCCDH and OPHA/ alPHa (8,17) by conducting a scoping review of peer-reviewed and grey literatures in order to identify indicators that a) currently exist and b) could be used to monitor and guide progress toward fulfillment of public health roles at the level of the local public health agency.

The field of public health services research, as a whole, is just emerging as an area of concentrated interest, thus it was unsurprising that relatively few relevant articles were located. Many of the indicators identified initially represented an association between public health performance and health status outcomes. While health outcome measures are useful to public health agencies to help them identify local populations' risk of experiencing inequity, for the purposes of evaluating performance in addressing inequity, process measures tend to be more sensitive than outcome measures to differences in quality of care. (21) Performance or process indicators can be used to demonstrate practice trends, showcase excellence and highlight areas that need improvements over time. Further, while public health agencies have a role in addressing population health outcomes through collaborations with other organizations within the health system, they cannot be held accountable for these outcomes. However, public health agencies' efforts to lead, support and participate in larger system-level efforts to improve population health outcomes as they relate to health equity can be measured. We have thus included these types of indicators in Role 4 - Lead, Support, Participate.

The findings from this research ought to be considered in light of some limitations. Articles outside of the search parameters, such as non-English ones, might have contributed further information. Further, discussions and analysis were done with a background focus related to the public health system in Ontario; interpretation of findings might be different in the context of a widely dissimilar system. Population health indicators related to outcomes were considered out of scope for this review, in part because the public health system is not solely responsible for these outcomes. Future efforts might identify those system-level outcomes that fall solely within the purview of public health.

Given that proper assessment and evaluation are discussed as crucial components of tackling issues of inequity in public health in both the NCCDH documents and the OPHS, the need for performance measures is even more imperative. Our review of the literature revealed few evidence-based, validated indicators that could be used by local Boards of Health to monitor and guide progress to address health inequities. Further, consultation with key experts suggested that, while the assessment of organizational level activity is important, there is a need for continued development of these indicators, including careful operationalization of concepts and establishing clear definitions for key terms. In addition, attention should be paid to the identified assessment gaps within each of the public health roles.

CONCLUSION

The literature, and indicators presented within it, do not always reflect the health equity activity in practice at local public health agencies. Recent reports have provided important examples in which Ontario public health agencies are working on the leading edge of practice to address health inequity. (22-24) Moving forward, our challenge is to build upon the selected indicators identified in the current review to derive a set of evidence-informed, organizational performance indicators that can be used to reflect the health equity activity within local public health agencies and to help guide future activity within each of the identified public health roles.

This work showed that few robust indicators were available. In the next phase of research, results from the scoping review, integrated with the feedback from expert informants, were used to inform the development of a set of indicators for testing in a sample of local public health agencies in Ontario. In addition, based on rich learnings and input from test sites and public health practitioners, a final set of indicators and a user guide have been developed to support their use. (25)

doi: 10.17269/CJPH.108.5889

REFERENCES

(1.) World Health Organization. Commission on Social Determinants of Health: Closing the Gap in a Generation. Final Report. Geneva, Switzerland: WHO, 2008.

(2.) Baum FE, Begin M, Houweling TAJ, Taylor S. Changes not for the fainthearted: Reorienting health care systems toward health equity through action on the social determinants of health. Am J Public Health 2009; 99(11):1967-74. PMID: 19762660. doi: 10.2105/AJPH.2008.154856.

(3.) Rasanathan K, Montesinos EV, Matheson D, Etienne C, Evans T. Primary health care and the social determinants of health: Essential and complementary approaches for reducing inequities in health. J Epidemiol Community Health 2011; 65(8):656-60. PMID: 19933684. doi: 10.1136/jech. 2009.093914.

(4.) World Health Organization. Social Determinants of Health: Key Concept. 2016. Available at: http://www.who.int/social_determinants/thecommission/ finalreport/key_concepts/en/ (Accessed July 28, 2016).

(5.) Graham H. Social determinants and their unequal distribution: Clarifying policy understandings. Milbank Q 2004; 82(1):101-24. PMID: 15016245.

(6.) McPherson C, McGibbon E. Intersecting contexts of oppression within complex public systems. In: Pycroft A, Bartollas C (Eds.), Applying Complexity Theory: A Whole Systems Approach to Criminal Justice and Social Work. Bristol, UK: Policy Press, 2014; 159-80.

(7.) Ontario Ministry of Health and Long-Term Care, Public Health Division. Ontario Public Health Standards. Toronto, ON: Queen's Printer for Ontario, 2008. Available at: http://www.health.gov.on.ca/en/pro/programs/ publichealth/oph_standards (Accessed July 10, 2016).

(8.) National Collaborating Centre for Determinants of Health (NCCDH). Integrating Social Determinants of Health and Health Equity into Canadian Public Health Practice: Environmental Scan 2010. Antigonish, NS: NCCDH, 2011. Available at: http://nccdh.ca/images/uploads/Environ_Report_EN.pdf (Accessed July 27, 2016).

(9.) Reutter L, Kushner KE. Health equity through action on the social determinants of health: Taking up the challenge in nursing. Nurs Inq 2010; 17(3):269-80. PMID: 20712665. doi: 10.1111/j.1440-1800.2010.00500.x.

(10.) Marmot M, Friel A, Bell R, Houweling TA, Taylor S; Commission on Social Determinants of Health. Closing the gap in a generation: Health equity through action on the social determinants of health. Lancet 2008; 372:166-169. PMID: 18994664. doi: 10.1016/S0140-6736(08)61690-6.

(11.) Kruk ME, Freedman LP. Assessing health system performance in developing countries: A review of the literature. Health Policy 2008; 85(3):263-76. PMID: 17931736. doi: 10.1016/j.healthpol.2007.09.003.

(12.) Ontario Ministry of Health and Long-Term Care. Health Services in Your Community: Public Health Units. Toronto, ON: OMHLTC, 2014. Available at: http://www.health.gov.on.ca/en/common/system/services/phu/ (Accessed July 27, 2016).

(13.) Pinto AD, Manson H, Pauly B, Thanos J, Parks A, Cox A. Equity in public health standards. A qualitative document analysis of policies from two Canadian provinces. Int J Equity Health 2012; 11:28. PMID: 22632097. doi: 10. 1186/1475-9276-11-28.

(14.) Ontario Ministry of Health and Long-Term Care. Ontario Public Health Organizational Standards. Toronto, ON: OMHLTC, 2011. Available at: http:// www.health.gov.on.ca/en/pro/programs/publichealth/orgstandards/docs/ org_stds.pdf (Accessed July 27, 2016).

(15.) Tran N, Moffat H, Tyler I, Kothari A. Advancing health equity through organizational standards and competencies: Examples of this promising practice in Ontario. Paper presented at: The Ontario Public Health Convention (TOPHC), Toronto, ON, 2013.

(16.) Manson H, Valaitis R, Bursey G, Dobbins M, Etches V, Garcia J, et al. Implementation of the Ontario public health standards in six Ontario health units: Key results and responses from provincial and health unit stakeholders. Paper presented at: The Ontario Public Health Conference, Toronto, ON, 2012.

(17.) Association of Local Public Health Agencies/Ontario Public Health Association Health Equity Workgroup. Health Equity Indicators. Toronto, ON: Association of Local Public Health Agencies/Ontario Public Health Association Health Equity Workgroup, 2013.

(18.) Arksey H, O'Malley L. Scoping studies: Towards a methodological framework. Int J Soc Res Methodol 2005; 8(1):19-32.

(19.) Levac D, Colquhoun H, O'Brien KK. Scoping studies: Advancing the methodology. Implement Sci 2010; 5:69. PMID: 20854677. doi: 10.1186/ 1748-5908-5-69.

(20.) National Collaborating Centre for Determinants of Health (NCCDH). Let's Talk Public Health Role for Improving Health Equity. Antigonish, NS: St. Francis Xavier University, 2013. Available at: http://nccdh.ca/resources/entry/letstalk- public-health-roles (Accessed July 27, 2016).

(21.) Mant J. Process versus outcome indicators in the assessment of quality health care. Int J Qual Health Care 2001; 13(6):475-80. PMID: 11769750.

(22.) Sudbury and District Health Unit. 10 Promising Practices to Guide Local Public Health Practice to Reduce Social Inequities in Health: Technical Briefing. Sudbury, ON: SDHU, 2011. Available at: www.sdhu.com (Accessed July 27, 2016).

(23.) Wellington-Dufferin-Guelph Public Health. Addressing Social Determinants of Health in Wellington-Dufferin-Guelph: A Public Health Perspective on Local Health, Policy and Program Needs. Guelph, ON: WDGPH, 2013. Available at: www.wdgpublichealth.ca/reports (Accessed July 27, 2016).

(24.) Johnston J, Maxwell T. No Barriers Health Equity for All: Toolkit and Practical Guide for Health and Community Service Providers. Windsor, ON: Windsor-Essex County Health Unit, 2015. Available at: www.wechu.org (Accessed July 27, 2016).

(25.) Salvaterra R, Cohen B, Antonello D, LeBer MJ, Kothari A, Lemieux S, et al. Indicators to guide health equity work in local public health agencies: A locally driven collaborative project in Ontario. J Health Promot Chronic Dis Prev 2017; submitted.

Received: September 16, 2016

Accepted: March 3, 2017

Katherine Salter, MSc, [1] Rosana Salvaterra, MD, MSc, [2] Deborah Antonello, BScN, MN, [3] Benita E. Cohen, PhD,[4] Anita Kothari, PhD, [1,5] Marlene Janzen LeBer, PhD, [6] Suzanne LeMieux, MA, PhD, [7] Kathy Moran, MHSc, [8] Katherine Rizzi, MHIS, [5] Jordan Robson, MSc, [3] Caroline Wai, MHSc [9]

Author Affiliations

[1.] Health and Rehabilitation Sciences, Western University, London, ON

[2.] Peterborough County-City Health Unit, Peterborough, ON

[3.] Algoma Public Health, Sault Ste. Marie, ON

[4.] Faculty of Health Sciences, University of Manitoba, Winnipeg, MB

[5.] School of Health Studies, Faculty of Health Sciences, Western University, London, ON

[6.] School of Leadership & Social Change, Brescia University College, Western University, London, ON

[7.] Sudbury and District Health Unit, Sudbury, ON

[8.] Durham Region Health Department, Whitby, ON

[9.] Toronto Public Health, Toronto, ON

Correspondence: Katherine Salter, PhD(C), c/o Graduate Program, Health and Rehabilitation Sciences, Elborn College, Western University, London, ON N6A 1H1, E-mail: [email protected]

Conflict of Interest: None to declare.

Caption: Figure 1. Identification of articles for inclusion in the review

Caption: Figure 2. Identification of grey literature for inclusion in the review Table 1. Abstracted indicators Category Number of indicators recorded Grey Peer- literature reviewed Cultural competency 9 4 Quality of care and 28 30 health services Mental health 9 2 Length of life 25 12 Illness, disease, injury, and wellness 34 30 Health risks and 64 7 behaviours Gender equity and 24 3 women's health Assessment 30 15 Physical environment 19 14 Income, social status, 45 36 education Civic engagement and 14 14 areas for collaboration Category Description of indicators retrieved Cultural competency * Number and types of anti-racism policies, percentages of families (in a specific location) that use English as a second language, recommendations re: levels and use of Aboriginal languages and reduction of language barriers in order to improve equitable access to services, spirituality, reflectiveness, and responsiveness to cultural norms or practices. Quality of care and * Various rates (e.g., infectious health services disease) in addition to service delivery markers such as time from referral to assessment and time from assessment to treatment. Recommendations included monitoring of patient experience as well as additional assessment of utilization patterns. Mental health * Measurements such as depression prevalence, number of suicide deaths, perceived work-related stress, as well as recommendations to reduce depression and anxiety rates among youth. No items from this category were shortlisted. Length of life * All indicators identified were health Illness, disease, injury, outcome indicators only. and wellness * Disease prevalence and incidence rates; recommendations for indicator development included a health and wellness plan targeting Aboriginal groups. Health risks and * Developed due to the correlation behaviours between items in this category and social determinants of health. Indicators identified in this category were those attempting to monitor risks and behaviours, such as smoking rates, obesity rates, alcohol consumption rates, drug use rates, and recommendations to measure lifestyle choices that have health-related outcomes. Gender equity and * Monitoring of maternal health issues women's health (e.g., maternity services available, gestational diabetes rates, percentage of mothers screened for postpartum depression, or prevalence of folic acid use in peri- conception period). Recommendations for indicator development suggested monitoring health needs by gender and conducting routine gender equity analyses. Assessment * All except 4 indicators were recommendations for indicator development; few indicators that were retrieved from the literature (i.e., "measures of administration efficiency" and "measures of efficacy, efficiency, and quality improvement") were found lacking in description and definition. Physical environment * Affordable housing rates, homelessness rates, living arrangements; recommendations to evaluate the quality of housing, monitor violence and crime rates according to geographic location, evaluation of built environments, use of health impact assessment tools, reduction of geographical maldistribution of services and supplies. Income, social status, * High school attendance and graduation education rates, number of families that have access to quality child care, employment rates, literacy rates, and percentage of income used for rent. * Income security, poverty rates, bankruptcy rates, number of single-parent households, immigration status, as well as recommendations to a) understand financial and non-financial barriers to access and b) identify and understand existing political processes and power relationships. Civic engagement and * Recommendations included recording the areas for collaboration number of inter/intra-community programs, monitoring the involvement of youth and elders in community decision-making, encouraging partnership development, developing cross-sectoral collaborations, enabling community empowerment to support marginalized groups, supporting partnered decision making. Table 2. Shortlisted indicators and feedback from expert informants Role (8) Shortlisted indicator Feedback Role 1--Assessing Measurement of the percentage of Retain and reporting on families with English as a health status and second language. what could be done to improve it Comparison of your Reassign organization's workforce diversity with the population diversity. Measurement of the percentage of Remove item children who have completed recommended vaccination programs. Measurement of percentage of Remove item elders who are offered fall prevention awareness initiatives. Measurement of diabetes rate. Remove item Measurement of the number of Reassign clients registered in methadone maintenance programs. Implementation of a needle Reassign exchange program that is located in the higher-need areas. Use of health impact assessment Reassign tools. Monitoring (the percentage of) Retain Board of Health reports on health statuses that include disaggregation of data by social determinants of health where possible Role 2--Modify/ The number of current culturally Retain reorient public sensitive policies, programs or health programs strategies employed; the type of culturally sensitive policies, programs or strategies employed; the perceived effectiveness of culturally sensitive policies, programs or strategies. Assessment of whether the Retain organization is conducting equity-focused performance assessments; use of pre-existing Health Equity Assessment tools. Use of indicators that are Remove item specific to issues of importance to Aboriginal communities. Assessment of whether the Remove item organization is conducting gender and equity analysis for the purpose of program planning. Assessment of strategies used to Retain support opportunities to increase the capacity of underserved populations. Assessment of plans for capacity Reassign building with relevant staff in population health thinking (e.g., through education and training).* Employment of a mechanism to Reassign ensure that operational planning includes a health equity assessment of programs and services.* Evaluation of how programs and Reassign services have changed or been developed based on the health equity assessment.* Following a strategic plan that Reassign describes how equity will be addressed.* Following a current operational Reassign plan that incorporates the identification and planning for priority populations? If yes, what is the process?* Role 3--Engage in Work/efforts/strategies to Reassign community and reduce language barriers to multisectoral equitable access. collaboration Strategies for the development Retain of community capacity. Involvement of vulnerable youth Remove item populations in community decision-making. Involvement of elder populations Remove item in community decision-making. Participation (by the Retain organization) in local poverty reduction efforts. Role 4--Lead/ The number of new diversity and Remove items support/participate anti-racist policies; the types with others of new diversity and anti-racist policies; the perceived effectiveness of the new diversity and anti-racist policies. The number and type of diversity Reassign among the organization workforce, especially managers in proportion to the diversity in the general population. The number of community needs Reassign assessments that have been conducted (within an assigned period of time). Note: Indicators appearing in bold were obtained from alPHa-OPHA. (17) Items followed by an asterisk were identified by the content experts as best fitting an additional organizational or systems category to be developed. Table 3. Gaps in assessment by role as identified by expert informants Indicators or concepts identified as missing by the expert participants Role 1--Assessing and * Indicators that address engagement of the reporting on health community or provision of results to foster status and community discussion, etc. what could be done to * Early childhood education improve it * Disaggregation of data by SDoH/attention to SDoH and equity issues Role 2--Modify/reorient * Standardized, precise language public health programs * Address issues of importance in a greater number of identifiable communities * There are no health outcomes listed * No engagement with the community * Understanding of existing services * Representation of underlying values Role 3--Engage in * Representation of underlying values community * Collaborations/partnerships; stakeholders and multisectoral * Who is vulnerable; who is affected by collaboration health equity? * Early childhood education Role 4--Lead/support/ * Types of advocacy activities, participate collaborations * Indicators focused more directly on policy development