The need to address growing health inequities - those unjust and avoidable differences in health between groups who have varied levels of social privilege related to factors such as wealth, power, education, gender or ethnicity - is an increasingly articulated global health priority. (1-3) Reducing these inequities will require strengthening the social, economic and environmental factors that influence health - commonly referred to as social determinants of health (SDoH) (4) - and eliminating their unequal distribution. (5) While root causes of health inequities will need to be tackled in broad policy arenas, essential areas of equity action for the health sector have been identified, and addressing the SDoH is viewed as fundamental to the design and provision of effective public health programs and activities. (1-3,5-8) However, addressing the SDoH, particularly as they pertain to issues of inequity, can be challenging as health inequities are often multi-factorial, values-oriented, and the result of complex relationships between social hierarchy, economic and financial restraint, and political ideology. (9) Given the complexities involved in addressing inequities in public health, it has become important to not only evaluate pertinent health outcomes, but also the strategies, processes and related activities of the local public health agencies responsible for implementing public health programs.

Assessing the performance of a health system, including the public health system, is imperative to ensure its functionality and success. Steps to reduce inequities in health have been identified through three principles: improve the conditions of daily life; tackle inequitable distribution of power; and measure the problem and evaluate action. (1,10) Most public health systems have tools in place to monitor and evaluate the functioning of the health system; (11) however, performance indicators, as they relate to how local public health agencies address issues of inequity, are far less common. Health equity performance indicators would help local public health agencies better understand their work in recognizing and addressing inequities that exist within their local populations.

Background

Ontarians are served by a system of 36 local public health agencies, each governed by a Board of Health that is responsible for the population within its geographic borders. These Boards are mandated to protect and promote the health of their communities under the Health Protection and Promotion Act and are led by a medical officer of health. (12) The release of the Ontario Public Health Standards (OPHS) in 2008 represented a step forward in the management of public health programs and services through the introduction of a revitalized performance management framework. (7) The OPHS established mandatory requirements for fundamental public health programs as well as provided a framework of foundational principles and standards intended to provide guidance toward the achievement of articulated program standards and requirements in support of improved accountability, equitable access, and the evaluation of public health programs and services. (7,13) Within the OPHS, addressing the SDoH to reduce health inequities is considered fundamental to the work of each local public health agency. Work to address health equity issues should thus be included in strategic organizational planning and addressed in the delivery and outcome evaluation across all programs and services. (7,14) However, recent research has reported a lack of consensus in practice regarding the way in which local public health agencies could and should address the social determinants and tackle health inequities at the local level. (15,16) While analysis and action on the SDoH "should be an integral part of normal practice of public health organizations and staff' (p. 24) in Canada, reported activity around health equity and SDoH has not been widespread. (8) Broad requirements for action have not been accompanied by guidance regarding implementation or assessment of whether efforts of local public health agencies are appropriate or sufficient to meet Board of Health responsibilities to address health inequities.

In 2013, a joint workgroup of the Association of Local Public Health Agencies (alPHa) and the Ontario Public Health Association (OPHA) developed an initial set of indicators derived from the requirements of the OPHS and the Ontario Public Health Organizational Standards. (17) However, the process of the workgroup was informal, and did not include a systematic literature review. In 2014, the local public health agency representatives engaged in the project reported here expressed a need for a set of evidence-based indicators, developed and tested in the context of public health practice in Ontario.

Research objective

The purpose of this literature review was to determine what organizational level indicators exist that could be used to monitor and guide the progress of local public health agencies in addressing the SDoH and reducing health inequities, as required by the OPHS and Ontario Public Health Organizational Standards.

METHOD

Our method was based on the six-stage standard scoping study framework proposed by Arksey and O'Malley, within which we a) identified a research objective (see above), b) identified relevant studies, c) selected studies, d) charted data, e) collated, summarized and reported results, and f) carried out a consultation exercise. (18)

Study identification

Relevant search terms were defined by a working group in consultation with a Health Equity Specialist working within a local public health agency. The search strategy featured key terms built around "public health" as well as measurement terms (indicators, evaluation, assessment, markers, and other variants), and terms focused on health inequity (e.g., health inequality, social determinants of health, social isolation, disparity). The complete strategy is provided in Supplementary File A (see the Article Tools section of the journal site for all supplementary files). Three electronic databases (i.e., PubMed, CINAHL and Scopus) were searched for relevant, English language, peer-reviewed literature published between January 2002 and May 2014. Systematic searches for relevant grey literature were conducted in the Canadian Health Policy Research Collection and desLibris databases. A research librarian affiliated with the project conducted Google-based custom searches for US State Government information and of Ontario Public Health Unit websites. In addition, documents were contributed from the files of the public health and academic expert members of the research team. The same time frame (January 2002-May 2014) was used to search grey literature sources as peer-reviewed. This period was chosen as it coincides with the increase in interest around health equity.

Study selection

Reports, both from peer-reviewed or grey sources, were considered for inclusion in the current review if a) they stated a research objective, aim or purpose within the areas of health equity, health inequality, health disparities, priority populations, vulnerable groups, and/or SDoH and b) the report addressed assessment or evaluation of health equity or SDoH policy, programs or organizational level activities or c) they identified/developed indicators intended for the evaluation of health equity or SDoH impacts. Reports examining population-level indicators only were excluded as they were deemed to lack sensitivity and specificity for actions limited to organizations within the public health sector. Reports with insufficient reporting details to support understanding of method or facilitate data abstraction were also excluded (e.g., editorials, commentary, conference proceedings, and abstracts) as were non-English language publications; reports were not assessed for quality. Identified articles were assessed for inclusion first by title and abstract review (round 1) and then by screening the full text of the articles (round 2). Assessment for inclusion was performed by two members of the research team on each round, and disagreements were resolved through discussion with another team member.

Charting the data

The research team collaborated to create a single, standard abstraction form to be used by team members to facilitate consistent data collection practices. In addition to the author, source and context, any identified indicators (description/ definition), development method and testing of indicators (e.g., feasibility, reliability, validity), suggestions for evaluation of reported indicators as well as additional information relevant to future indicator development or testing were recorded. For the purposes of this review, indicators and factors for future indicator development or testing recorded during data extraction will be referred to simply as indicators.

Collating, summarizing, reporting

Indicators, either developed or recommended for future development, were recorded. These indicators were reviewed and then grouped together by theme. In all, 12 broad thematic categories were identified as follows: 1) cultural competency; 2) quality of care and health services; 3) mental health; 4) elder health; 5) length of life; 6) illness, disease, injury or wellness; 7) health risks and behaviours; 8) gender equity and women's health; 9) assessment (of public health programs and services); 10) physical environment; 11) income, social status and education; and 12) civic engagement and areas for collaboration.

The list of indicators and identified factors for development were examined by a working group from within the research team to select the items perceived as most relevant for examination of health equity activity at an organizational level. If consensus could not be achieved among working group members regarding inclusion of an indicator, it was discussed within the larger research group. The selected indicators were then categorized according to four key organizational roles for public health action to reduce health inequities. The roles were identified and validated in a national consultation with the Canadian public health community by the National Collaborating Centre for Determinants of Health (NCCDH):8 1) assess and report on the health of populations describing health inequalities and inequities; 2) modify and orient public health interventions in consideration of the unique needs and capacities of priority populations; 3) engage in community and multisectoral collaboration; and 4) lead/ participate and support other stakeholders in policy analysis, development and advocacy.

Consultation

To gather additional information, and seek out other perspectives regarding the meaning and applicability of the preliminary results of the review and summary process described above, we elected to conduct a consultation process. (18,19) Provincial, national and international health equity and indicator development experts were identified using existing contacts within the research team to supplement key authors highlighted during the literature review process. Potential participants were invited to engage in individual, one-hour long, semi-structured interviews, to examine items extracted from the literature for issues of face validity within the public health role framework and to offer opinions regarding relative importance, possible assessment gaps and recommended areas for indicator development. Invitations were extended by e-mail to a total of 18 possible participants worldwide and 13 telephone interviews were conducted. Prior to each interview, participants received an information package containing a description of the shortlisted items and were asked to consider the importance, usefulness and NCCDH role classification of each item. All interviews were recorded and transcripts produced verbatim. Analysis of transcript data was performed using NVivo [R] software (version 10) by two members of the research team. A summary document presenting results of this analysis for each indicator and role, noting general role comments, specific indicator concerns and potential gaps in assessment, was generated to support a team discussion of the review results and facilitate the revision of the indicator short list.

Ethics approval

This research received formal ethics approval from Western University (ref: HSREB 105503). All interview participants provided informed consent prior to participation.

RESULTS

A total of 10 254 records were identified through the search for peer-reviewed articles, after the removal of all duplicates. After all records were screened and assessed for eligibility, a total of 18 peer-reviewed studies were included in the synthesis process. The process of identifying peer-reviewed articles for inclusion is documented in Figure 1. Searches of the Canadian Health Policy Research Collection database and the desLibris database identified a total of 228 possible items for inclusion. Custom searches conducted by the research librarian and document references provided by public health team members provided another 46 possible items. All items were screened over 2 rounds, as with the peer-reviewed articles. In the first round, 164 articles were excluded based on a review of titles, and abstracts or executive summaries where these were available. In round 2, 8 additional articles were identified from handsearching; however, based on a review of complete texts, an additional 88 articles were excluded (see Figure 2). The remaining 30 articles were identified for inclusion in the review. A complete list of all articles and reports included is provided in Supplementary File B.

One hundred and seventy-two indicators were collected from the peer-reviewed literature and an additional 302 indicators were collected from grey literature sources and classified according to the categories described previously. A description of the indicators recorded from the identified grey and peer-reviewed literature sources is provided in Table 1.

In selecting a short list of indicators, the working group determined that many of the recorded indicators were restricted to health outcomes often measured at the community or population level (e.g., infectious disease rates, preventable hospitalization rates, birth outcomes, maternal mortality, etc.) and did not necessarily capture organizational or externally-focused public health activities. All such indicators were excluded. Team discussion of the remaining indicators included issues around a) how a given concept could be translated into a measurable indicator of health equity process at the level of the organization, b) whether a given indicator could be altered to highlight equity-seeking populations, and c) whether a proposed indicator was within public health's capacity to measure or monitor, particularly at the level of the local public health agency. Selected indicators, including those developed by the joint working group of alPHa-OPHA, were also examined, and classified, in terms of their applicability within the role framework proposed by the NCCDH. (8)

Consultation feedback

From the written feedback provided and the recorded transcripts of the interviews, summaries were created to highlight feedback related to indicator importance, recommendations for development, indicator retention, and role classification. The short list of indicators (n = 28) and brief key informant feedback are provided in Table 2.

The most common reasons offered for removal of an indicator by the key informants were that 1) it was not an indicator of the performance of equity-related activity at the level of an organization, or 2) it was redundant, i.e., being assessed as part of another indicator. Indicators that were considered for retention were all noted as requiring significant clarification, definition and development. Additionally, a number of indicators (see Table 2) were identified as having a focus on organization systems, capacity and support for equity activities. Expert informants urged consideration of an additional category or role to accommodate these and any other similar indicators that might be developed. The process of consultation and review prompted the creation of a revised list of possible indicators, suggestions for development, as well as a list of assessment gaps to be addressed within each role (Table 3). In response to the expert feedback, an additional category, entitled "Organizational and System Development" was added to reflect the approaches to and means through which local health agencies may strengthen and fulfill activities within each of the other public health roles. (8,20)

DISCUSSION

The current project spans several phases of research. In this first phase, we built upon recent reports from the NCCDH and OPHA/ alPHa (8,17) by conducting a scoping review of peer-reviewed and grey literatures in order to identify indicators that a) currently exist and b) could be used to monitor and guide progress toward fulfillment of public health roles at the level of the local public health agency.

The field of public health services research, as a whole, is just emerging as an area of concentrated interest, thus it was unsurprising that relatively few relevant articles were located. Many of the indicators identified initially represented an association between public health performance and health status outcomes. While health outcome measures are useful to public health agencies to help them identify local populations' risk of experiencing inequity, for the purposes of evaluating performance in addressing inequity, process measures tend to be more sensitive than outcome measures to differences in quality of care. (21) Performance or process indicators can be used to demonstrate practice trends, showcase excellence and highlight areas that need improvements over time. Further, while public health agencies have a role in addressing population health outcomes through collaborations with other organizations within the health system, they cannot be held accountable for these outcomes. However, public health agencies' efforts to lead, support and participate in larger system-level efforts to improve population health outcomes as they relate to health equity can be measured. We have thus included these types of indicators in Role 4 - Lead, Support, Participate.

The findings from this research ought to be considered in light of some limitations. Articles outside of the search parameters, such as non-English ones, might have contributed further information. Further, discussions and analysis were done with a background focus related to the public health system in Ontario; interpretation of findings might be different in the context of a widely dissimilar system. Population health indicators related to outcomes were considered out of scope for this review, in part because the public health system is not solely responsible for these outcomes. Future efforts might identify those system-level outcomes that fall solely within the purview of public health.

Given that proper assessment and evaluation are discussed as crucial components of tackling issues of inequity in public health in both the NCCDH documents and the OPHS, the need for performance measures is even more imperative. Our review of the literature revealed few evidence-based, validated indicators that could be used by local Boards of Health to monitor and guide progress to address health inequities. Further, consultation with key experts suggested that, while the assessment of organizational level activity is important, there is a need for continued development of these indicators, including careful operationalization of concepts and establishing clear definitions for key terms. In addition, attention should be paid to the identified assessment gaps within each of the public health roles.

CONCLUSION

The literature, and indicators presented within it, do not always reflect the health equity activity in practice at local public health agencies. Recent reports have provided important examples in which Ontario public health agencies are working on the leading edge of practice to address health inequity. (22-24) Moving forward, our challenge is to build upon the selected indicators identified in the current review to derive a set of evidence-informed, organizational performance indicators that can be used to reflect the health equity activity within local public health agencies and to help guide future activity within each of the identified public health roles.

This work showed that few robust indicators were available. In the next phase of research, results from the scoping review, integrated with the feedback from expert informants, were used to inform the development of a set of indicators for testing in a sample of local public health agencies in Ontario. In addition, based on rich learnings and input from test sites and public health practitioners, a final set of indicators and a user guide have been developed to support their use. (25)

doi: 10.17269/CJPH.108.5889

REFERENCES

(1.) World Health Organization. Commission on Social Determinants of Health: Closing the Gap in a Generation. Final Report. Geneva, Switzerland: WHO, 2008.

(2.) Baum FE, Begin M, Houweling TAJ, Taylor S. Changes not for the fainthearted: Reorienting health care systems toward health equity through action on the social determinants of health. Am J Public Health 2009; 99(11):1967-74. PMID: 19762660. doi: 10.2105/AJPH.2008.154856.

(3.) Rasanathan K, Montesinos EV, Matheson D, Etienne C, Evans T. Primary health care and the social determinants of health: Essential and complementary approaches for reducing inequities in health. J Epidemiol Community Health 2011; 65(8):656-60. PMID: 19933684. doi: 10.1136/jech. 2009.093914.

(4.) World Health Organization. Social Determinants of Health: Key Concept. 2016. Available at: http://www.who.int/social_determinants/thecommission/ finalreport/key_concepts/en/ (Accessed July 28, 2016).

(5.) Graham H. Social determinants and their unequal distribution: Clarifying policy understandings. Milbank Q 2004; 82(1):101-24. PMID: 15016245.

(6.) McPherson C, McGibbon E. Intersecting contexts of oppression within complex public systems. In: Pycroft A, Bartollas C (Eds.), Applying Complexity Theory: A Whole Systems Approach to Criminal Justice and Social Work. Bristol, UK: Policy Press, 2014; 159-80.

(7.) Ontario Ministry of Health and Long-Term Care, Public Health Division. Ontario Public Health Standards. Toronto, ON: Queen's Printer for Ontario, 2008. Available at: http://www.health.gov.on.ca/en/pro/programs/ publichealth/oph_standards (Accessed July 10, 2016).

(8.) National Collaborating Centre for Determinants of Health (NCCDH). Integrating Social Determinants of Health and Health Equity into Canadian Public Health Practice: Environmental Scan 2010. Antigonish, NS: NCCDH, 2011. Available at: http://nccdh.ca/images/uploads/Environ_Report_EN.pdf (Accessed July 27, 2016).

(9.) Reutter L, Kushner KE. Health equity through action on the social determinants of health: Taking up the challenge in nursing. Nurs Inq 2010; 17(3):269-80. PMID: 20712665. doi: 10.1111/j.1440-1800.2010.00500.x.

(10.) Marmot M, Friel A, Bell R, Houweling TA, Taylor S; Commission on Social Determinants of Health. Closing the gap in a generation: Health equity through action on the social determinants of health. Lancet 2008; 372:166-169. PMID: 18994664. doi: 10.1016/S0140-6736(08)61690-6.

(11.) Kruk ME, Freedman LP. Assessing health system performance in developing countries: A review of the literature. Health Policy 2008; 85(3):263-76. PMID: 17931736. doi: 10.1016/j.healthpol.2007.09.003.

(12.) Ontario Ministry of Health and Long-Term Care. Health Services in Your Community: Public Health Units. Toronto, ON: OMHLTC, 2014. Available at: http://www.health.gov.on.ca/en/common/system/services/phu/ (Accessed July 27, 2016).

(13.) Pinto AD, Manson H, Pauly B, Thanos J, Parks A, Cox A. Equity in public health standards. A qualitative document analysis of policies from two Canadian provinces. Int J Equity Health 2012; 11:28. PMID: 22632097. doi: 10. 1186/1475-9276-11-28.

(14.) Ontario Ministry of Health and Long-Term Care. Ontario Public Health Organizational Standards. Toronto, ON: OMHLTC, 2011. Available at: http:// www.health.gov.on.ca/en/pro/programs/publichealth/orgstandards/docs/ org_stds.pdf (Accessed July 27, 2016).

(15.) Tran N, Moffat H, Tyler I, Kothari A. Advancing health equity through organizational standards and competencies: Examples of this promising practice in Ontario. Paper presented at: The Ontario Public Health Convention (TOPHC), Toronto, ON, 2013.

(16.) Manson H, Valaitis R, Bursey G, Dobbins M, Etches V, Garcia J, et al. Implementation of the Ontario public health standards in six Ontario health units: Key results and responses from provincial and health unit stakeholders. Paper presented at: The Ontario Public Health Conference, Toronto, ON, 2012.

(17.) Association of Local Public Health Agencies/Ontario Public Health Association Health Equity Workgroup. Health Equity Indicators. Toronto, ON: Association of Local Public Health Agencies/Ontario Public Health Association Health Equity Workgroup, 2013.

(18.) Arksey H, O'Malley L. Scoping studies: Towards a methodological framework. Int J Soc Res Methodol 2005; 8(1):19-32.

(19.) Levac D, Colquhoun H, O'Brien KK. Scoping studies: Advancing the methodology. Implement Sci 2010; 5:69. PMID: 20854677. doi: 10.1186/ 1748-5908-5-69.

(20.) National Collaborating Centre for Determinants of Health (NCCDH). Let's Talk Public Health Role for Improving Health Equity. Antigonish, NS: St. Francis Xavier University, 2013. Available at: http://nccdh.ca/resources/entry/letstalk- public-health-roles (Accessed July 27, 2016).

(21.) Mant J. Process versus outcome indicators in the assessment of quality health care. Int J Qual Health Care 2001; 13(6):475-80. PMID: 11769750.

(22.) Sudbury and District Health Unit. 10 Promising Practices to Guide Local Public Health Practice to Reduce Social Inequities in Health: Technical Briefing. Sudbury, ON: SDHU, 2011. Available at: www.sdhu.com (Accessed July 27, 2016).

(23.) Wellington-Dufferin-Guelph Public Health. Addressing Social Determinants of Health in Wellington-Dufferin-Guelph: A Public Health Perspective on Local Health, Policy and Program Needs. Guelph, ON: WDGPH, 2013. Available at: www.wdgpublichealth.ca/reports (Accessed July 27, 2016).

(24.) Johnston J, Maxwell T. No Barriers Health Equity for All: Toolkit and Practical Guide for Health and Community Service Providers. Windsor, ON: Windsor-Essex County Health Unit, 2015. Available at: www.wechu.org (Accessed July 27, 2016).

(25.) Salvaterra R, Cohen B, Antonello D, LeBer MJ, Kothari A, Lemieux S, et al. Indicators to guide health equity work in local public health agencies: A locally driven collaborative project in Ontario. J Health Promot Chronic Dis Prev 2017; submitted.

Received: September 16, 2016

Accepted: March 3, 2017

Katherine Salter, MSc, [1] Rosana Salvaterra, MD, MSc, [2] Deborah Antonello, BScN, MN, [3] Benita E. Cohen, PhD,[4] Anita Kothari, PhD, [1,5] Marlene Janzen LeBer, PhD, [6] Suzanne LeMieux, MA, PhD, [7] Kathy Moran, MHSc, [8] Katherine Rizzi, MHIS, [5] Jordan Robson, MSc, [3] Caroline Wai, MHSc [9]

Author Affiliations

[1.] Health and Rehabilitation Sciences, Western University, London, ON

[2.] Peterborough County-City Health Unit, Peterborough, ON

[3.] Algoma Public Health, Sault Ste. Marie, ON

[4.] Faculty of Health Sciences, University of Manitoba, Winnipeg, MB

[5.] School of Health Studies, Faculty of Health Sciences, Western University, London, ON

[6.] School of Leadership & Social Change, Brescia University College, Western University, London, ON

[7.] Sudbury and District Health Unit, Sudbury, ON

[8.] Durham Region Health Department, Whitby, ON

[9.] Toronto Public Health, Toronto, ON

Correspondence: Katherine Salter, PhD(C), c/o Graduate Program, Health and Rehabilitation Sciences, Elborn College, Western University, London, ON N6A 1H1, E-mail: [email protected]

Conflict of Interest: None to declare.

Caption: Figure 1. Identification of articles for inclusion in the review

Caption: Figure 2. Identification of grey literature for inclusion in the review

Table 1. Abstracted indicators

Category                      Number of indicators
                                     recorded

                                Grey        Peer-
                             literature    reviewed

Cultural competency              9            4

Quality of care and              28           30
health services

Mental health                    9            2

Length of life                   25           12
Illness, disease, injury,
and wellness
                                 34           30

Health risks and                 64           7
behaviours

Gender equity and                24           3
women's health

Assessment                       30           15

Physical environment             19           14

Income, social status,           45           36
education

Civic engagement and             14           14
areas for collaboration

Category                      Description of indicators retrieved

Cultural competency         * Number and types of anti-racism
                            policies, percentages of families (in a
                            specific location) that use English as a
                            second language, recommendations re:
                            levels and use of Aboriginal languages
                            and reduction of language barriers in
                            order to improve equitable access to
                            services, spirituality, reflectiveness,
                            and responsiveness to cultural norms or
                            practices.

Quality of care and         * Various rates (e.g., infectious
health services             disease) in addition to service delivery
                            markers such as time from referral to
                            assessment and time from assessment to
                            treatment. Recommendations included
                            monitoring of patient experience as well
                            as additional assessment of utilization
                            patterns.

Mental health               * Measurements such as depression
                            prevalence, number of suicide deaths,
                            perceived work-related stress, as well
                            as recommendations to reduce depression
                            and anxiety rates among youth. No items
                            from this category were shortlisted.

Length of life              * All indicators identified were health
Illness, disease, injury,   outcome indicators only.
and wellness
                            * Disease prevalence and incidence
                            rates; recommendations for indicator
                            development included a health and
                            wellness plan targeting Aboriginal
                            groups.

Health risks and            * Developed due to the correlation
behaviours                  between items in this category and
                            social determinants of health.
                            Indicators identified in this category
                            were those attempting to monitor risks
                            and behaviours, such as smoking rates,
                            obesity rates, alcohol consumption
                            rates, drug use rates, and
                            recommendations to measure lifestyle
                            choices that have health-related
                            outcomes.

Gender equity and           * Monitoring of maternal health issues
women's health              (e.g., maternity services available,
                            gestational diabetes rates, percentage
                            of mothers screened for postpartum
                            depression, or prevalence of folic acid
                            use in peri- conception period).
                            Recommendations for indicator
                            development suggested monitoring health
                            needs by gender and conducting routine
                            gender equity analyses.

Assessment                  * All except 4 indicators were
                            recommendations for indicator
                            development; few indicators that were
                            retrieved from the literature (i.e.,
                            "measures of administration efficiency"
                            and "measures of efficacy, efficiency,
                            and quality improvement") were found
                            lacking in description and definition.

Physical environment        * Affordable housing rates, homelessness
                            rates, living arrangements;
                            recommendations to evaluate the quality
                            of housing, monitor violence and crime
                            rates according to geographic location,
                            evaluation of built environments, use of
                            health impact assessment tools,
                            reduction of geographical
                            maldistribution of services and
                            supplies.

Income, social status,      * High school attendance and graduation
education                   rates, number of families that have
                            access to quality child care, employment
                            rates, literacy rates, and percentage of
                            income used for rent. * Income security,
                            poverty rates, bankruptcy rates, number
                            of single-parent households, immigration
                            status, as well as recommendations to a)
                            understand financial and non-financial
                            barriers to access and b) identify and
                            understand existing political processes
                            and power relationships.

Civic engagement and        * Recommendations included recording the
areas for collaboration     number of inter/intra-community
                            programs, monitoring the involvement of
                            youth and elders in community
                            decision-making, encouraging partnership
                            development, developing cross-sectoral
                            collaborations, enabling community
                            empowerment to support marginalized
                            groups, supporting partnered decision
                            making.

Table 2. Shortlisted indicators and feedback from expert informants

Role (8)                    Shortlisted indicator         Feedback

Role 1--Assessing      Measurement of the percentage of   Retain
and reporting on       families with English as a
health status and      second language.
what could be done
to improve it

                       Comparison of your                 Reassign
                       organization's workforce
                       diversity with the population
                       diversity.

                       Measurement of the percentage of   Remove item
                       children who have completed
                       recommended vaccination
                       programs.

                       Measurement of percentage of       Remove item
                       elders who are offered fall
                       prevention awareness
                       initiatives.

                       Measurement of diabetes rate.      Remove item

                       Measurement of the number of       Reassign
                       clients registered in methadone
                       maintenance programs.

                       Implementation of a needle         Reassign
                       exchange program that is located
                       in the higher-need areas.

                       Use of health impact assessment    Reassign
                       tools.

                       Monitoring (the percentage of)     Retain
                       Board of Health reports on
                       health statuses that include
                       disaggregation of data by social
                       determinants of health where
                       possible

Role 2--Modify/        The number of current culturally   Retain
reorient public        sensitive policies, programs or
health programs        strategies employed; the type of
                       culturally sensitive policies,
                       programs or strategies employed;
                       the perceived effectiveness of
                       culturally sensitive policies,
                       programs or strategies.

                       Assessment of whether the          Retain
                       organization is conducting
                       equity-focused performance
                       assessments; use of pre-existing
                       Health Equity Assessment tools.

                       Use of indicators that are         Remove item
                       specific to issues of importance
                       to Aboriginal communities.

                       Assessment of whether the          Remove item
                       organization is conducting
                       gender and equity analysis for
                       the purpose of program planning.

                       Assessment of strategies used to   Retain
                       support opportunities to
                       increase the capacity of
                       underserved populations.

                       Assessment of plans for capacity   Reassign
                       building with relevant staff in
                       population health thinking
                       (e.g., through education and
                       training).*

                       Employment of a mechanism to       Reassign
                       ensure that operational planning
                       includes a health equity
                       assessment of programs and
                       services.*

                       Evaluation of how programs and     Reassign
                       services have changed or been
                       developed based on the health
                       equity assessment.*

                       Following a strategic plan that    Reassign
                       describes how equity will be
                       addressed.*

                       Following a current operational    Reassign
                       plan that incorporates the
                       identification and planning for
                       priority populations? If yes,
                       what is the process?*

Role 3--Engage in      Work/efforts/strategies to         Reassign
community and          reduce language barriers to
multisectoral          equitable access.
collaboration

                       Strategies for the development     Retain
                       of community capacity.

                       Involvement of vulnerable youth    Remove item
                       populations in community
                       decision-making.

                       Involvement of elder populations   Remove item
                       in community decision-making.

                       Participation (by the              Retain
                       organization) in local poverty
                       reduction efforts.

Role 4--Lead/          The number of new diversity and    Remove items
support/participate    anti-racist policies; the types
with others            of new diversity and anti-racist
                       policies; the

                       perceived effectiveness of the
                       new diversity and anti-racist
                       policies.

                       The number and type of diversity   Reassign
                       among the organization
                       workforce, especially managers
                       in proportion to the diversity
                       in the general population.

                       The number of community needs      Reassign
                       assessments that have been
                       conducted (within an assigned
                       period of time).

Note: Indicators appearing in bold were obtained from alPHa-OPHA.
(17) Items followed by an asterisk were identified by the content
experts as best fitting an additional organizational or systems
category to be developed.

Table 3. Gaps in assessment by role as identified by expert
informants

                          Indicators or concepts identified as missing
                          by the expert participants

Role 1--Assessing and     * Indicators that address engagement of the
reporting on health       community or provision of results to foster
status and                community discussion, etc.
what could be done to     * Early childhood education
improve it                * Disaggregation of data by SDoH/attention
                          to SDoH and equity issues

Role 2--Modify/reorient   * Standardized, precise language
public health programs    * Address issues of importance in a greater
                          number of identifiable communities
                          * There are no health outcomes listed
                          * No engagement with the community
                          * Understanding of existing services
                          * Representation of underlying values

Role 3--Engage in         * Representation of underlying values
community                 * Collaborations/partnerships; stakeholders
and multisectoral         * Who is vulnerable; who is affected by
collaboration             health equity?
                          * Early childhood education

Role 4--Lead/support/     * Types of advocacy activities,
participate               collaborations
                          * Indicators focused more
                          directly on policy development