Assessing healthcare surgical performance using data envelopment analysis approach.
Ghosh, Biswadip ; Moreno, Abel A.
Abstract
Clinicians and hospital administrators rely on information models
for healthcare management and decision-making. However, healthcare
surgical performance measurements include both qualitative and
quantitative data, often with conflicting and interdependent variables.
As a result many statistical modeling approaches can break down with
healthcare data. Other more resilient algorithms, such as Data
Envelopment Analysis (DEA) and Fuzzy Composite Programming (FCP) hold
promise to address these issues. This paper applies DEA to
comprehensively assess the surgical performance. The results of the DEA
model are compared to the results obtained from a prior fuzzy composite
programming (FCP) analysis to establish additional validity.
Keywords: Healthcare Ddisease Management, Multiple Criteria
Decision Making (MCDM), Data Envelopment Analysis (DEA)
I. INTRODUCTION
The use of information systems (IS) in healthcare organizations is
on the rise. Such applications include knowledge management systems,
decision support systems and reporting systems based on patient record
management systems. Among the reasons for this trend are pressures to
reduce costs, which have been growing at an unsustainable rate and to
improve the quality of healthcare. Also these systems can help
healthcare professionals to cope with information overload and to learn
about and utilize current research developments into their practice.
Reports indicate that several healthcare organizations are proceeding to
introduce evidence-based medicine and disease management practices by
implementing information systems based on this clinical information
(McGrath, et al., 2008). The recent increase in the use of Electronic
Health Record (EHR) systems in health care facilities has resulted in a
huge amount of clinical data being collected and available online. Such
data is presenting opportunities for creating information systems for
various healthcare organizational management and decision making
purposes (Figure 1). Personnel at multiple levels in a healthcare
organization can rely on such information systems to create and deploy
analytical models that facilitate decision-making. For example, medical
chiefs and hospital directors need to track resource utilization and
outcomes of selected treatment and procedures and plan unit based
resource allocation and standardized procedures (Epstein, 2006).
Healthcare system policy makers also need information from across a
healthcare network to make strategic decisions on standardization of
treatment protocols and procedures. Clinicians need historical patient
outcome information to facilitate decisions on elective treatment (e.g.
elective surgeries) and judge the suitability of treatment options and
medical procedures for a presenting patient.
[FIGURE 1 OMITTED]
A classification of information systems that facilitate the
processes of decision making in an organization are referred to as
Decision Support Systems (DSS). Most DSS offer managers functionality
intended to support all phases of decision making--intelligence, design,
choice and implementation. DSS technologies support(1) the general goals
of reducing the uncertainty in the decision making process, such as
framing the right questions and problem(s) to solve, (2) building a
model to evaluate choices and estimating the impact of the choices on
one or more objectives and (3) the capability to evaluate changes in
assumptions, model inputs and parameter values on a chosen decision. All
activities involve the efficient and accurate collection, management,
processing and application of data/information to the decision making
process steps.
Data Envelopment Analysis (DEA) is a useful modeling platform for
complex decision making scenarios, as it allows for use of different
types of data which have large variability in the data set. Real life
situations such as in healthcare organizations are often different
because the actual values of the selected measurement criteria may
exhibit variability as well have imprecision in the way they are
collected. Statistical data analysis techniques are able to account for
variability but may not work well with imprecision, as well as criteria
that are not statistically independent (e.g. surgical wait time and
complications). By using DEA, an area/volume is used to represent each
scenario, instead of a single point (statistical approach) to get a more
complete classification of each scenario under variability. This leads
to better decision making in these domains, such as healthcare.
The goals of this research are as below:
1. Use Data Envelopment Analysis (DEA) to evaluate the performance
of surgical Units in 6 different hospitals.
2. Compare the results of DEA analysis with analysis done with
Fuzzy Composite Programming (FCP)
3. Demonstrate how DEA analysis can help identify the factors that
can be worked on by the lower performing units to improve their
performance.
II. MEASURING HEALTH CARE QUALITY
Healthcare organizations can vary greatly by size, scope,
geographic dispersion, patient mix, treatment policies for medications
and patient procedures. However, the end result, in effect of the
success or failure of a healthcare organization is the outcome of the
diagnosis and treatment of the patients' condition. Outcomes can be
influenced by other patient factors such as age, sex, severity of the
disease, lifestyle, body weight, blood pressure, etc. For example
patient death could be an inevitable outcome in many situations and
cannot always be used as an indicator of the failure of a care process
(Lezzoni, 1994). These factors determine a risk factor, which is
different for each patient, patient group and patient load at a given
facility. Hence in decision support systems for comparing healthcare
organizations, risk adjusted outcome measures are needed. Therefore the
success of a health care system should be measured by patient outcomes,
such as treatment compliance, patient satisfaction and risk adjusted
complication rates.
There are important limitations on the sole of use of patient
outcomes as indicators of the process of care (Donabedian, 1976). The
overall classification measures must also include the pathways of
medical care--programs and structures, which are important for the
delivery of the care and should be part of any measured of success or
failure of the healthcare institution. Process measures can be collected
for resource planning and utilization tracking needs as the delivery of
patient care is through these clinical processes (Donabedian, 1976). A
care process is a workflow or a set of activities around the delivery of
patient care. Care processes are delivered by different units in a
hospital, such as ICU and pre and post surgical medical units.
Institutional measures must include measures of these hospital units.
Finally these care processes are highly dependent on the structure
or settings in which care takes place and the instrumentalities of which
it is a product of. These structures include the administrative and
related infrastructure that support and direct the provision of care,
the utilization of the facilities and equipment, the nature of the
medical staff, the timely access of the facilities to the patient
(Donabedian, 1968).
These three dimensions of medical care measurement (Donabedian,
1966) structures, processes and outcomes- are illustrated in Figure 2.
[FIGURE 2 OMITTED]
III. DATA ENVELOPMENT ANALYSIS MODEL
DEA assess the relative efficiency of performance units by
obtaining the maximum of a ratio of weighted outputs to weighted inputs
(Charnes, et.al., 1978; Charnes and Cooper, 1985; Moreno and Lall,
1999). The fundamental formulation for the relative efficiency of a
performance unit is as follows:
[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]
where, s indicates the number of outputs, m the number of inputs, n
the number of performance units, yrj the value of the r-th output of the
j-th performance unit, xij the value of the i-th input of the j-th
performance unit and ur and vi the variable weights to be determined by
the solution. Notice that the formulation allows for multiple output (s)
and multiple input (m) measures, extending the traditional single-input,
single-output efficiency ratio analysis to multi-output, multi-input
situations. In the DEA model, in general terms, the larger value of an
output variable the better, while smaller the value of an input variable
the better. However, the efficiency ratio for a performance unit is the
ratio of inputs and outputs. For output variables in the DEA model, if
lower values are considered better, or for input variables if higher
values are considered better, then the inverse value of that variable is
used in the DEA efficiency calculation formula. The maximum efficiency
value of 1 for each performance unit is limited in value to 1 by the n
constraints. A relative efficiency value of 1 for a given performance
unit would indicate that there is no other performance unit capable of
producing better outputs with the same amounts of inputs. In this study,
a hospital Unit showing a relative efficiency value of 1 would imply
that for the Unit's level of inputs, no better output would be
produced by any of the other hospital Units under evaluation (Moreno and
Lall, 1999).
IV. FUZZY COMPOSITE PROGRAMMING MODEL
FCP is one of MCDM techniques, which can handle mixed indicator
data (quantitative and qualitative), and also work with conflicting,
uncertain and hierarchical criteria. FCP methodology was developed by
Bardossy and Duckstein (1992). There have been a lot of successful
applications of FCP in the DSS literature (Lee, et al., 1992;
Hagemeister, et al., 1996; Ghosh, 2008; Sadip and Veitch, 2002;
Prodanovic and Simonovic, 2002).
The normalization is done by using the best and worst basic
indicator values that are described by the following equation (Lee, et
al., 1992):
[[beta].sub.ij] = [f.sub.ij] - [f.sup.-.sub.ij]/[f.sup.+.sub.ij] -
[f.sup.-.sub.ij] (When [f.sup.+.sub.ij] is best)
Or
[[beta].sub.ij] = [f.sup.+.sub.ij] - [f.sub.ij]/[f.sup.+.sub.ij] -
[f.sup.-.sub.ij] (When [f.sup.+.sub.ij] is best)
FCP is based on a Fuzzy Composite Index (FCI). The equation is:
[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]
Where, Lj is Fuzzy Composite Index for the B+1 level group j of B
level indicators; wij is weight of B level indicators in group j; pj is
balancing factors among indicators for group j; fij+ is the best value
of ith fuzzy indicators for group j; fij- is the worst value of ith
fuzzy indicators for group j; fij is the value of ith fuzzy indicators
for group j.
The final fuzzy composite index, which is used for ranking, is
obtained by calculating the FCI from basic level to top level. The
weight parameters for indicators at different levels (wij) are
established based on the degree of importance that decision makers feel
each indicator has relative to other indicators of the same group
(Bardossy and Duckstein, 1992).
The balancing factors ([p.sub.j]) reflect the importance of maximal
deviations between indicators in the same group, and determine the
degree of substitution between indicators of the same group. Low
balancing factors (equal to 1) are used for a high level of allowable
substitution. High balancing factors (equal to 3) are used for minimal
substitution (Bardossy and Duckstein, 1992). The best value
([f.sub.ij+]) stands for the maximum possible value of the indicator,
and the worst value ([f.sub.ij-]) stands for the minimum possible value
of indicator.
V. RESEARCH MODEL
The research model is shown in Figure 3. The focus of this research
is on the development of a fuzzy decision making model to rank several
hospitals in their surgical performance. As described in section 2, the
hierarchical model contains three first level indicators of (1) care
structures, (2) care processes and (3) care outcomes. Prior healthcare
measurement research has proposed varied indicators for each of the
above three measures (Dlugacz, 2006). Items that measure care structure
could include indicators such as accessibility, utilization and the
training and experience of hospital staff. In our research model, care
structure is measured using 2 quantitative indicators - distance in
miles of the patient's reported residence to the hospital and the
average wait time for a procedure in weeks. The indicators to measure
process of care typically include the patient turnaround time in various
departments and activities - length of stay in various Units, operating
room turnaround time, patient throughput using admit and discharge
times, etc. In this model, indicators for the process of care
measurement include measures of average overall length of stay in days,
the average duration of surgery in hours and percentage of surgeries
involving same day discharge.
The indicators in the measurement of outcomes were based on using a
combination of quantitative and qualitative survey data from discharged
patients. The quantitative measures include the surgical volume of the
hospital in number of cases handled per month, the risk adjusted
complication rate for a monthly period and the percentage of patients in
compliance with post discharge prescribed medication for the monthly
period. The qualitative measure include data from a patient survey for
two questions (using a Liekert scale of 1-7) - (1) Level of satisfaction
with the treatment provided and the (2) whether the patient felt that
the perceived benefits of their treatment outweighed the risks involved.
[FIGURE 3 OMITTED]
VI. RESULTS
The research methodology consists of measurement of each of the
indicators over the patient mix in 15 units for 6 hospitals. Each
hospital unit provided a complete data set.
The DEA results from Table 1, suggest that Hospital 5 has the best
performing Units and hospital 2 has the worst performance. Hospital 6 is
a close second best performance. Hospitals 1, 3 and 4 are middle
performers.
The lowest efficiency ratio (0.5605) is exhibited by Unit #5 in
Hospital 2. This Unit's data is analyzed further in Table 2 to
ascertain factors that help to improve its efficiency ratio. The DEA
algorithm establishes a reference set of performance units with perfect
efficiency scores and then compares the unit under calculation against
that reference set. Table 2 lists the values of the input and output
variables of Unit 5 of hospital 2 and the values of those variables for
its reference set of Units (Unit 7 in hospital 1, Unit 6 in hospital 6
and Units 3 and 9 in hospital 6). The comparison indicates that Unit 5
in hospital 2 is deficient in surgical time and wait time areas along
with surgical volume and patient satisfaction scores.
(A) FCP Results at Hospital Level
The ranking of the hospitals and the final FCI values are shown in
Table 3 (Ghosh, 2008). From Table 3, we can see the comprehensive
assessment results of organization effectiveness for the six hospitals.
Among these six hospitals, 5 has the best performance, while 2 has the
worst performance (Ghosh, 2008).
The final ranking based on Hospital performance is close to that
based on structures of care (Ghosh, 2008). For example, for units E and
F are ranked as first and second, respectively by both the overall FCI
score and the structure score. The overall score and the structure score
also correspond on the least effective hospitals, A and B. The above
congruence in the scores for the top two and bottom two performing
hospitals indicate that structures of care plays the most important role
on assessing hospital performance in the fuzzy model.
Other second level indicators (processes of care and outcomes of
care) have less impact on measuring the hospital performance in the
fuzzy model. For each of those dimensions, there were at least 3
mismatches with the overall ranking (Table 4). Under structures of care,
the ranking based on wait time is the closest to that based on the
structure indicator and the final ranking (Table 5). So, wait time plays
the most important role in assessing structures of care and hospital
performance in the fuzzy model.
VII. CONCLUSIONS
This study aimed to build a multi-criteria decision making model
using data envelopment analysis and fuzzy composite programming to
compare the surgical performance of several Units in six hospitals. By
drawing on past epidemiological research, criteria was selected for
measuring structures, processes and outcomes of care to build the final
DEA and FCP models. Both quantitative data and qualitative data were
used in the hierarchical model. As seen from this research, both data
envelopment analysis (DEA) and Fuzzy Composite Programming (FCP) are
appropriate decision making model to work with mixed indicator data
(quantitative and qualitative), as well as with conflicting, uncertain
and hierarchical criteria. There was agreement among the results
obtained from DEA and FCP. Both algorithms, FCP and DEA ranked hospital
5 as the best performing hospital, followed by Hospital 6. Both
algorithms ranked hospital 2 as the worst performing hospital.
DEA allows for finding the variables in which a Unit is performing
poorly, while FCP allows for pin pointing the most important factors
that play a role in hospital performance. By analyzing the second and
third level rankings in FCP, structures of care played the most
important role in assessing hospital performance. Other second level
indicators (processes of care and outcomes of care) had less effect on
the measurement of hospital performance. Inside structures of care, wait
time had the most impact on hospital performance.
References
Bardossy, A. and Duckstein, L. (1992), "Analysis of a Karstic
Aquifer Management by Fuzzy Composite Programming", Water Resources
Bulletin (28: 1), 1992, pp. 63-73.
Charnes, A. C. and Cooper, W. W. (1985), "Preface to Topics in
Data Envelopment Analysis", Annals of Operations Research (2),
59-94.
Charnes, A. C., Cooper, W. W. and Rhodes, E. (1978),
"Measuring the Efficiency of Decision Making Units", European
Journal of Operational Research (2), 429-444.
Donabedian, A. (1966), "Evaluating the Quality of Medical
Care", Milbank Memorial Fund Quarterly (44), pp. 166-203.
Donabedian, A. (1968), "The Evaluation of Medical Care
Programs", Bull. N.Y. Academy of Medicine (44:2), pp. 117-124.
Donabedian, A. (1976), "Measuring and Evaluating Hospital and
Medical Care", Bull. N.Y. Academy of Medicine (52:1), pp. 51-59.
Epstein, A. J. (2006). Do Cardiac Surgery Report Cards Reduce
Mortality? Assessing the Evidence. Medical Care Research and Review, 63
(4), 403-426.
Ghosh, B. (2008), "Assessing Surgical Performance in
Healthcare Institutions using Fuzzy Composite Programming", 2008
IEEE International Conference on Industrial and Information Systems,
Kharagpur, India, Dec 8-10.
Hagemeister, M. Jones, D. and Woldt, W. (1996), "Hazard
Ranking of Landfills Using Fuzzy Composite Programming", Journal of
Environmental Engineer, April, 1996, pp. 248-258.
Lee, Y. L., Dahab M. and Bogardi, I. (1992), "Nitrate risk
assessment under uncertainty", Journal of Water Resources, Planning
and Management (118:2), 1992, pp 151-165
Lezzoni, L. I. (1994), "Risk Adjustment for Measuring Health
Outcomes", Health Administration Press, Ann Arbor, MI.
McGrath, K., Hendy, J., Klecun, E. and Young, T. (2008), The Vision
and Reality of 'Connecting For Health': Tensions,
Opportunities, and Policy Implications of the UK National Programme.
Communications of the Association for Information Systems, 23 (33).
Moreno, A. A. and Lall, V. (1999), "Decision Models for Robot
Selection: A Data Envelopment Analysis Approach", IJQM (5:2),
August, pp. 1-11.
Prodanovic, P. and S. Simonovic, S. (2002), "Comparison of
Fuzzy Set Ranking Methods for Implementation in Water Resources Decision
Making", Canadian Journal of Civil Engineering (29), 2002, pp.
692-701.
Sadip R. and Veitch, B. (2002), "An Integrated Approach to
Environmental Decision-making for Offshore Oil and Gas Operations",
Canada-Brazil Oil & Gas HSE seminar and Workshop, March 11-12, 2002.
Simon, H. A., "The New Science of Management Decision"
Prentice-Hall, Englewood Cliffs, NJ, 1977.
BISWADIP GHOSH * AND ABEL A. MORENO
* Computer Information Systems, Metropolitan State University of
Denver, Denver, USA, E-mail:
[email protected];
[email protected]
Table 1
DEA Assessment Results--Efficiency Ratios
HOSPITAL 1 2 3
Unit # 1 1 0.8451 0.7628
Unit # 2 0.7762 1 1
Unit # 3 0.9022 0.7579 0.7926
Unit # 4 0.8909 0.7750 0.7684
Unit # 5 0.9447 0.5605 0.7527
Unit # 6 0.9448 0.9932 0.8967
Unit # 7 1 0.9643 0.8877
Unit # 8 0.9082 0.7971 1
Unit # 9 0.8252 0.6285 0.7901
Unit # 10 1 0.7940 1
Unit #11 1 0.8828 0.6337
Unit #12 1 0.6170 0.7118
Unit # 13 1 0.7950 1
Unit # 14 0.9419 1 0.6852
Unit # 15 1 0.9958 0.7372
Number of Perfect
Efficiency Scores (1) 7 2 4
Performance Rank 3th 6th 5th
HOSPITAL 4 5 6
Unit # 1 0.9244 1 1
Unit # 2 1 1 1
Unit # 3 0.7111 1 1
Unit # 4 1 0.9907 0.8487
Unit # 5 0.8832 0.9921 1
Unit # 6 0.7614 1 0.8100
Unit # 7 0.8213 1 1
Unit # 8 1 1 0.5649
Unit # 9 0.6910 1 1
Unit # 10 0.7720 1 0.7611
Unit #11 1 0.9822 0.8589
Unit #12 1 1 1
Unit # 13 1 0.9732 0.9819
Unit # 14 0.7620 0.7047 1
Unit # 15 0.9946 1 1
Number of Perfect
Efficiency Scores (1) 6 10 9
Performance Rank 4th 1st 2nd
Table 2
DEA Assessment Results--Unit #5 in Hospital #2
DEA Algorithm Reference Set
Variable Hospital 2,
Unit # 5 Hospital Hospital
VALUE 1 Unit 7 5 Unit 6
Input Variables (Lower the better)
Same Day 0 0 92.5
Length of Stay 8 5 4
Surgical Time 5.167 3.25 2.72
Distance 11.75 5.88 10.94
Wait 6.06 6.23 1.92
Output Variables (Higher the better)
Complication Rate 5.035 5.15 1.70
Comply 88.46 95.18 81.93
Volume 50 58 57
PS 1 6 5
PSBR 4 7 4
DEA Algorithm Reference Set
Variable
Hospital Hospital 6
6 Unit 3 Unit 9
Input Variables
(Lower the better)
Same Day 0 81.13
Length of Stay 7 5
Surgical Time 0 3.33
Distance 41.67 3.17
Wait 2.73 1.78
Output Variables
(Higher the better)
Complication Rate 2.64 2.41
Comply 91 99.02
Volume 58 111
PS 7 4
PSBR 7 4
Table 3
FCP Assessment Results (Ghosh, 2008)
HOSPITAL 1 2
FCP Index 0.532 0.513
Rank 5th 6th
HOSPITAL 3 4
FCP Index 0.541 0.535
Rank 3rd 4th
HOSPITAL 5 6
FCP Index 0.585 0.578
Rank 1st 2nd
Table 4
Second Level Indicators (Ghosh, 2008)
No Structure Process Outcome Final
FCI # FCI # FCI # Rank
5 0.635 1 0.521 1 0.645 1 1
6 0.630 2 0.509 2 0.629 2 2
3 0.609 4 0.424 5 0.622 4 3
4 0.611 3 0.440 3 0.602 5 4
1 0.604 5 0.411 6 0.627 3 5
2 0.503 6 0.438 4 0.593 6 6
Table 5
Third Level Indicators for Structure (Ghosh, 2008)
No Volume Distance Wait Time Final Rank
5 3 4 1 1
6 2 6 2 2
3 1 3 3 3
4 5 1 5 4
1 4 2 4 5
2 6 5 5 6
