Healthcare Policy

Healthcare Policy 5(3) March 2010 : 49-65.doi:10.12927/hcpol.2013.21639
Research Paper

Making Health System Performance Measurement Useful to Policy Makers: Aligning Strategies, Measurement and Local Health System Accountability in Ontario

Jeremy Veillard, Tai Huynh, Sten Ardal, Sowmya Kadandale, Niek S. Klazinga and Adalsteinn D. Brown

Abstract

This study examined the experience of the Ontario Ministry of Health and Long-Term Care in enhancing its stewardship and performance management role by developing a health system strategy map and a strategy-based scorecard through a process of policy reviews and expert consultations, and linking them to accountability agreements. An evaluation of the implementation and of the effects of the policy intervention has been carried out through direct policy observation over three years, document analysis, interviews with decision-makers and systematic discussion of findings with other authors and external reviewers. Cascading strategies at health and local health system levels were identified, and a core set of health system and local health system performance indicators was selected and incorporated into accountability agreements with the Local Health Integration Networks. Despite the persistence of such challenges as measurement limitations and lack of systematic linkage to decision-making processes, these activities helped to strengthen substantially the ministry's performance management function.

Canada's provinces face challenges with regard to costs, efficiency, access and the quality and safety of their healthcare services (Health Council of Canada 2007). In Ontario, healthcare expenditures have been steadily growing, accounting for 46% of the total provincial budget and reaching CAD$40.4 billion in 2008/09 (Ontario Ministry of Finance 2008). At the same time, the expectations of Ontarians have risen with respect to timely access to high-priority services such as cancer treatments and primary healthcare services. Policy makers in Ontario are seeking to balance these expectations with available resources while promoting performance, and have attempted various strategies over recent years such as strengthening primary healthcare and setting up Local Health Integration Networks (LHINs) to enhance health system performance and improve system integration (MoHLTC 2004). Ontario's 14 LHINs oversee over 150 hospital corporations operating on over 200 sites, over 600 long-term care homes, home care programs and thousands of community agencies, while management of drug programs and funding for the province's 23,000 physicians remain provincial responsibilities.

Over the past decade, a growing body of research has suggested that the use of strategy-based performance management tools in the public sector can result in substantial improvements in both health outcomes and cost-effectiveness gains (Jha et al. 2003; Kaplan and Norton 2005, 2006; Porter and Teisberg 2004). For example, the Veterans Health Administration in the United States achieved significant improvements in targeted health outcomes between 1995 and 2000 by linking its strategy, a core set of performance indicators, and a vigorous performance management system built on strengthened information management (Kizer 1999; Asch et al. 2004; Jha et al. 2003; Perlin et al. 2004; Perlin 2006). In the meantime, the strategic use of performance measurement through implementation of the balanced scorecard model (Kaplan and Norton 1992, 1996) at the Cleveland Clinic, the Mayo Clinic and Duke University Children's Hospital (Inamdar et al. 2002) has been associated with improvements in clinical and financial performance. Other examples demonstrate that focusing on clear linkages among strategy, performance measurement and accountability through performance measurement is a key feature in performance improvement efforts and can lead to increased value for health systems (Bevan and Hood 2006; Lomas 2003; Porter and Teisberg 2004).

Before 2004, Ontario had experimented with a limited number of strategy-based performance management elements. The province had developed extensive analytic capabilities through research institutes focused on the health system; developed a substantial hospital report card process that linked into common health system strategies (Brown et al. 2005, 2006); and had created a strategy-based scorecard for its cancer system that was used for both public reporting and performance management of regional cancer systems (Greenberg et al. 2005). Starting in 2004, the focus on the performance of the healthcare system changed in a number of ways: the government implemented a core strategy that focused on targeted initiatives to reduce wait times and strengthen primary healthcare; it created Local Health Integration Networks (LHINs) to devolve large amounts of managerial authority to the local level in order to improve integration and efficiency across the health system; it created a vehicle for public reporting of performance by establishing the Ontario Health Quality Council; and finally, it created several reform teams called Health Results Teams to drive performance improvement (MoHLTC 2004).

As planning and funding for the system shifted to the local level, these reforms had major implications for the Ministry of Health and Long-Term Care (MoHLTC). The ministry had to move away from hands-on management of the health system towards a stewardship model (Pfeffer and Sutton 2006) through which it established and communicated goals, held partners accountable for these goals and established policies for the health system.

The size, complexity and lack of clearly articulated strategies for the health system as a whole meant that the existing reform strategies had to be reviewed and synthesized to build a strategy map covering the health system in its entirety. This paper explores the Ontario experience since 2005 in developing and using a health system strategy map and strategy-based scorecard to enhance the stewardship role of government, and discusses how the information garnered from this approach has been used to strengthen the health system by aligning strategies, performance measurement and accountability.

The paper attempts to answer two key questions: How can strategy-based performance measurement be developed in the absence of an explicit and comprehensive strategy for the whole health system? And more broadly, how can this information be used by health ministries to target specific performance improvements for its agents? A retrospective evaluation of the implementation and effects of the policy intervention (Contandriopoulos et al. 2000) has been carried out through direct policy observation over three years, document analysis, interviews of decision-makers and systematic discussion of findings with other authors and external reviewers. Four of the authors of this paper (JV, TH, SA, AB) were privileged observers when the other two authors (SK, NK) were not involved in the policy intervention. The authors documented the findings with data as much as possible, grounded them into theory and referenced them with the relevant scientific literature. Furthermore, all policy papers and documents quoted are referenced and publicly available. Overall, this paper presents the methods and results of the policy intervention and discusses the valuable lessons and the applicability of this approach to other contexts.

Methods: Linking Strategy-Based Performance Measurement and Accountability

Developing the health system strategy map

The MoHLTC applied the strategy mapping approach to healthcare (Greenberg et al. 2005; Persaud and Nestman 2006), in which (a) government priorities and objectives are identified, (b) these aims are grouped into a logical set of strategic goals and (c) the goals are mapped in relation to one another. The methodological steps in formulating strategy maps have been previously defined by Kaplan and Norton (2004). However, as there was no single overarching strategic document in the system, the strategy would have to be derived from public statements of government intent and existing documents that detailed often silo-based planning efforts.

The Ontario health system strategy map was developed through six consecutive steps, which took place between November 2004 and March 2005:

  • Step 1: Common themes in health system policies, investments and public statements were identified, grouped by strategic themes and mapped in a logical sequence through a comprehensive policy review of different information sources, such as the MoHLTC budget submission and policies.
  • Step 2: Intersectoral cabinet submissions and related policies affecting health system strategies were reviewed and mapped within the health system strategic themes.
  • Step 3: External experts were consulted to validate the groupings of strategic themes and develop a draft health system strategy map outlining cause-and-effect relationships.
  • Step 4: The draft health system strategy map was shared with other ministries and stakeholder groups.
  • Step 5: Relevant inputs from the consultation process were incorporated in the draft health system strategy map, which was then internally validated by different committees and ultimately by the Executive Management Committee of the MoHLTC.
  • Step 6: The first iteration of the scorecard was distributed to 13 international experts in health system performance assessment, who appraised the model.

Once the framework was established, the next task was to identify how performance in Ontario could be measured against this foundation.

Developing a strategy-based health system performance scorecard

Despite the absence of a single performance measurement framework for the health system, there was an abundance of available performance information on health system performance (Hamilton 2006; Health Canada 2002, 2004, 2006; Health Council of Canada 2007). Although the review team identified over 2,000 performance and volume measures, few indicators were aligned to the strategic themes identified. A pre-screen of this inventory of measures was carried out by the MoHLTC in order to pre-select a set of indicators for review by a technical expert panel. The criteria used for pre-selection were validity and alignment with strategic themes; data quality issues; calculation feasibility and timeliness; and possibly the feasibility of reporting at multiple levels of the health system.

The review of inventoried indicators revealed that only 156 initially met these criteria and of these, only 54 could be cascaded to reflect different levels within the healthcare system. These indicators were submitted to an 18-member health system performance expert panel chosen based on technical knowledge of health system performance indicators. Members evaluated each indicator based on available evidence summarized in descriptive sheets detailing the rationale and the supporting evidence for inclusion in the scorecard. Different selection criteria were reviewed to identify those most appropriate to this exercise, detailed in Table 1.


TABLE 1. Criteria used for indicator selection by expert panel
Criteria Description
Importance Reflects critical aspects of health system functioning and the strategic dimension
Relevance Provides information that can be used to monitor and measure health system performance over an extended period of time
Feasibility The needed data required are readily available or obtainable with reasonable effort
Reliability The indicator produces consistent results
Validity The indicator is an accurate reflection of the dimension it is supposed to assess

 

Members used the descriptive sheets, gathering related evidence to rate the indicators against each criterion using a five-point Likert scale (a psychometric scale in which respondents specify their level of agreement with a statement). The results were collated in a report and used as the basis for a modified Delphi workshop (a systematic forecasting process that utilizes independent experts). Although an attempt was made to distribute indicators across the system goals articulated in the strategy map, a small number of performance dimensions ended up without accepted indicators. Consequently, several additional indicators were proposed by the expert panel, and descriptive sheets were commissioned for use in a second meeting in which panelists made their final recommendations for a set of 26 system performance measures. In accordance with the feasibility criteria, all 26 measures selected by the experts could be calculated by ministry analysts using readily available administrative and survey data.

Linking strategy, measurement and accountability

Performance improvement requires aligning strategy, performance information, resource allocation, incentives and accountability (Jha et al. 2003). The MoHLTC conceptualized its performance management cycle, in which the ministry (a) sets its strategic priorities, (b) selects key performance indicators related to strategy to gauge progress, (c) uses these indicators to support resource allocation processes, (d) holds those receiving resources accountable for results and (d) assesses whether performance improvements have the desired impact on the performance of the health system in order to adjust strategies accordingly (Figure 1).


Click to Enlarge
 

It is important to emphasize that the development of this process did not start at the top with the intentional development of a single strategy. Rather, the process of developing the performance management structure led to a reverse engineering of the strategy and stimulated the development of other tools to help align activities to it.

In order to link strategy, performance measurement and accountability, the MoHLTC undertook with its LHINs the joint exercise of incorporating performance indicators aligned with the health system strategy map and scorecard into accountability agreements, following a process similar to the one used for the development of the health system scorecard. The ability of LHINs to influence specific performance indicators was an additional criterion that was included in the selection of performance indicators.

Results

The 2005 Ontario health system strategy map

All health system–related strategies identified within the MoHLTC and across the government of Ontario were mapped, grouped within strategic themes and broken down into subdimensions of performance (Table 2).


TABLE 2. Ontario health system strategy map performance dimensions and subdimensions
Performance dimensions Performance subdimensions
Increase availability of high-quality, relevant evidence Appropriateness of allocation of resources
Availability of evidence
High-quality evidence
Increase access to and uptake of evidence for decision-making and accountability Increased access to evidence
Increased uptake of evidence
Increase productive use and appropriate allocation of resources across the system Appropriateness of resource allocation to achieve health system outcomes
Productive use of resources to achieve financial efficiency
Increase access to key healthcare services Availability of programs and services
Improve patient-centredness, integration and quality of health services Appropriateness
Acceptability
Responsiveness
Competence
Safety
Continuity of care
Improve healthy behaviours through health promotion and disease prevention Health promotion
Disease prevention
Improve clinical outcomes Clinical effectiveness
Improve health status Health conditions
Human function
Well-being
Mortality
Increase sustainability and equity of the health system Financing
Technology/capital infrastructure
Human resources
Confidence

 

These strategic themes were articulated in a way that linked the intermediate objectives and ultimate goals of the Ontario health system (WHO 2000). The strategy map shows that the value of other intermediate objectives and their associated actions is measured in terms of their effect on health system ultimate goals, such as improving health status, equity and health system sustainability (Figure 2).


Click to Enlarge
 

Core set of performance indicators

The 26 performance indicators selected covered all dimensions of the framework except for the first dimension, which lacked a quantitative measure meeting the pre-defined criteria. Therefore, a qualitative assessment was undertaken by experts and reviewed by the initial panel. Table 3 outlines the final set of indicators selected for inclusion. Each indicator is linked to a strategy map goal and subdimension of performance.


TABLE 3. Core set of performance indicators for the Ontario health system scorecard
Performance dimension Performance indicators
Increase availability of high-quality, relevant evidence Qualitative assessment of the availability of high-quality, relevant evidence for decision-making
Increase access to and uptake of evidence for decision-making and accountability Percentage of clinical cases being treated according to clinical practice guidelines
Increase productive use and appropriate allocation of resources across the system Percentage of alternate level of care (ALC) days
Emergency department visits that could be managed elsewhere
Hospitalization rate for ambulatory care–sensitive conditions
Increase access to key healthcare services Median wait times in priority areas: cancer surgery, cardiac procedures, cataract surgery, joint replacement, MRI/CT scan, long-term care placement
Population aged 12 and over who report having a regular medical doctor
Regular provider of diabetes care
Percentage of population who report unmet need
Improve patient-centredness, integration and quality of health services Percentage of patients with cancer who died in acute care beds
Perceptions of availability and quality of healthcare services
Inpatient readmission rates for acute myocardial infarction, psychiatric conditions, neonatal care
Percentage of people accommodated in their first choice of long-term care home
Percentage of adverse events (in-hospital fractures, new stage 2+ skin ulcers in chronic-stay patients)
Improve healthy behaviours through health promotion and disease prevention Risk factors for chronic disease
Flu vaccination
Preventive screening
Improve clinical outcomes 30-day post-hospital acute myocardiaI infarction (AMI) survival rate
5-year survival rate for prostate, breast, colorectal and lung cancer
Measure of functional improvement for rehabilitation patients
Improve health status Teenage pregnancy rates
Sexually transmitted disease (STD) rates
Potential years of life lost (PYLL)
Health-adjusted life expectancy (HALE) for overall population
Increase sustainability and equity of the health system Healthcare spending
Change in productivity
Change in health human resources supply

 

Finally, all 26 performance indicators were calculated using historical data with five-year trends where available by the MoLTC and a number of partner organizations including the Canadian Institute for Health Information, the Institute for Clinical Evaluative Sciences and Cancer Care Ontario. Results were interpreted in continuous collaboration with the members of the technical expert panel involved in the selection of the set of performance indicators and those organizations involved in data calculation.

Local health system accountability agreements

Prior to the creation of the LHINs, performance management between the payer (MoHLTC) and the providers was largely focused on financial sustainability and in some cases, volumetric measures. The set of performance indicators included in the health system scorecard was a relevant performance management tool to ensure strategic alignment between the health system and local health system strategies.

In 2007, the first generation of accountability agreements were developed between the MoHLTC and the newly created LHINs, which receive about half of the MoHLTC's budget (Bhasin and Williams 2007). Accountability agreements comprising 10 performance indicators aligned with the health system scorecard were developed (Table 4). Five developmental (pilot) performance indicators covering additional performance dimensions were included in the agreement for monitoring by the LHINs and inclusion in accountability agreements in the mid term. These agreements were in turn cascaded to other levels of the health system, such as hospitals or long-term care facilities. A ministry team calculates quarterly results for all performance indicators and posts them in a dashboard that flags performance occurring outside negotiated corridors. LHINs must report quarterly on action plans related to their performance and are accountable for attaining specific annual performance goals. It is anticipated that LHINs will be accountable for an integrated and balanced set of outcome-based performance indicators, with consequences for low-performing LHINs.


TABLE 4. Core set of performance indicators for accountability agreements between the MoHLTC and the LHINs
Performance dimension Accountability indicators Developmental indicators
Improve coordination and integration of services within local health system Percentage of alternate level of care (ALC) days N/A
Rate of emergency department visits that could be managed elsewhere
Hospitalization rate for ambulatory care–sensitive conditions (ACSC)
Median wait time to long-term care home placement
Increase access to key healthcare services 90th percentile wait times for cancer surgery N/A
90th percentile wait times for diagnostic (MRI/CT) scan
90th percentile wait times for cardiac bypass procedures
90th percentile wait times for cataract surgery
90th percentile wait times for hip and knee replacement
Improve patient-centredness, patient safety and quality of healthcare services Readmission rates for acute myocardial infarction (AMI) Perception of change in quality of care
Percentage of chronic-stay patients in complex continuing care with new stage 2 or greater skin ulcers
Percentage of in-hospital cancer deaths as a proportion of all cancer deaths
Psychiatric readmission rates in hospitals
Increase sustainability and equity of the health system N/A Change in hospital productivity

 

Discussion

Lessons from the Ontario experience: Achievements and areas for improvement

While many of the principles articulated by Kaplan and Norton (1992, 1996, 2004) in their work on balanced scorecards and strategy maps have been adopted through this experience, significant modifications were made to the way in which these tools were applied for the following reasons:

  • Context of strategy. While the strategies related to specific reform, efforts were clearly defined by the government of Ontario (e.g., wait time reduction strategy), the absence of an overarching strategy for the health system as a whole meant that the strategy map had to be built based on a review and synthesis of the existing collection of disparate strategies. Therefore, the resulting strategy map can be interpreted only as an articulation of the system's emergent, rather than deliberate, strategy (Mintzberg 1994). Owing to the dynamic nature of the policy environment, constructing a strategy map for the health system on the basis of explicit intent was not possible. The resultant scorecard should therefore be viewed only as a snapshot of system performance in areas of strategic importance, which may change from year to year, rather than as a true evaluation of the effectiveness of a particular strategy.
  • Unit of analysis. As opposed to the traditional application of the balanced scorecard, the unit of analysis is the entire health system. Therefore, measures chosen are focused largely on system outcomes.
  • Role of the ministry. While there is a parallel between the stewardship role of the MoHLTC and that of a traditional corporate headquarters, the former does not have the same degree of control over its "business units" (e.g., provider organizations) as do typical corporations. This makes the job of creating strategic alignment across the system much more difficult.

Several lessons can be drawn from this policy intervention.

First, it illustrated the value of using strategy-based performance information for decision-making. The regular monitoring of a core set of strategy-based performance indicators by the ministry's decision-makers helped refocus the role of the MoHLTC on health system outcomes and its overall stewardship function (WHO Regional Office for Europe 2008).

Second, the process for developing the health system scorecard was important in building credibility for health system performance assessment and improvement. Separating the process of strategy mapping (by policy makers), selecting performance indicators (by experts) and negotiating local health system performance improvement targets (by MoHLTC and LHIN executives) are strong assets in building a culture of trust, accountability and performance improvement.

Finally, if the development of accountability agreements between the MoHLTC and the LHINs served as a powerful tool to steer local health system performance, it is important to ensure that local management can respond to local needs while still meeting system-level priorities. If the scorecard provides any support for ensuring this balance, it is by articulating a set of goals – the desired outcomes and their indicators for the system – that can help shift attention away from more restrictive process requirements that would reduce responsiveness to local needs. As the performance measurement system develops, it will be important to ensure that the overall number of indicators is limited so that local management has the ability to introduce additional, local indicators.

However, several challenges arose from this experience.

First, there were difficulties in defining, measuring and regularly monitoring the performance of the health system. In 2006, the strategy map was redeveloped to broaden the scope of health system performance dimensions covered, and a few initial indicators were excluded from the core set and replaced owing to reliability and validity issues.

The second set of challenges relates to the need to provide concise and synthetic information to policy makers about complex systems such as a health system. These efforts have been frustrated by a dearth of effective tools such as visually representative aids (Spiegelhalter 2005). The work has also been limited by difficulties in summarizing overall performance into composite measures.

Third, the strategy map was set through consultative techniques, which require different contributors to take on the interests of the health system. However, performance management in the Canadian context is typically implemented through a negotiated process whereby groups represent their own interests. This approach can lead to a gap between intended strategy and negotiated levels of performance. This challenge may be addressed by allowing LHINs and other partners to determine how best to maximize overall performance in a way that adjusts for local needs.

Finally, the process of systematically linking performance information to the decision-making cycle of the MoHLTC has also proven to be challenging. Even if the links among strategy, performance measurement and accountability have been clearly established, the link to the resource allocation process has to be further developed (Sharpe and Keelin 1998). In addition, the performance improvement phase of the cycle has to be further strengthened through benchmarking and continuous performance improvement activities (Schoen et al. 2006; Kaplan and Norton 2005).

Conclusion

A strategy-based approach offers an innovative way to make health system performance information measurement relevant to the policy environment: establishing the strategic context, selecting indicators to setting performance targets at the local health system level and incorporating them into performance agreements. The Ontario experience suggests that the development and use of strategy-based scorecards can be useful to policy makers if clear principles are respected: performance information has to be relevant, credible and intuitive. However, the main challenge remains in systematically embedding performance information in the decision-making processes of health ministries, and implies a strong corporate discipline as well as investing in priority-setting capacities in order to allocate resources more strategically. Overall, aligning strategies, measurement and local health system accountability proved feasible in Ontario and is, in our opinion, a promising approach to be taken up by other constituencies to improve health system performance.


Pour des mesures du rendement du système de santé utiles aux responsables de politiques : harmonisation des stratégies, des mesures et de la responsabilité du système de santé local en Ontario

Résumé

Cette étude se penche sur l'expérience du ministère ontarien de la Santé et des Soins de longue durée visant à améliorer son rôle dans la gestion du rendement en développant un schéma stratégique et une carte de pointage pour le système de santé, par le biais d'un processus de révision des politiques et de consultations auprès d'experts, puis en établissant des liens avec les ententes de responsabilité. Une évaluation de la mise en œuvre et des effets de l'intervention a été effectuée par une observation directe des politiques sur une période de trois ans, par l'analyse de documents, par des entrevues auprès de décideurs et par des discussions systématiques sur les résultats avec d'autres auteurs et des réviseurs externes. Des stratégies successives au niveau des systèmes de santé général et local ont été identifiées et un ensemble central d'indicateurs du rendement pour ces systèmes de santé a été choisi puis intégré aux ententes de responsabilité avec les réseaux locaux d'intégration des services de santé. Malgré la persistance des problèmes liés aux limites et au manque de liens systématiques avec le processus de décision, ces activités ont aidé à renforcer de façon appréciable les rôles liés à la gestion du rendement, au Ministère.

About the Author(s)

Jeremy Veillard, MA, MHA, Regional Adviser for Health Policy and Equity, World Health Organization Regional Office for Europe, Copenhagen, Denmark

Tai Huynh, MBA, Project Manager, Portfolio Management, Health System Strategy Division, Ontario Ministry of Health and Long-Term Care, Toronto, ON

Sten Ardal, MA, Director, Health Analytics, Health Systems Information Management Division, Ontario, Ministry of Health and Long-Term Care, Adjunct Professor, York University (Faculty of Health) and University of Toronto (Faculty of Health Policy, Management & Evaluation), Toronto, ON

Sowmya Kadandale, MPH, Policy Analyst, World Health Organization, Geneva, Switzerland

Niek S. Klazinga, MD, PHD, Professor of Social Medicine, Academic Medicial Centre, University of Amsterdam, Amsterdam, The Netherlands

Adalsteinn D. Brown, DPHIL, Assistant Professor, Department of Health Policy, Management & Evaluation, University of Toronto, Assistant Deputy Minister, Health System Strategy Division, Ontario Ministry of Health and Long-Term Care, Toronto, ON

Correspondence may be directed to: Adalsteinn D. Brown, DPhil, Assistant Professor, Department of Health Policy, Management & Evaluation, University of Toronto, 155 College St., Rm 478, Toronto, ON M5T 3M6; tel.: 416-327-7261; fax: 416-978-1466; e-mail: Adalsteinn.brown@ontario.ca.

Acknowledgment

We thank the following staff from the Ontario Ministry of Health and Long-Term Care (Health Results Team for Information Management) for their contribution to this paper: Karen Born, Christine Brown, Erin Gilbart, Anjali Misra, Sanober Motiwala, Anmar Salman, Emily Simmonds, Emily Siu, Angus Steele and Brenda Tipper. We would also like to thank Dr. Govin Permanand from the World Health Organization Regional Office for Europe for his detailed review and comments on this paper.

References

Asch, S.M., E.A. McGlynn, M.M. Hogan, R.A. Hayward, P. Shekelle, L. Rubenstein, J. Keesey, J. Adams and E.A. Kerr. 2004. "Comparison of Care for Patients in the Veterans Health Administration and Patients in a National Sample." Annals of Internal Medicine 141(12): 938–45.

Bevan, G. and C. Hood. 2006. "Have Targets Improved Performance in the English NHS?" British Medical Journal 332(7538): 419–22.

Bhasin K. and A.P. Williams. 2007. "Understanding LHINs: A Review of the Health System Integration Act and the Integrated Health Services." Retrieved January 6, 2010. <http://www.ryerson.ca/crncc/download/UnderstandingLHINs-FinalJuly5th.pdf>.

Brown, A.D., L.M. Alikhan, G.A. Sandoval, N.L. Seeman, G.R. Baker and G.H. Pink. 2005. "Acute Care Hospital Strategic Priorities: Perceptions of Challenges, Control, Competition and Collaboration in Ontario's Evolving Healthcare System." Healthcare Quarterly 8(3): 34–45.

Brown, A.D., L.M. Alikhan and N.L. Seeman. 2006. "Crossing the Strategic Synapse: Aligning Hospital Strategy with Shared System Priorities in Ontario, Canada." Healthcare Management Review 31(1): 34–44.

Contandriopoulos, A.P., F. Champagne, J.L. Denis and M.C. Avargues. 2000. "L'Évaluation dans le domaine de la santé : concepts et méthodes." Revue Épidemiologie et santé publique 48: 517–39.

Greenberg, A., H. Angus, T. Sullivan and A.D. Brown. 2005. "Development of a Set of Strategy-Based, System-Level Cancer Care Performance Indicators in Ontario, Canada." International Journal for Quality in Healthcare 17: 107–14.

Hamilton, C. 2006. Healthy Provinces, Healthy Canadians: A Provincial Benchmarking Report. Ottawa: Conference Board of Canada.

Health Canada. 2002. Healthy Canadians: A Federal Report on Comparable Indicators. Ottawa: Author.

Health Canada. 2004. Healthy Canadians: A Federal Report on Comparable Indicators. Ottawa: Author.

Health Canada. 2006. Healthy Canadians: A Federal Report on Comparable Indicators. Ottawa: Author.

Health Council of Canada. 2007. Healthcare Renewal in Canada: Measuring Up? Ottawa: Author.

Inamdar, N., R. Kaplan, M. Bower and K. Reynolds. 2002 (May/June). "Applying the Balanced Scorecard in Healthcare Provider Organizations." Journal of Healthcare Management 47: 179–96.

Jha, A.K., J.B. Perlin, K.W. Kizer and R.A. Dudley. 2003. "Effect of the Transformation of the Veterans Affairs Healthcare System on the Quality of Care." New England Journal of Medicine 348(22): 2218–27.

Kaplan, R.S. and D.P. Norton. 1992 (January/February). "The Balanced Scorecard: Measures That Drive Performance." Harvard Business Review 70(1): 71-8.

Kaplan, R.S. and D.P. Norton. 1996 (January/February). "Using the Balanced Scorecard as a Strategic Management System." Harvard Business Review 74(1): 75-85.

Kaplan, R.S. and D.P. Norton. 2004. Strategy Maps: Converting Intangible Assets into Tangible Outcomes. Boston: Harvard Business School Press.

Kaplan, R.S. and D.P. Norton. 2005 (October 1). "The Office of Strategy Management." Harvard Business Review 73(10): 72-80.

Kaplan, R.S. and D.P. Norton. 2006. "How to Implement a New Strategy without Disrupting Your Organization." Harvard Business Review 84(3): 100–9.

Kizer, K.W. 1999. "The 'New VA': A National Laboratory for Health Care Quality Management." American Journal of Medical Quality 14(1): 3–20.

Lomas, J. 2003. "Health Services Research: More Lessons from Kaizer Permanente and Veterans Affairs Healthcare System." British Medical Journal 327(7427): 1301–2.

Ministry of Health and Long-Term Care (MoHLTC). 2004. "Ontario's Health Transformation Plan: Purpose and Progress." Transcript of speech by George Smitherman, Minister of Health. Retrieved January 4, 2010. <http://www.health.gov.on.ca/english/media/speeches/archives/sp_04/sp_090904.pdf>.

Mintzberg, P. 1994. The Rise and Fall of Strategic Planning: Reconceiving Roles for Planning, Plans, Planners. New York: The Free Press.

Ontario Ministry of Finance. 2008. "Strengthening Ontario's Future by Investing in Health Care." Press release. Retrieved January 4, 2010. <http://www.fin.gov.on.ca/english/budget/ontariobudgets/2008/bk3.html>.

Perlin, J.B. 2006. "Transformation of the US Veterans Health Administration." Health Economy, Policy and Law 1(Pt. 2): 99–105.

Perlin, J.B., R.M. Kolodner and R.H. Roswell. 2004. "Veterans Health Affairs: Quality, Value, Accountability and Information as Transformation Strategies for Patient-Centred Care." American Journal of Managed Care 10(11): 828–36.

Persaud, D.D. and L. Nestman. 2006. "The Utilization of Systematic Outcome Mapping to Improve Performance Management in Health Care." Health Services Management Research 19(4): 264–76.

Pfeffer, J. and R.I. Sutton. 2006. "Evidence-Based Management." Harvard Business Review 84(1): 62–74, 133.

Porter, M.E. and E.O. Teisberg. 2004. "Redefining Competition in Health Care." Harvard Business Review 82(6): 64–76, 136.

Schoen, C., K. Davis, S.K.H. How and S.C. Schoenbaum. 2006. "US Health System Performance: A National Scorecard." Health Affairs 25(6): w457–w475.

Sharpe, P. and T. Keelin. 1998. "How SmithKline Beecham Makes Better Resource-Allocation Decisions." Harvard Business Review 76(2): 45–46.

Spiegelhalter, D.J. 2005. "Funnel Plots for Comparing Institutional Performance." Statistics in Medicine 24(8): 1185–202.

World Health Organization (WHO). 2000. The World Health Report 2000. Health Systems: Improving Performance. Geneva: Author.

World Health Organization (WHO) Regional Office for Europe. 2008. "Stewardship/Governance of Health Systems." Background document to the 2008 Regional Committee. Retrieved January 4, 2010. <http://www.euro.who.int/Document/RC58/RC58_edoc09.pdf>.

Comments

Be the first to comment on this!

Note: Please enter a display name. Your email address will not be publically displayed