Client: Loddon Mallee Murray (LMM) Medicare Local is one of 61 Medicare Locals established by the Australian Government to co-ordinate primary health care delivery, tackle local health needs and fill identified service delivery gaps.
LMM Medicare Local is lead organisation for the Loddon Mallee Murray Region Partners in Recovery program (LMMR PIR), an Australian Government mental health initiative. Partners in Recovery is not intended to add a new layer of mental health services; rather it has a dual role:
- to increase coordination between existing services involved with a consumer and engage new services to meet the unaddressed needs of consumers and their carers.
- to work with other organisations that come into contact with the client group to effect change that improves the functioning of the region’s service system.
Project: In 2014, LMM Medicare Local sought an external evaluator for the LMMR Partners in Recovery program. In addition to being an innovative program with a dual mandate, a number of elements created added complexity:
- Ten organisations across New South Wales and Victoria from multiple sectors including primary health, specialist and community mental health and housing are involved in LMMR Partners in Recovery.
- Four of these organisations – MIND Australia, Northern District Community Health Services, St Luke’s Anglicare, Golden City Support Services – are contracted to host Partners in Recovery support facilitators.
- Unique regional characteristics include a large geographical area, limited public transport and mental health services, indicators of socioeconomic hardship and sizable Aboriginal and Torres Strait Islander communities.
- The Australian Government has commissioned Urbis Pty Ltd to evaluate the Partners in Recovery initiative nationally.
Following a competitive tender process, LMM Medicare Local engaged Larter to evaluate LMMR Partners in Recovery over two years.
Approach: Larter worked with LMM Medicare Local and other Partners in Recovery stakeholders to:
- Develop a project management plan for the evaluation
- Consult with LMM Medicare Local, the Partners in Recovery reference group and support facilitators on the evaluation design
- Develop a LMMR PIR program logic to guide the evaluation design
- Research and identify data collection tools
- Draft an evaluation framework
Larter approach to developing the evaluation framework included:
- Aligning evaluation activities to the Urbis’ Evaluation Framework and LMM Medicare Local’s statutory reporting to the Department of Health
- Drawing elements from multiple evaluation methodologies including logic modelling method[1], Collaborative Outcomes Reporting, action research[2] Participatory Performance Story Reporting Technique and Most Significant Change Technique[3] to measure the process, impact, outcome and unintended consequences of the program
- Identifying existing tools to quantify the strength of Partners in Recovery partnerships (VicHealth partnership checklist) and the recovery orientation of the organisations and staff involved in PIR (ROSSAT)
- Developing new data collection tools including two surveys to measure consumers’ satisfaction with, and experience of, Partners in Recovery
- Developing interview schedules for consumers and carers, the Partners in Recovery reference group and operations group
- Establishing methodological and data triangulation strategies to cross-verify data
- Embedding action learning workshops for stakeholders at key points in the evaluation to foster reflection on the findings and instigate change.
Outcome: Larter and LMM Medicare Local established a productive working relationship that enabled us to deliver the evaluation framework within a very short timeframe. Stakeholders were enthusiastic about the evaluation, providing feedback and comment on the evaluation design.
Contact us to discuss your program evaluation or read more about our consulting services.
[1] Owen, J. and Rogers, P. 2000, Program Evaluation, Allen and Unwin, St Leonards, NSW.
[2] Torres, R. T., & Preskill, H. (2001). Evaluation and organizational learning: Past, present, and future. American Journal of Evaluation, 22(3), 387-395.
[3] Dart, J.J. 2008 Report on Outcomes and Get Everyone Involved: The Participatory Performance Story Reporting Technique. Sourced 111/9/1014 at www.clearhorizon.com.au/tag/collaborative-outcomes-reporting/