current issue  
back issues
  contact the editor  
  Overview home  
  contents  

Preamble

Having just completed an interesting interview with Dr Elizabeth Sommerland as part of the evaluations of the 1994 CAUT grants (now referred to as CUTSD grants) I realised that the materials from my CAUT evaluation report might be of interest to others planning to apply for these grants, or in turn to those about to write up evaluations. Dr Sommerland and I had discussed evaluation design at some length, noting that this was a very important component of these projects, not just in the writing up stage, but also in the proposal stage. A well constructed evaluation plan means that data is collected during the process of the project and the writing up is simply a matter of collation and critical thinking when it is all over. Well, maybe not that simple! What I present here is the basis of the report forwarded to CAUT at the completion of my project : Self Directed Learning in Nutrition Counselling. I present is as such so that readers may take interest in what I found, but also to display how I constructed the report from data quite deliberately collected during the process of project implementation. The headings used are those required at the time.

The Report

1. Achievement of project objectives

The main objective of the project was to establish a teaching program which encouraged self directed learning (SDL) attributes in student dietitians. This was achieved in the context of skills development in nutrition counselling. The program was based on a structured learning environment comprising experiences in a teaching clinic, the completion of SDL worksheets following the clinic and class seminars which reflected on clinic experiences. Evidence of SDL attributes was then sought from (and established in ) a range of data.

The project objectives, as outlined in the proposal were:

2. Major achievements of the project

The main achievement of the project was that it produced what it set out to do - develop self directed learning attributes in the students. Evaluation results indicated that this often took the form of self-assessment skills, although there were clear indications of other pedagogical benefits. Completed SDL worksheets provided evidence of students' abilities to assess their performance and outline further learning activities to improve performance. They also provided information on students' perceptions of professional behaviour which in turn became the basis of classroom discussions. From a teaching perspective this information lead to a recognition of the different ways in which students made sense of the phenomenon under study. The students appeared to gain confidence in discussing their experience which was validated by the class discussions. Self confidence ratings for performance indicators were significantly higher at the end of the program (p<0.05). Focus groups were held with the students at the end of the project (November, 1994) and midway through the ensuing 21 week placement (May, 1995). In both cases the students were very positive about the learning program and provided insights into the process, some of which had not been anticipated. To summarise, the major concern for the students was how their perceptions of performance matched that of the assessors, recognising that self directed learning in this context was still subject to an interdependent relationship with the teacher. (Students rated their performance and the estimated supervisors rating at a higher level than the actual rating that was given by the supervisor in the first half of the program.) They commented that the SDL project prepared them well for the practical placement, albeit highlighting a number of gaps in the system with respect to supervisor preparation. Significantly, they stated that they were able to apply the newfound approaches to SDL to other areas of learning. Supervisors comments were also sought on the difference between students over the years, and while they felt unable to comment significantly for a variety of reasons (eg staff changes), the returned performance assessment forms indicated that this group of students started at higher levels of competence than students in previous years.

Another major achievement of the project was the development of the teaching guide for the program. The reference group provided very useful assistance in evaluating early drafts and there is a higher than expected demand for the final product. This is partly because the value to supervisors had not been foreseen. An unexpected positive outcome then becomes the strengthening of the relationship between the university and industry sector, the latter providing significant support to the university in the educational process. Setting up the teaching clinics was also successful, though not a simple process, particularly given ethical issues associated with quality of service and the tape recording of confidential interviews. Strict adherence to well articulated protocols was necessary. Once established, however, the teaching clinic provided a rich and rewarding environment for both teaching and service provision based on co-operation and respect.

3. The teaching development

Practical outcome : improving teaching and learning

In terms of teaching, the main practical outcome of the project was the pedagogical design for developing SDL in students. On reflection, this need not be limited to the context of nutrition counselling for student dietitians. The teaching process drew on the model espoused by Boud (1993) on experiential learning, with the elements of preparation, experience and reflection. It provided a useful practical example of how this model might be implemented in health personnel education. Because the teaching content drew substantially on professionally defined competency standards, the project also lent practical support to efforts aimed at integrating competency standards into a teaching program.

From a learning perspective, the SDL project provided students with a framework for developing self direction in learning. The approach taken to nutrition counselling can be applied to any other subject, and indeed students were found to be doing just that. As one student stated in the November focus group

' at the beginning it was kind of an unusual thing to be doing, but now it seems more natural...this is what we do, this is how we work. In everything, not just [nutrition counselling].'

There was more to the learning outcomes than SDL skills, however. Students raised significant issues concerning the assessment of competency standards. Whilst self direction may be an ideal concept, one that is targeted in supporting lifelong learning (Candy, 1991), it is subject to a number of contingencies in higher education institutions. For students, assessment is a major focus, and in this project many did not feel comfortable with commenting on their performance without hearing the teacher's assessment first. They expressed doubt about their ability to judge, given their relative lack of experience and a fear of ignorance. Comments were made focusing on the teacher:

'it's also nice to know your view of where you stand'

'we've become more aware of your characteristics and what you want'.

These sentiments were carried over into the teaching surveys conducted by the Centre for Staff Development. The results were compared to those achieved in the previous year using the student's t test and Mann-Whitney Confidence Intervals. There was a significant difference between the two years in the scores on questions relating to assistance with learning and feedback, where the SDL group produced lower scores, again indicating their discomfort with the self directed process. Other indicators relating to information on the course, assessment, enthusiasm for the subject and presentation of material remained highly regarded.

These issues carried over into the placement where students acknowledged that different supervisors had different approaches to competency assessment and the main learning objective was often to elicit the supervisor's approach. In other circumstances the students felt the supervisors were seeking student guidance on how to make judgements on performance. The range of experiences in this regard merely highlighted the need to address gaps in supervisor training and to link up the developments at the university much more with the placement sites. The unexpected outcome of this would be an improvement in the standard of education and supervision across all venues for learning. Through discussions with supervisors it was further recognised that the project would have implications for continuing education of supervisors in the health system. The SDL activities could be re-organised in such a way that they could be adopted by practitioners in quality assurance programs. This then had implications for the design of the manual and its broader distribution beyond the 6 accredited courses in Australia. In summary, the improvements to education were the development of SDL attributes in students and the recognition that the full context of student learning needs to be addressed when introducing an innovation. This acknowledges the interdependence of components in the learning environment (Chene, 1983; Brookfield, 1993). The practical outcomes centred on the development of the manual and its distribution to the accredited centres in Australia and to supervisors in the field.

Implementation of the project

 The project was conducted according to the proposal, meeting the schedule as planned. In order to do this the program had to be organised very quickly and once operating it was necessary to keep going at a fast pace. It is difficult establishing, monitoring and evaluating a project in a 12 month period and our success could be attributed to a number of factors:

  • much of the ground work in establishing the teaching clinic had already be done the previous year. A pilot teaching clinic had been trialed for a short time, ethics approval had been submitted and there was a good working relationship between the academics and the Area Health Service dietitians. This understanding lead to the secondment of one of these dietitians onto the project who quickly became active in its development.

  • the reference group was also previously operational, having worked together on the national competency standards project and continuing with a keen interest in the practical application of its products. Although located across Australia, it was possible to monitor progress by sending material to review in the mail, making phone calls and corresponding on email.

    Integration into total learning process

     The project has been fully incorporated into the curriculum of the MSc(Nutrition and Dietetics) course. The subjects involved are

    GHMD931 Dietetics 1 : Primary Health Care (session 1)

    GHMD933 Communication in Nutrition and Dietetics (session 2).

    The subject resources contain the SDL learning materials and the teaching clinics continue to be operational. There are some organisational contingencies with these clinics, eg people do not always turn up for appointments, but this is used as a learning experience, given that it represents the 'real world'. It is important to have the whole process systematically organised, however. This also means having a number of teachers and supervisors involved in the process so that its success doesn't rely on one individual. In terms of dissemination of results, it was useful taking advantage of opportunities such as the national conference of the Dietitians Association of Australia and similar professional meetings to present the project. It was also tabled at meetings of the executive of the NSW Institute of Dietitians and the External Advisory Committee of the MSc(Nutrition and Dietetics) course, both of which were attended by representatives of a range of institutions. Duplication of effort was avoided as the profession in Australia is very small. There are only 7 accredited courses and the co-ordinators are all members of the Dietitians Association of Australia. Many are active members of the national competency standards project and this group has taken a lead role in professional education. It is relatively easy to communicate across the profession by these means and through other channels such as national newsletters. Outside the profession, DEET publications on CAUT projects indicated that this product was new to the health field.

    Monitoring and evaluation

    Positive outcomes were achieved for each of the evaluation indicators listed in the original proposal, much of which has already been discussed. These outcomes were : The focus group held with students in November 1994 raised a number of issues, some of which have already been discussed. The students also suggested the project had an impact on competition in the class. One student said the learning was 'much more personal' and another that

    'it took away the competition. The onus was on yourself and your own personal development and what (somebody else) did ... had no bearing on your progress.'

    The competency standards were both a help and a hindrance. Students said the standards provided useful direction and a baseline to work from but they were also a distraction in the interactive process. One student said 'it's a bit hard to let go of' the performance indicators and take the interaction as it occurs naturally. This feeling was supported by the supervisors who found some students a little wooden in their approach. One student said the competency standards were 'a hard concept to grasp and learn with'. This was raised again in the second focus group following the placement program as students expressed supervisors' ambivalence with the competency assessment form. The issues raised concerned individual style, the level of competency acceptable at various stages of the placement program (including a reticence to acknowledge competence too early), and the use of a quantitative scale as measurement. The project did appear to raise the level of discussion on professional competency, drawing attention to complex issues including the notion of social construction. The reference group met formally in November 1994 to review the draft manual and evaluation results. Ideas for presenting the material were raised and suggestions for making the manual more 'user friendly' were taken. The overall response was very positive and requests for the manual were much greater than originally anticipated. The project is being considered for the course at QUT and multiple copies have been requested from Deakin, Flinders and Newcastle Universities with the aim of incorporating the supervisors.

    Dissemination of information

    Information on the project has been disseminated in a range of ways, including presentations and papers. Multiple copies have been distributed across Australia to include all of the accredited courses and significant others in the health system.

    Support from the university

    The University of Wollongong has fully supported the project and its continued implementation. This support has included References

    Boud, D. (1993). "Experience as the base for learning." Higher Education Research and Development 12(1): 33-43.

    Brookfield, S. (1993). "Self-directed learning, political clarity and the critical practice of adult education." Adult Education Quarterly 43(4): 227-242.

    Candy, P. (1991). Self-Direction for Lifelong Learning. San Francisco, Jossey-Bass.

    Chene, A. (1983). "The concept of autonomy in adult education: a philosophical discussion." Adult Education Quarterly 34(1): 38-47.

    (end of report)

    Conclusion

    The reporting process is useful in formalising the evaluation of the projects. In time the findings of these evaluations should inform policies for the future directions of CAUT grants. I have certainly found that my project has raised another set of questions which will need addressing in perhaps another way.

    From a mechanical point of view, it is difficult to get an innovation up and running, completed and evaluated in a 12 month period. One of the main issues is putting on staff for that period of time and starting in January. Pre-work would appear to be essential. Perhaps CAUT would consider extending the time to say 18 months to ease the pressure for projects conducted over the full teaching year.

    In summary, one of the most rewarding aspects of the CAUT process was meeting with other grantees and sharing ideas. This was an interdisciplinary activity with a common bond in valuing teaching . It has the capacity to bring groups together who may not otherwise do so, and it encourages creativity and renewal in the teaching process.