Week 8

I am preparing for the negotiation of my evaluation plan with Bronwyn. I have been avidly reading other posts and some of the feedback and comments being posted. I have revisited my own evaluation plan so that I can form a better question.

The question I have now formed is "How can an existing hotel management paper be re-developed using elearning methods which will best meet the needs of the target audience and lecturers?"

The outcome of the evaluation plan should give me clarity of the chosen elearning guidelines. At this stage, further to Bronwyn's comments on other blogs, I now need to have a set of sub questions.

Week 7 - Negotiate evaluation plan

Time to put back on my learning hat after a very busy Easter with family from overseas! and the past couple of days attending a two day conference for all staff at Otago Polytechnic. (Bronwyn was the conference organiser!) There were four strands to the conference: Teaching and Learning, Effective use of Technology, Self Care and Sustainability with a wide range of key note speakers, panel discussions, workshops, lectures and other hands-on activities. I have come away reflecting on many aspects of my own teaching and learning. I have learnt new skills, especially through attending a workshop on Camtasia, I am very keen to put this new found skill into practice. Maybe even for my presentation later in this paper – watch this space.

I have reflected on my last blog where I considered my evaluation plan. Here is a little more detail of where I see my project heading.

Area of teaching: I am currently involved in teaching on the Bachelor of Applied management degree in the area of Hotel Management at both second and third year level. These papers are currently only available face to face; however, there is a strong mandate for papers to be more ‘flexible’ in their delivery, whether this is fully on-line or some form of ‘blended’ delivery.

Purpose of evaluation plan: A needs assessment will be carried out to establish suitable eLearning options when teaching hotel management. I have revisited the eLearning guidelines, which are a wonderful tool and revised my chosen guidelines to: TD1 - Is the use of e-learning appropriate to the intended learning outcome?
TO10 - Is good elearning practice available to staff in a way they can adopt and adapt in their own work?

Framework: The evaluation plan will be based on a needs assessment which Rossett (1995, as cited in Reeves & Hedberg, 2003) explains is a way to gather information to assist professionals in making recommendations about how to solve a problem or introduce new technology. By carrying out a needs assessment it will enable the research to evaluate the viability of the project and explore a variety of possibilities. As part of this evaluation, I will make use of the evaluation framework, called the 'eclectic-mixed methods-pragmatic paradigm' (EMMPP) as it will allow for a more flexible approach to the research. This framework also recongnises that multiple perspectives are necessary to “triangulate information and conclusions regarding complex phenomena” (Reeves & Hedberg, 2003, p. 35)

Data Collection:
By choosing the EMMPP it will allow for a wider scope of data collection. The first method will be to review the organisations strategic information on flexible and elearning as this will form a good basis of my research and will give direction for my interview questions. Secondly, I will interview two experts within the organisation. Thirdly, it is important to have information from the students; however, due to time limitations for this particular project, I will only carry out the first two at this stage of the project.

I now look forward to negotiating this plan with Bronwyn, when we catch up next week. I also welcome comments from others.

Reference

Reeves, T. C., & Hedberg, J. C. (2003). Interactive Learning Systems Evaluation. Englewood Cliffs, NJ: Educational Technology Publications.

Weeks 5 & 6

Over these two weeks we have been asked to look at different evaluation models, presented by Professor Tom Reeves. As I only teach F2F I have been struggling with the getting my head around the whole evaluation concept. After last nights class meeting, I think things have suddenly become clearer and I can see a way forward, great when everything starts to click.

The outline of my project is that I will carry out a needs assessment (or needs analysis, front end analysis) on a third year paper I teach as part of the Hotel major of the Bachelor of Applied Management degree. As lecturers, we have been asked to consider delivering these Level 7 papers more 'flexibly', however, that does not mean at this stage that they have to be totally online. Students at present can access powerpoint slides, readings and link via Blackboard.

The first stage and way of introduction and background to the project will be a review of some of the information that we have been given as lecturers, this will form the basis to conduct a needs assessment, which will evaluate the viability of putting the course online, the opportunity to engage with students in what they want and also discuss with experts in the field on eLearning.
I have suggested in my previous post that I will use the eclectic-mixed methods pragmatic approach and I still believe that it will work well in this context.

I have revisited the eLearning guidelines, which are a wonderful tool and revised my chosen guidelines to:

TD1 - Is the use of e-learning appropriate to the intended learning outcome? - this will be crucial to answer to ensure that the proposed eLearning fits with the learning outcomes of the paper.

ST1 - Do you have a way to identify needs and respond to them - if I am to move towards more a more flexible delivery approach, I want a better understanding from the outset of what students would like from this type of delivery, I believe that it is crucial to ensure connectivity between the students, their learning and lecturer.

Although, limited to only two guidelines, I would also like to consider the following guideline:
TO10 - Is good elearning practice available to staff in a way they can adopt and adapt in their own work? - in some ways, that is the reason that I have enrolled onto to this course, as I want to learn and understand some of the issues with elearning.

I have also started to consider the adoption of multiple methods evaluation (Mark and Shortland, 1987 as cited in Reeves & Hedberg, 2003) as an appropriate model, I believe that this will ensure that the final product will work well and through surveys (or focus groups - undecided after last nights discussion what will work best and considering the credibility of student answers ) of students, interviews with an elearning expert (and maybe further discussion with staff outside of my school area) and review of strategic plan documentation. This triangulation should give a reasonable indication and support for the outcomes and success of adopting flexible options to a F2F course.

I am still trying to find/digest some articles that fit with my project, so will post that part when I have that information. I welcome your comments and feedback.

Reference
Reeves, T. C., & Hedberg, J. C. (2003). Interactive Learning Systems Evaluation. Englewood Cliffs, NJ: Educational Technology Publications.

Week 4 Evaluation models and paradigms

I have read with interest the different inquiry paradigms and evaluation models. It is probably fair to say that I often steer away from anything that may have ‘quantitative’ in the title as I do not see myself as a numbers person or the reliance on measuring, variables and the analysis of relationships among statistical information, although this is changing slightly as a number of my colleagues along with our research director within the school are being put through our paces of a quantitative refresher course.

I can relate more to the constructivist-hermeneutic-interpretivist-qualitative paradigm, as it allows for the emphasis on the human being as a primary evaluation instrument and some of the methods used such as observation draw on this.

However, despite this, I also have to think about the task at hand and as I am not involved with a eLearning course at the moment and the courses I teach are all F2F, I will need to look at a needs assessment as part of my evaluation for this course.

I also need to re-visit the eLearning guidelines and look further at which will be more suited to my project.

I am veering towards the eclectic-mixed methods-pragmatic paradigm which is capable of handling the complexity of contemporary society and technology Casti, 1994; Pascale, Millerman & Goja, 2000; Sedgwick, 1993 as cited in Reeves & Hedberg, 2003). By drawing on multiple perspectives this allows a variety of evaluation methods to be used and also the ‘pragmatic’ aspect reflects the practical orientation that may be used and if it does not work the first time you can only try again. This paradigm allows a flexible approach to the design, delivery and evaluation of interactive learning systems (Reeves & Hedberg, 2003) and also allows researchers to adopt different perspectives and theoretical approaches depending on the practical needs of a project. One of the strengths of this approach is there are no ‘right’ approaches and maintaining an open approach is essential. This paradigm should offer a pragmatic way of conceptualising evaluation design.


Reeves, T. C., & Hedberg, J. C. (2003). Interactive Learning Systems Evaluation. Englewood Cliffs, NJ: Educational Technology Publications.

Week 3 eLearning Guide for Quality

This week’s task is to identify 2 eLearning guidelines relevant to my area of practice (which is in the School of Applied Business). At present I am not involved in teaching any eLearning subjects, however, we are being encouraged to with some of our papers to allow flexibility for students that may want to take some of our courses. I am hoping that by undertaking this course, I will have a better understanding.

I found the eLearning guidelines to be very interesting reading. The guidelines contend that to successfully implement elearning guidelines, they should be done at an institution wide level, not necessarily just at an individual or course level. This got me thinking a little more about resources, time, development for elearning courses etc, then I realised that I was getting off track from the question at hand and realised that I had to make my choices!

I found it quite hard to make my choice of only two and to know whether these will really be the ones that will help and guide me through my elearning project.

The two I have chosen are:
TD1 Is the use of e-learning appropriate to the intended learning outcomes?
TT13 Does the teacher evaluate the eLearning during the course to identify its effectiveness and how to improve it?

I have chosen TD1 as my first guideline, as I am still questioning the use of eLearning and whether it is the answer to learning. Although it may not be the final solution, I can see some benefits towards the use of elearning, however, there is also the issue of time, energy and effort that is required to successfully introduce eLearning that ensures a quality product at the end. I suppose I want assurance that there is learner/teacher engagement throughout the course (same as f2f) and that the needs of the learner are being met.

I have chosen TT13 as my second guideline, as I am interested in how you can evaluate progress and the effectiveness of the course during the teaching. When you are f2f you can often guage the learner/teacher engagement, students may talk informally with you and there are informal evaluation mechanisms that we use within our programme. Again, this is really ensuring that the needs of the user are being met.

The first task as part of the course Evaluation eLearning for best practice, I have been asked to comment on the following questions:
# Why is evaluation important to you and how do you define it?
# What sort of evaluations mentioned on the presentation are familiar to you already and why?
# Why is quality important in eLearning?

In answer to the first question, why evaluation is important to me, my initial thoughts were to consider how evaluation validates my resources and delivery as a lecturer within my course and context. In many ways evaluation gives you the confidence in the materials, resources, teaching and learning strategies that I would use in the classroom and the assessment material. As most of my teaching is face to face, I may need to look at different strategies as I embark into the world of elearning. Evaluation comes from a number of different sources and by way of definition.

Course evaluation by students, pre and post moderation of assessment material (peer review), usability of teaching methods, feedback from students in class, student retention and results, industry feedback.

A number of evaluations were presented within Bronwyn’s slides and on the wiki, although familiar with many, I may not have practiced many. At the end of last year, my HOD observed my teaching within the classroom. I am interested in exploring some of these evaluation methods with regards to eLearning.

As I am new to eLearning, I would have to question “Why is quality important in eLearning?’ – and ask is quality not important in all learning?, should there be a distinction. I see quality as a needs assurance, that it is cost effective, that student retention and completion of a course is good and that the outcomes for students is positive.

I thought I should introduce myself a little more, for my fellow colleagues taking this paper. I am a lecturer at Otago Polytechnic Dunedin, I teach hotel management papers and co-ordinate our year 3 students on their Cooperative Education Project within the School of Applied Business. I have been teaching at the polytechnic for 13 years.

This is my first paper of the Graduate Certificate in Applied eLearning. My main motivations for enrolling onto this programme is to gain first hand experience and knowledge of elearning, so that I may be able to incorporate this into my own teaching which may in turn increase the student experience and enhance their learning, as we are being asked to provide students more flexibility with their learning.