For the purposes of your curriculum design proposal, you will focus on developing a plan for summative evaluation. Consider the procedures for summative evaluation described in your learning resources. What might the goals of evaluation be for your group's curriculum design? What should the indicators of success be, and what would be the best orientation and design for the evaluation? By Monday, the Week 5 Facilitator should create a page in the group wiki designated for "Evaluation." By Tuesday, post to your group wiki a description of the evaluation plan you would recommend be implemented in conjunction with your curriculum design proposal. Include a description of the indicators of success, the orientation of the evaluation, and how the evaluation would be conducted in order to gather the necessary data. Return to your group wiki and review the evaluation plans posted by your group members. How do your ideas compare with those of your colleagues? Do some ideas overlap? Do the ideas of your colleagues cause you to have a different perspective? By Wednesday, each group member should post his or her suggestion for the group's final plan for evaluation. This suggestion should reflect a combination of what you consider to be the best ideas from all group members. It is the responsibility of the Week 5 Facilitator to review each group member's suggestion and create a final plan for evaluation that reflects the majority opinion. By Thursday, the Week 5 Facilitator should post the final plan for evaluation to the group wiki. The plan should include a description of the indicators of success, the orientation of the evaluation, and at least one specific example of how the evaluation would be conducted in order to gather the necessary data. Group members should visit the wiki, review the plan, and use the wiki or other communication means to resolve any disagreements. In addition, the Facilitator should post the URL of this wiki content to the Week 5 area of the Group Project discussion board. Your Instructor will visit your group's wiki to ensure that all group members participated in this assignment and to approve your group's plan for evaluation. Note: Please see the Group Project 2 area for a table summarizing the curriculum design proposal tasks due this week. Use this table to help budget your time and ensure you complete assignments on time.
This is a busy week...two assignments for our group project. Good Luck everyone!
I think the orientation should be goal based and qualitative in nature.
I think an Instruction-Posttest design should be adequate to evaluate our course. As this is an introductory course, each medical technician needs to go through the course to ensure that they can display the desired skills and knowledge the chief requires. Testing after the course will determine whether the course was effective or not, and whether any value was added to the company based on the technician’s new skills.
What do you all think?
Lucia
___
Mark's Suggestions I think that Lucia has a good start. I would like to elaborate further and maybe add something to it:
Goals:
Will testing on the course content prove the display the students' mastery of course materials and procedures?
Will the students understand the varied concepts of the practices of an EMT?
Could students reach milestones of the course in the allotted timeframe?
Indicators of success:
Are students able to understand the physical compounds of medications commonly found in an ambulance and know the alternatives for patients with conflicting medical conditions?
Are students able to successfully administer meds, CPR, and Defib?
Are the students about to read off technical information found on monitors to further patient status assessment and make changes when needed?
I would agree with Lucia's statement that the orientation should be goal based and qualitative. I hope that we can all find a bit in everything we suggest to properly create a presentation that is a good proposal to this imagined chief for this project.
- Mark
Susan's thoughts
Goals
Did the students gain the knowledge necessary to complete the course?
Can the student perform CPR and operate the AED correctly?
Were the methods of instruction proper for each objective?
Objectives
Did the student operate the ambulance properly and did the student follow all safety requirements?
Did the student assess the patient correctly?
Did the student perform CPR correctly?
Did the student file the necessary report correctly?
Evaluation
Small Grp Evaluation – The group of students are varied in their knowledge and learning. A questionnaire is used for each lesson to determine if the proper information and correct teaching format is being used. The end of class critique can get full and complete feedback from all of the students involved.
Thanks,
Susan
Carmela's thoughts....
I agree with everything that you all have written thus far....Rather than duplicate efforts, I will pick those that I like most...
Goals:
Did the learner acheive the goals of this course?
Can the student successfully assess their patient?
Can the student perform CPR and operate the AED successfully?
Objectives:
Did the student file the necessary report and was it done correctly?
Did the student operate the ambulance properly and follow all the safety requirements?
Are the students about to read off technical information found on monitors to further patient status assessment and make changes when needed?
Evaluation:
Small group evaluation seems like the way to go with this assignment. It will allow us, the designers, to receive feedback directly from the learnere/user of the curriculum. This evaluation would serve best after the first entire offering allowing us to to improve the curriculum prior to subsequent delivery. We utilize this within my organization now and it has proven to be very successful. I do think we may be able to pair with with another type of evaluation to really get a good read on how our curriculum design is being interpreted. Expert reviews from the Chief could be another example. Just some thoughts.
Thanks for all of the hard work! - Carmela Level 1 Evaluation Plan
Purpose:What are the reasons for conducting the evaluation?
This is a new course and facilitators need to know if it works. Level 1 will determine if the course content and material relate to each other and to find out how the class felt about the effectiveness of the instruction and materials. This is used to make minor changes in the course.
Participants:Who will participate in the evaluation? Why?
Students- They are the participants of the course and are in a position to critique the course
Timing:When will the evaluation take place in the course of the program? Why?
On the final class meeting the level 1 evaluation will be given. This will allow plenty of time for the participants to give thoughtful answers to the questions. (think the course evals we take after each course)
Tools and Methods:What tool(s) will be used to gather the necessary information?
How will the tool(s) be administered? By whom and to whom? Provide a brief description, as well as at least two specific examples of items that might be included in the evaluation tool.
How might technology be used? How will you ensure validity of the data collection process?
A questionnaire will be used to gather the information. The participants will use a computer program to anonymously answer question. The instructor will facilitate but the questionnaire is self-lead. A 4 point Likert Scale will be used with the evaluation divided into 3 categories by objective. A summary the end of the questionnaire can also be utilized. Because the evaluation is computerized the program will automatically collect and synthesize the answers to give a comprehensive look at the program. This method ensures anonymity. Each question will pop up independently so that the student is not distracted by following questions or the length of the questionnaire. The focus will be one question at a time.
Contextual Factors:What contextual factors may influence the evaluation?
Implementation factors can affect the outcome of the evaluation. If not given enough time the students will rush through and not give thoughtful answers. If sent home they will be less likely to complete and return the survey.
Sources:What are the potential sources of information?Why are they appropriate?
Participants- They are the people actually in the position to learn the material and critique the delivery process. Facilitator- the facilitator can ask that certain areas of instruction be focused on if they are unsure of the effectiveness of that particular delivery system. Client- They understand the main goal of the program and can give insight as to what areas to focus on.
Results:How will the results of the evaluation be analyzed and used?
Benchmarking will be used with any performance standard falling below 3 on a 4 point scale considered for change. Any area that scores below the benchmark will have reasonable changes made
Level 2 Evaluation Plan
Purpose: GOALSWhat are the reasons for conducting the evaluation?
Facilitator improvement or change, redesign of program or content, an instant feedback to participants
Participants:Who will participate in the evaluation?Why?
Participants- Students will be evaluated to see if they retained the information, thus telling us if the instruction methods were successful
Timing:When will the evaluation take place in the course of the program? Why?
5 tests will be administered over x-many class periods. A pretest will determine the students prior knowledge, 3 interim tests will evaluate progress and a post test to evaluate performance of skill
Tools and Methods:What tool(s) will be used to gather the necessary information?
How will the tool(s) be administered? By whom and to whom? Provide a brief description, as well as at least two specific examples of items that might be included in the evaluation tool.
How might technology be used? How will you ensure validity of the data collection process?
Pretest- True/False- faster and easier to administer and grade Interim- combination of short essay for explanation of handling an emergency situation. Multiple choice for recall Matching for identifying Listing for recall of multiple items Post test- taken on the last class meet will be a skill performance in all of the goals and objectives (driving the ambulance, assessing the patient, making drug interaction decisions, and filing out forms). Level 1 evaluation will be administered on the final day of class.
Contextual Factors:What contextual factors may influence the evaluation?
Test anxiety can interfere with successful testing, fear of repercussion from results, and fear of knowing weaknesses and not having the drive to fix the issues
Sources:What are the potential sources of information?Why are they appropriate?
The source material should come straight from the course objectives, activities, and content
Results: GOALS OF EVALUATIONHow will the results of the evaluation be analyzed and used?
Results will be used to determine if the students gained the knowledge and if the program design was effective. If not then the program can be redesigned using data from the level 2 evaluations
Level 3 Evaluation Plan
Purpose:What are the reasons for conducting the evaluation?
The reasons for conducting the evaluation is to ensure that the students at the military academy has retained the knowledge expressed in the course objectives and that they are applying the knowledge in other areas of their education career and for military rank advancement.
Participants:Who will participate in the evaluation?
Students, instructors
Timing:When will the evaluation take place in the course of the program? Why?
The evaluation will take place six months later when as and on the job observation. The student will have to display a working knowledge of the course objectives. The reason for the time lapse is to observe if the students have retained and can apply the goals and objectives to the job.
Tools and Methods:What tool(s) will be used to gather the necessary information?
How will the tool(s) be administered? By whom and to whom? Provide a brief description, as well as at least two specific examples of items that might be included in the evaluation tool.
The data gathered in the level 2 evaluations will be used in the level 3 evaluations. The knowledge level before training will be compared to post training evaluations. A performance contract will be implemented to ensure that there is an opportunity for practice of newly learned skills. After 6 months of being on the job as EMTs the learners will once again be evaluated. Students will be observed and evaluated during the 6 months by superior officers and instructors. The performance data will be logged on a checklist. An oral interview/exam will be performed with each participant to assess the retention of knowledge of stated goals and objectives.
Contextual Factors:What contextual factors may influence the evaluation?
The lack of weekly or at least bi-monthly tactics training sessions can inhibit transfer. The learners must be provided with the opportunity to practice and hone their skills, however, since the students will be put right to work as EMTs this should not be a problem
Sources:What are the potential sources of information?Why are they appropriate?
Potential sources of information are on the job observation, Interviews with students and instructors.
Results:How will the results of the evaluation be analyzed and used?What information will the various stakeholders find useful?
The results of this level 3 evaluation will be used to determine if the training course better prepares EMT for the field and will be able to determine if the new course impacted the participants in a positive way and if they should continue to offer the course.
Level 4 Evaluation Plan
Purpose:What are the reasons for conducting the evaluation?
The main purpose of the level 4 evaluation is to determine if the outcome of training offsets the cost of training.
Participants:Who will participate in the evaluation?
Students, Police Chief, and City Officials
Timing:When will the evaluation take place in the course of the program? Why?
Ideally the evaluation would take place a year or more after training. Preferably as an employee evaluation that all EMTs will have to take as a condition of advancement or pay increases. The reason for the time lapse is to learn if the participants will make better than their predecessors and if the course had an impact on their performance in the field as compared to previous graduates from the previous course.
Tools and Methods:What tool(s) will be used to gather the necessary information?
How will the tool(s) be administered? By whom and to whom? Provide a brief description, as well as at least two specific examples of items that might be included in the evaluation tool.
The level 2 assessments will be utilized as comparison tools. The information is already gathered throughout the courses and training tactics by observation and assessments. A control group will also be utilized and the performances of both groups will be compared and analyzed for any outside influences that affected transfer All information will be gathered through normal testing that is administered by the fire department.
Contextual Factors:What contextual factors may influence the evaluation?
The length of time between training and evaluation is a possible roadblock. Leaders may want quicker answers to their questions about ROI and participants may forget some of the information. Level 4 evaluations are expensive to administer. It is difficult to separate outside variables that affect the performance of the participants and control group.
Sources:What are the potential sources of information?Why are they appropriate?
Tracking the performance of the selected students throughout their educational career provides information for level 4 evaluations. The test scores will be reviewed to track the progression of the participants and the control group. Interviews with superiors regarding the performance of the participants in contrast to the performance of the control group.
Results:How will the results of the evaluation be analyzed and used?What information will the various stakeholders find useful?
The information will be used to see if the training prepared the students for officer school. This will inform officials if the training provides a positive ROI and the military science course should continue to be offered.
Ash
Hi Ashley,
Is this the plan?
Thanks,
Susan
Susan, When I took Evaluation and Assessment last quarter this is the template that we used to create course evals. It answers all of the questions and yall can tweak it as you see fit. I just thought that I would show yall what it looks like. I guess I should have said that =) We can tailor this to suit our needs. I kinda already did slightly, but it can be more in depth if you like, but this is what an evaluation plan worksheet looks like. We have to make sure that we include level 3 AND level 4 evaluations even though they are difficult and time consuming. Since this is for a city entity the cost of the evaluations will be factored into the overall cost of the training. We really do want to see where these students are six months to a year from now.
anyways, that is why it took me so long to post to the wiki today. I was doodling with this for yall to see. I am not sure if you have taken Evaluation and Assessments yet. I still have the book (I have a book hoarding problem and refuse to sell my books back) so if yall have any questions that may be answered from Evaluation Basics by Donald McCain, just let me know. This book is more in depth than what our current text is.
OH and this encompasses the whole course from start to finish, from needs assessments to the level four that takes place a year after the course is complete.
Ash
Hi Ashley,
Actually, that is in two classes, thanks for the heads up! Thanks for the explanation.
Nice job,
Susan
Lucia's final thoughts on the evaluation plan:
Hi all,
I have looked at all the submissions and I think we've all had some great input. Here are my final thoughts:
Has the course improved the technician’s level of competence on the job?
Were the course learning methods appropriate for the content that had to be learnt?
Will testing on the course content prove the display the students' mastery of course materials and procedures?
Will the students understand the varied concepts of the practices of an EMT?
Could students reach milestones of the course in the allotted timeframe?
Did the student operate the ambulance properly and did the student follow all safety requirements?
Did the student assess the patient correctly?
Did the student file the necessary report correctly?
Indicators of success:
Are technicians able to operate an ambulance and adhere to the safety regulations?
Are technicians able to conduct CPR and use the defribilator?
Are technicians able to successfully assess a patient?
Are technicians able to file an incident report correctly?
Are students able to understand the physical compounds of medications commonly found in an ambulance and know the alternatives for patients with conflicting medical conditions?
I think the orientation should be goal based and qualitative in nature.
Evaluation
Small Grp Evaluation – The group of students are varied in their knowledge and learning. A questionnaire is used for each lesson to determine if the proper information and correct teaching format is being used. The end of class critique can get full and complete feedback from all of the students involved. It will allow us, the designers, to receive feedback directly from the learnere/user of the curriculum. This evaluation would serve best after the first entire offering allowing us to to improve the curriculum prior to subsequent delivery.
Expert reviews from the Chief could be another example.
Mark's Final Thoughts on the Evaluation Plan...
I reviewed over everyone's thoughts and suggestions on what the evaluation plan should and I share the thoughts of everyone as to where we are going with this. Ashley your thoughts were especially informative, but I had to print the chart out so your responsible for the tree I had to kill to get it :P j/k. Anyway here are my thoughts before a decision is made and I look forward to reviewing over the rest of everyone's final thoughts/suggestions before I submit the actual plan tomorrow.
The goals and objectives should answer the following questions (Which will probably look a lot like everyone else has stated):
Will the students achieved goals relating to a successful completion of the course and provided for a full understanding of responsibilities?
Will technical apsects of the course demonstrate to the students a comprehensive understanding of procedures and using equipment - i.e. the AED, monitoring equipment, IVs, etc.
Does the driving training reflect the rules and regulations for operating safely and responsibly the ambulance according to guidlines under the law?
Will the students understand the roles and obligations of EMTs?
Will the students understand the procedures for patient assessment, physical compounds of medications, alternatives for meds, implementation, and monitoring of patient status?
Will the students be able to fill out all required paper work to include incident reports?
Indicators of Success:
Any of the above goals and objectives achieved will reflect upon the indicators of success; as this is has become a common response in all of our points.
Quality before quantity is my motto in terms of orientation and course content.
I would choose a design review and a small-group option for the evaluations purposes of this course. A design review could help in eliminating a lot of revisions that might have to be made during the design of the course. If the course is broken into sections a design review can be done at the beginning of each section. Small-group evaluations are a good pick too in helping to provide more effective review of revisions.
I would also like to point out that Ashley's level 3 and 4 evaluation plans would be congruent with re-enstatment of licenses or certifications of EMT obligations. In career fields that require a certification or license (having been in a few career fields myself) evaluations to determine whether or not one will be allowed another term with a license or certification is pivotal to ensuring that those employees continually understand their job and obligations for recertification. So I would suggest combining Ashley's level 3 & 4 Evaluation Plan to be an implementation of a refresher course. I decided to put this in paragraph form instead of outline form because it does not make sense to repeat it. The point is that in certifications and liscenses, aside from recertifications or applications to extend licenses, further evaluation of the students' (at this point upon completion of the course - EMTs) work ethic and adherence to policies, regulations, practices, etc of the trade are solely up to the employer. The point of our design and the instructor's implementation of the course is to ensure that they know everything they need to know to be given their proper certifications as EMTs. Good work everyone and I look forward to the rest of the final suggestions. Have a happy 4th and a great week!
Mark Bullard
Hi Mark,
The goals being used, do match the requirements of the course. I like the design review and the small grp evals. You get critiques before and after the course. After rereading Ashley's chart, Level 3 and 4 does talk about retraining events and would be useful with the annual requirements.
Nice job!
Susan Mark, I am so sorry. I could have emailed you the word doc. I will plant a tree next week to replace the one I made you kill :D Ash
Our Evaluation Plan
The goals and objectives should answer the following questions:
Will the students achieve goals relating to a successful completion of the course?
Does the course provide for a full understanding of responsibilities of an EMT?
Will technical apsects of the course demonstrate to the students a comprehensive understanding of procedures and using equipment - i.e. the AED, monitoring equipment, IVs, etc.
Does the driver training reflect the rules and regulations for operating safely and responsibly the ambulance according to guidlines under the law?
Will the students understand the roles and obligations of EMTs?
Will the students understand the procedures for patient assessment, physical compounds of medications, alternatives for meds, implementation, and monitoring of patient status?
Will the students be able to fill out all required paper work to include incident reports?
Indicators of Success:
Any of the above goals and objectives achieved will reflect upon the indicators of success; as this is has become a common response in all of our points.
Ashley's level 3 & 4 Evaluation Plan to be an implementation of a refresher course.
Where timing takes place a year after achieved certification from introductory course.
The point being that EMTs are evaluated on their experiences to demonstrate their continued competency for certification and allow for advancement within the work environment.
These evaluations are a review of an EMT’s work ethic, adherence to policies and regulations, conduct, patient assessment, etc.
Quality before quantity is my motto in terms of orientation and course content.
I would choose a design review and a small-group option for the evaluations purposes of this course. A design review could help in eliminating a lot of revisions that might have to be made during the design of the course. If the course is broken into sections a design review can be done at the beginning of each section. Small-group evaluations are a good pick too in helping to provide more effective review of revisions. These two options are good examples of gathering the neccessary information to show how well the students have comprehended the course content and define cost effectively how the course itself has done to help instructors relay educational materials.
Group D,
Let me know if anything is out of place or needs to be tweaked, but I believe I have covered everything.
Mark Bullard
Thanks Mark - that all looks good. However, please just do a grammar check before you submit the final copy and do slides, as some sentences do not make sense to me, such as " Will the students achieved goals relating to a successful completion of the course and provided for a full understanding of responsibilities?"- could you perhaps phrase these in another way?
Thanks
Lucia
I changed that sentence and read over the others again to make sure that there was not any more mistakes. The first goal, instead of it all being once sentence I broke down into two as you can see. I hope that helped.
Group D,
For the purposes of your curriculum design proposal, you will focus on developing a plan for summative evaluation. Consider the procedures for summative evaluation described in your learning resources. What might the goals of evaluation be for your group's curriculum design? What should the indicators of success be, and what would be the best orientation and design for the evaluation?
By Monday, the Week 5 Facilitator should create a page in the group wiki designated for "Evaluation."
By Tuesday, post to your group wiki a description of the evaluation plan you would recommend be implemented in conjunction with your curriculum design proposal. Include a description of the indicators of success, the orientation of the evaluation, and how the evaluation would be conducted in order to gather the necessary data.
Return to your group wiki and review the evaluation plans posted by your group members. How do your ideas compare with those of your colleagues? Do some ideas overlap? Do the ideas of your colleagues cause you to have a different perspective?
By Wednesday, each group member should post his or her suggestion for the group's final plan for evaluation. This suggestion should reflect a combination of what you consider to be the best ideas from all group members.
It is the responsibility of the Week 5 Facilitator to review each group member's suggestion and create a final plan for evaluation that reflects the majority opinion.
By Thursday, the Week 5 Facilitator should post the final plan for evaluation to the group wiki. The plan should include a description of the indicators of success, the orientation of the evaluation, and at least one specific example of how the evaluation would be conducted in order to gather the necessary data. Group members should visit the wiki, review the plan, and use the wiki or other communication means to resolve any disagreements. In addition, the Facilitator should post the URL of this wiki content to the Week 5 area of the Group Project discussion board.
Your Instructor will visit your group's wiki to ensure that all group members participated in this assignment and to approve your group's plan for evaluation. Note: Please see the Group Project 2 area for a table summarizing the curriculum design proposal tasks due this week. Use this table to help budget your time and ensure you complete assignments on time.
This is a busy week...two assignments for our group project. Good Luck everyone!
-Mark.
LUCIA’s evaluation plan:
Our goals would be to answer the questions of:
Indicators of success:
I think the orientation should be goal based and qualitative in nature.
I think an Instruction-Posttest design should be adequate to evaluate our course. As this is an introductory course, each medical technician needs to go through the course to ensure that they can display the desired skills and knowledge the chief requires. Testing after the course will determine whether the course was effective or not, and whether any value was added to the company based on the technician’s new skills.
What do you all think?
Lucia
___
Mark's Suggestions
I think that Lucia has a good start. I would like to elaborate further and maybe add something to it:
Goals:
Indicators of success:
I would agree with Lucia's statement that the orientation should be goal based and qualitative. I hope that we can all find a bit in everything we suggest to properly create a presentation that is a good proposal to this imagined chief for this project.
- Mark
Susan's thoughts
Goals
Did the students gain the knowledge necessary to complete the course?
Can the student perform CPR and operate the AED correctly?
Were the methods of instruction proper for each objective?
Objectives
Did the student operate the ambulance properly and did the student follow all safety requirements?
Did the student assess the patient correctly?
Did the student perform CPR correctly?
Did the student file the necessary report correctly?
Evaluation
Small Grp Evaluation – The group of students are varied in their knowledge and learning. A questionnaire is used for each lesson to determine if the proper information and correct teaching format is being used. The end of class critique can get full and complete feedback from all of the students involved.
Thanks,
Susan
Carmela's thoughts....
I agree with everything that you all have written thus far....Rather than duplicate efforts, I will pick those that I like most...
Goals:
Did the learner acheive the goals of this course?
Can the student successfully assess their patient?
Can the student perform CPR and operate the AED successfully?
Objectives:
Did the student file the necessary report and was it done correctly?
Did the student operate the ambulance properly and follow all the safety requirements?
Are the students about to read off technical information found on monitors to further patient status assessment and make changes when needed?
Evaluation:
Small group evaluation seems like the way to go with this assignment. It will allow us, the designers, to receive feedback directly from the learnere/user of the curriculum. This evaluation would serve best after the first entire offering allowing us to to improve the curriculum prior to subsequent delivery. We utilize this within my organization now and it has proven to be very successful. I do think we may be able to pair with with another type of evaluation to really get a good read on how our curriculum design is being interpreted. Expert reviews from the Chief could be another example. Just some thoughts.
Thanks for all of the hard work! - Carmela
Level 1 Evaluation Plan
(think the course evals we take after each course)
How will the tool(s) be administered? By whom and to whom? Provide a brief description, as well as at least two specific examples of items that might be included in the evaluation tool.
How might technology be used? How will you ensure validity of the data collection process?
Facilitator- the facilitator can ask that certain areas of instruction be focused on if they are unsure of the effectiveness of that particular delivery system.
Client- They understand the main goal of the program and can give insight as to what areas to focus on.
Level 2 Evaluation Plan
How will the tool(s) be administered? By whom and to whom? Provide a brief description, as well as at least two specific examples of items that might be included in the evaluation tool.
How might technology be used? How will you ensure validity of the data collection process?
Interim- combination of short essay for explanation of handling an emergency situation.
Multiple choice for recall
Matching for identifying
Listing for recall of multiple items
Post test- taken on the last class meet will be a skill performance in all of the goals and objectives (driving the ambulance, assessing the patient, making drug interaction decisions, and filing out forms).
Level 1 evaluation will be administered on the final day of class.
Level 3 Evaluation Plan
The reason for the time lapse is to observe if the students have retained and can apply the goals and objectives to the job.
How will the tool(s) be administered? By whom and to whom? Provide a brief description, as well as at least two specific examples of items that might be included in the evaluation tool.
After 6 months of being on the job as EMTs the learners will once again be evaluated.
Students will be observed and evaluated during the 6 months by superior officers and instructors. The performance data will be logged on a checklist.
An oral interview/exam will be performed with each participant to assess the retention of knowledge of stated goals and objectives.
Interviews with students and instructors.
Level 4 Evaluation Plan
How will the tool(s) be administered? By whom and to whom? Provide a brief description, as well as at least two specific examples of items that might be included in the evaluation tool.
A control group will also be utilized and the performances of both groups will be compared and analyzed for any outside influences that affected transfer
All information will be gathered through normal testing that is administered by the fire department.
Level 4 evaluations are expensive to administer.
It is difficult to separate outside variables that affect the performance of the participants and control group.
Interviews with superiors regarding the performance of the participants in contrast to the performance of the control group.
Hi Ashley,
Is this the plan?
Thanks,
Susan
Susan, When I took Evaluation and Assessment last quarter this is the template that we used to create course evals. It answers all of the questions and yall can tweak it as you see fit. I just thought that I would show yall what it looks like. I guess I should have said that =) We can tailor this to suit our needs. I kinda already did slightly, but it can be more in depth if you like, but this is what an evaluation plan worksheet looks like. We have to make sure that we include level 3 AND level 4 evaluations even though they are difficult and time consuming. Since this is for a city entity the cost of the evaluations will be factored into the overall cost of the training. We really do want to see where these students are six months to a year from now.
anyways, that is why it took me so long to post to the wiki today. I was doodling with this for yall to see. I am not sure if you have taken Evaluation and Assessments yet. I still have the book (I have a book hoarding problem and refuse to sell my books back) so if yall have any questions that may be answered from Evaluation Basics by Donald McCain, just let me know. This book is more in depth than what our current text is.
OH and this encompasses the whole course from start to finish, from needs assessments to the level four that takes place a year after the course is complete.
Ash
Hi Ashley,
Actually, that is in two classes, thanks for the heads up! Thanks for the explanation.
Nice job,
Susan
Lucia's final thoughts on the evaluation plan:
Hi all,
I have looked at all the submissions and I think we've all had some great input. Here are my final thoughts:
Our goals would be to answer the questions of:
Indicators of success:
I think the orientation should be goal based and qualitative in nature.
Evaluation
Small Grp Evaluation – The group of students are varied in their knowledge and learning. A questionnaire is used for each lesson to determine if the proper information and correct teaching format is being used. The end of class critique can get full and complete feedback from all of the students involved. It will allow us, the designers, to receive feedback directly from the learnere/user of the curriculum. This evaluation would serve best after the first entire offering allowing us to to improve the curriculum prior to subsequent delivery.
Expert reviews from the Chief could be another example.
Mark's Final Thoughts on the Evaluation Plan...
I reviewed over everyone's thoughts and suggestions on what the evaluation plan should and I share the thoughts of everyone as to where we are going with this. Ashley your thoughts were especially informative, but I had to print the chart out so your responsible for the tree I had to kill to get it :P j/k. Anyway here are my thoughts before a decision is made and I look forward to reviewing over the rest of everyone's final thoughts/suggestions before I submit the actual plan tomorrow.
The goals and objectives should answer the following questions (Which will probably look a lot like everyone else has stated):
Will the students achieved goals relating to a successful completion of the course and provided for a full understanding of responsibilities?
Indicators of Success:
Quality before quantity is my motto in terms of orientation and course content.
I would choose a design review and a small-group option for the evaluations purposes of this course. A design review could help in eliminating a lot of revisions that might have to be made during the design of the course. If the course is broken into sections a design review can be done at the beginning of each section. Small-group evaluations are a good pick too in helping to provide more effective review of revisions.
I would also like to point out that Ashley's level 3 and 4 evaluation plans would be congruent with re-enstatment of licenses or certifications of EMT obligations. In career fields that require a certification or license (having been in a few career fields myself) evaluations to determine whether or not one will be allowed another term with a license or certification is pivotal to ensuring that those employees continually understand their job and obligations for recertification. So I would suggest combining Ashley's level 3 & 4 Evaluation Plan to be an implementation of a refresher course. I decided to put this in paragraph form instead of outline form because it does not make sense to repeat it. The point is that in certifications and liscenses, aside from recertifications or applications to extend licenses, further evaluation of the students' (at this point upon completion of the course - EMTs) work ethic and adherence to policies, regulations, practices, etc of the trade are solely up to the employer. The point of our design and the instructor's implementation of the course is to ensure that they know everything they need to know to be given their proper certifications as EMTs. Good work everyone and I look forward to the rest of the final suggestions. Have a happy 4th and a great week!
Mark Bullard
Hi Mark,
The goals being used, do match the requirements of the course. I like the design review and the small grp evals. You get critiques before and after the course. After rereading Ashley's chart, Level 3 and 4 does talk about retraining events and would be useful with the annual requirements.
Nice job!
Susan
Mark, I am so sorry. I could have emailed you the word doc. I will plant a tree next week to replace the one I made you kill :D
Ash
Our Evaluation Plan
The goals and objectives should answer the following questions:
Indicators of Success:
Quality before quantity is my motto in terms of orientation and course content.
I would choose a design review and a small-group option for the evaluations purposes of this course. A design review could help in eliminating a lot of revisions that might have to be made during the design of the course. If the course is broken into sections a design review can be done at the beginning of each section. Small-group evaluations are a good pick too in helping to provide more effective review of revisions. These two options are good examples of gathering the neccessary information to show how well the students have comprehended the course content and define cost effectively how the course itself has done to help instructors relay educational materials.
Group D,
Let me know if anything is out of place or needs to be tweaked, but I believe I have covered everything.
Mark Bullard
Thanks Mark - that all looks good. However, please just do a grammar check before you submit the final copy and do slides, as some sentences do not make sense to me, such as "
Will the students achieved goals relating to a successful completion of the course and provided for a full understanding of responsibilities?"- could you perhaps phrase these in another way?
Thanks
Lucia
I changed that sentence and read over the others again to make sure that there was not any more mistakes. The first goal, instead of it all being once sentence I broke down into two as you can see. I hope that helped.
Mark
Nice job Mark!
Susan