Assessment Materials

Part 1

Marie A. Abate, BS, PharmD.
Professor of Clinical Pharmacy & Director of Assessment

School of Pharmacy Objectives

Differentiate among the different types of assessment in academia and explain why assessment is needed.
Review the basic steps that should be included in a learning outcomes assessment plan.
Explain the differences between direct and indirect measures of student learning.
Describe the principles and characteristics of effective program assessment.
Describe the potential barriers and challenges to effective program assessment efforts.

Required Readings

James Madison University. Characteristics of an Effective Assessment Program, 2003. Accessed at: http://www.jmu.edu/assessment/resources/EffectiveProgram.htm, March 2007.
Van der Vleuten, CPM, Dolmans DHJM, Scherpbier AJJA. The need for evidence in education. Med Teach 2000;22:246-50.
Banta TW. Moving assessment forward: enabling conditions and stumbling blocks. New Directions for Higher Education 1997;25:79-91.

Questions & Answers

Faculty often have questions about learning outcomes assessment. Several of the more commonly asked questions and their answers follow.

What is meant by “assessment,” and how does it differ from “evaluation?”

Assessment is an ongoing process used to discover what, how, when, and which students learn with regard to expected learning outcomes. The term “evaluation” is also used to make a conclusion about student learning. However, it can be differentiated from “assessment” in that assessment is ongoing, with changes made as needed based upon the data obtained.

What types of assessment are there?

Assessment can be focused on the individual student, the course, or the program (Figure):

Figure – Types of Assessment

The foundation of assessment is the determination of individual student mastery of the desired learning outcomes. Student assessment includes instructor assessment of the student, student self-assessment, and peer assessment. The assessment can be formative, i.e., performance feedback designed primarily to improve learning, or summative, i.e., assignment of a final grade or decision about performance that could affect student progress in a program. Methods for assessing student performance include quizzes/tests/exams (e.g., multiple-choice, matching, true/false, short answer, essay, etc.); written or verbal cases studies, papers, essays, assignments, problems, projects, journals, portfolios, presentations; simulations (e.g., computer-based, use of real patients or actors, real or mock practice settings, etc.); and direct observation (e.g., experiential rotations, etc.).

Course assessment involves determining the effectiveness of individual courses in achieving their desired learning outcomes for students, and program assessment involves determining the effectiveness of the entire program in achieving its educational outcomes. Assessment at both the course and program levels involves setting appropriate criteria and high standards for learning quality; systematically gathering, analyzing, and interpreting evidence of this learning from a variety of diverse sources; and using the evidence to document and explain performance. “Completing the loop,” i.e., using the results to make changes to improve learning and student performance, is a necessary step in the assessment process.

Why is learning outcomes assessment important?

The assessment processes used in courses or a program will directly affect how and what the students do and learn. In addition, interpretation and analysis of assessment results enable a program to identify how well it is achieving its desired outcomes. It also allows the program to development, plan, and implement improvements in its curriculum when desired outcomes are not fully met. Assessment should serve to focus a program’s attention on student learning. It should create linkages and enhance coherence of subject matter within courses and across the curriculum.

What basic steps should be included in a program’s learning outcomes assessment plan?

There are several important steps to include in an assessment plan:

Identify the program’s mission statement as it relates to education. Having a mission statement will help keep the unit “on track” during the assessment process.
Student learning (or educational) outcomes should be developed. These outcomes should include all the key skills, abilities, knowledge, attitudes, and values that the program expects its graduates to possess.
Collect data to determine the extent to which the outcomes are being achieved. For each student learning outcome, assessment method(s) should be developed that allow for the collection of these data.
Consider how the program will determine whether or not it is “successful” in meeting the outcomes for its students. For example, are all students expected to achieve a certain proficiency level for an outcome? Would the program be satisfied if only a certain percentage of students achieve a certain level of proficiency? What should this percentage and level be? Thus, the program should identify criteria for success for each learning outcome. Although this step can be difficult, it is very important. To accomplish this, it can be helpful to consider the “cut-off” point at which the program would want to initiate curriculum changes based upon the student data obtained.
Review and analyze the data collected using the outcomes’ criteria for success as a guide.
Determine actions to take based upon the findings. The actions should directly address the specific problem(s) identified.
Plans should be made for future data collection after changes are implemented to determine whether they were successful.

Students are already being given grades in their program’s courses – why isn’t this sufficient for assessment purposes?

Courses usually cover several topics as well as encompass a variety of skills. For example, a course might review four or five (or more) major subject areas and require students to use several different cognitive levels of learning in these areas, such as knowledge, comprehension, analysis, and evaluation. Thus, a student’s final course grade would represent a composite of all these different aspects. A student might perform well in several areas but poorly in one or more areas, and still achieve a high course grade. Further, a course might have an attendance policy and a portion of the grade could simply involve regular attendance. Because of all these factors, overall course grades will not usually provide definitive information about how well students can master specific, targeted learning outcomes. From a program perspective, individual course grades also do not provide a clear picture of how well students can retain, integrate and apply topics across courses or years of a curriculum, or to actual patient care or practice situations. For all of these reasons, overall course grades are not an acceptable measure for assessment purposes. However, data obtained from activities, assignments, or exercises within courses that target specific types of knowledge/skills/attitudes/values could be valuable assessment measures.

Where and how should a program assess student learning (i.e., achievement of its desired outcomes)?

Student learning should be measured in a variety of places within a program and by different methods. For example, there are certain skills that students might need to possess early on, and the program might want to assess the extent of student mastery of these skills towards the beginning of its program. There might also be skills that students would be expected to develop with time and practice; these are the types of skills that could be assessed towards the end of the program, perhaps in experiential rotations. There are still other skills that students might be expected to begin to develop early, but to continue to demonstrate progress and sophistication with time. These skills can be assessed both early and later in the program. Methods to assess student learning have been divided into two types, direct measures and indirect measures.

What is meant by “direct measures” of student learning?

These are methods by which the level of a student’s knowledge, skills or attitudes/values are directly determined based upon comparison to measurable learning objectives or outcomes. Students must demonstrate what they know or what they can do with the knowledge they have. Direct measures can be incorporated (embedded) into appropriate courses or they can be administered outside of courses.

What are examples of direct measures of student learning?

Some methods that provide direct evidence of student learning and development include the following:

Portfolios that collect students’ work over time - provide longitudinal evidence of student learning and development
Course-embedded assignments (e.g., collecting data from certain exercises, problems, projects, etc. within a specific course[s] that target desired abilities) - provide evidence of how well students transfer learning into different contexts
Comprehensive projects - provide evidence of how well students integrate and apply principles, concepts, and abilities
Direct observations and ratings of student behavior - provide evidence of how well students practice or apply an ability, such as how well they communicate verbally or participate in collaborative problem solving activities
Internships or experiential rotations - provide evidence of students’ problem-solving abilities and performance in a work environment
Performance on a case study or with simulated patients - provide evidence of students’ abilities to apply, synthesize and solve problems; these may be used over time to track the development of students’ knowledge or abilities
Essays - provide evidence of students’ abilities to represent ideas, solve problems, synthesize information
Examinations or quizzes - provide evidence of how well students’ achievement matches program expectations
Standardized or national licensure tests - provide evidence of student achievement based on norms established outside of an institution

(Adapted from http://www.naspa.org/NetResults/article.cfm?ID=558)

What is meant by “indirect measures” of student learning?

Indirect measures are used to determine the perceived extent of a student’s knowledge or skills. Indirect measures should generally not be used as the only type of measure for particular knowledge or skills. Rather, they serve to complement direct measures by providing another insight into students and their learning process.

What are examples of indirect measures of student learning?

Some indirect methods include the following:

Alumni, employer, or faculty surveys - provide perceptions about what students know or their abilities to apply knowledge
Student surveys or questionnaires - provide self-reports about what students feel they know and their abilities to apply knowledge
Student or employer focus groups - provide interpretations or perceptions of student learning
Follow-up studies of graduates - provide evidence of how well an institution prepared students for practice or advanced work
Percentage of students who go on to graduate school - provides evidence of how well an institution prepared students for advanced education or training
Retention statistics - provide evidence of overall program success
Job placement statistics - provide evidence of how well students were prepared for entry into the workplace

(Adapted from http://www.naspa.org/NetResults/article.cfm?ID=558)

Is it appropriate to use a single measure of student learning, or should multiple assessment methods be used routinely?

It is best to use several different kinds of measures to assess student learning. Keep in mind that individuals learn and can demonstrate their learning in different ways. Further, the assessment method used should correspond with the knowledge or skill being measured. For example, if students are being assessed on their ability to solve a problem and formulate a reasonable solution, then at least one of the assessment methods incorporated should involve providing students with a problem that they are asked to solve. The actions taken as a result of data collected from an assessment measure should be based on our professional judgment as educators, after taking into consideration the findings from other related assessments. For example, if students appear unable to solve a particular type of problem, this could be the result of several different factors. Perhaps students lack the knowledge necessary to solve that particular problem but can solve other problems involving different medical conditions. Or, students might possess the knowledge but lack sufficient practice on how to apply this knowledge. As another possibility, the skill measured might not reasonably be expected to fully develop until after the student enters practice and obtains additional work experience (follow-up data from graduates or focus groups could provide insight here).

(Adapted from http://www.aahe.org/Bulletin/may2.htm)

Principles and Characteristics

The remainder of the material in this section will focus specifically on program assessment, although the same principles generally apply to assessment of courses. Several methods that can be used to assess or evaluate student performance are discussed in Part III of the assessment module.

A successful learning outcomes assessment plan should allow the faculty to make informed changes in the curriculum that improve student performance. The student evaluation and assessment methods employed will also determine how and what students learn since they will focus attention on those aspects. For example, if acquisition of factual knowledge is predominantly assessed, students will primarily study to acquire and memorize facts. This had been referred to as a “steering effect.” Thus, assessment is critical to educational effectiveness. What are the key principles and characteristics of effective program assessment practices? They are summarized in Table 1 and reviewed in the paragraphs that follow.

Table 1. Summary of Principles/Characteristics of Effective Program Assessment^1,3

Integrated into culture
Ongoing and sustained
Based upon appropriate student learning outcomes
Reflects learning as multidimensional and integrated
Considers experiences leading to outcomes
Involves representatives from across educational community
Part of several practices to promote change
Undertaken in receptive, supportive, enabling environment
Basis for funding/re-allocation decisions
Directed by competent, trustworthy individuals
Regularly re-evaluated

Assessment must be integrated into a program’s culture, meaning that it ultimately becomes part of the normal, routine responsibilities of both faculty and administrators. Assessment is not a one-time undertaking. Rather, it should be ongoing with sustained commitment by all departments and faculty. It should help clarify questions or concerns that faculty care about, and it should provide evidence in areas important for decisions. Assessment should be based upon clear, explicit, focused, and measurable student learning outcomes that in turn reflect the educational mission and goals.

Assessment should reflect learning as multidimensional and integrated and should reveal performance over time. Thus, multiple methods that are carefully selected, with consideration given to their reliability and validity, should be used to assess student learning outcomes. Attention must be given not only to the outcomes themselves but also to the experiences leading to those outcomes. Thus, the processes of teaching and curriculum development used to enhance student learning help define successful outcomes assessment.

Assessment should involve representatives from across the educational community, including faculty, staff, students, and external stakeholders such as employers. All of these persons can provide valuable input into, and should be included throughout, the various aspects of the assessment process (e.g., planning, implementation, review of assessment instruments, review and analysis of data, etc.). Assessment should also be part of a larger set of practices to promote change, such as holding assessment-related faculty development sessions, having ongoing faculty discussions related to assessment and learning, and using assessment data to make curricular changes.

Assessment is most effective when undertaken in an environment that is receptive, supportive and enabling. This includes having strong support from administrators, adequate resources for implementation, creation of an atmosphere of trust that data will be used for improvement and not for punitive measures, and the establishment of avenues for communicating results to a variety of audiences. If the findings from assessment measures indicate that changes are needed, funding decisions or reallocation of resources should be made to ensure that the changes can be implemented successfully. Although it is important for assessment efforts in a program to be directed by persons who are competent and motivated, all faculty must play a role in and assume responsibility for assessment quality. Finally, assessment plans should themselves be re-evaluated on a regular basis. As programs develop, implement, review, and refine their learning outcomes assessment plans, they should continually strive for incorporation of the principles and characteristics of successful assessment programs.

Barriers and Challenges

Although the development and implementation of an assessment process might sound relatively easy, it is not. Many potential problems and pitfalls exist that can undermine its success. Several of these are shown in Table 2 and are described below.

Table 2. Assessment-Related Barriers and Challenges

Faculty-Related

Use of unclear, confusing assessment terminology
Lack of education/training in assessment practices
Proceeding too rapidly or at too large a scope
Inefficient use of existing methods/measures
Lack of appreciation of need to make changes

Resource-Related

Time-consuming, difficult process
Complex interrelationships among input, process, and output factors
Financial/budget constraints

Student-Related

Lack of understanding of purpose
Inefficient, time-consuming process
Unfair or unrealistic process
Lack of feedback

Lack of faculty support is often a major barrier to the successful implementation of assessment efforts. To help ensure that faculty participate in and embrace assessment, care should be taken to use a shared language and concepts and to establish the proper supportive environment. Start slowly to establish a shared trust and to avoid overburdening already busy faculty. Share assessment findings as soon as possible with faculty so they can observe progress being made. Use existing evaluation or assessment instruments and data from current courses and learning experiences to the extent appropriate (i.e., care must be taken that the measures are valid and reliable). Encourage faculty who are responsible for covering the same learning outcomes to work together on assessment undertakings.

Faculty may sometimes feel that as long as their graduates pass licensure or similar examinations, additional assessment methods are unnecessary. However, such examinations do not measure all desired outcomes in students, including professional values, ethics, judgment processes, the ability to integrate and apply knowledge in actual practice situations, or skills requiring interpersonal interactions with patients and other health care professionals.

There are several resource-related barriers to the establishment of an ongoing assessment process. Even with the best of intentions, the process of developing clear, measurable learning outcomes, as well as appropriate criteria or objectives for the knowledge, skills, and behaviors involved with those outcomes, can be time-consuming and difficult. This is compounded by the fact that most health sciences faculty lack background and education/training in these areas. Thus, adequate resources are needed to develop, implement, and maintain a sound program assessment plan.

Finally, the students themselves can be barriers to assessment activities. Students can be resistant to or suspicious of assessment efforts if they do not understand their purpose, i.e., to ultimately improve the educational process and their learning, or if they feel the assessment processes take too much of their time, if they are unfair, or if they are unrealistic. As a result, it is important that students understand the purpose of assessment undertakings and that they receive feedback about their performance to enhance learning.

Despite the possible barriers and challenges that exist, assessment efforts can be not only successful, but also enjoyable. Careful planning, incorporation of the key principles and characteristics of effective program assessment processes, enthusiastic faculty that are open to new experiences and change, and appropriate support and resources all help to ensure that a program can implement and maintain ongoing, high quality assessment activities.

Part 2

Part II - Assessment Principles, Approaches, and Barriers

Exercise

Select a student learning outcome for your course or program and develop both direct and indirect measures that could be used to assess this outcome. What criteria will you for each of these direct and indirect measures to determine “success” of your students?
Identify two of the principles and characteristics of effective assessment practices and brainstorm specific ways in which these could be incorporated into your program.
Identify those barriers or challenges to the implementation of successful assessment efforts that you feel could exist in your program. How might each of these be minimized or eliminated?