Introduction to Assessment

Part 1

Module Purpose

Assessment is becoming increasingly important in higher education as a means for demonstrating and promoting quality in student learning. However, this is also an area of much confusion for faculty. Faculty might not understand the assessment terminology being used, why course grades are not sufficient assessment methods, the purposes of assessment, or the various assessment methods that could be employed. This module provides an overview of the evaluation and assessment of student learning in health sciences education from a variety of perspectives, including the assessment of programs, courses, and individual students.

Module Organization

The module is divided into three major Parts as follows:

  • Part I: Introduction, Development of Student Learning Outcomes/Competencies and a Competency-Based Curriculum
    Includes sections involving: Background, Student Learning Outcomes, Development of a Student Learning Outcomes/Competencies and a Competency-Based Curriculum
  • Part II: Assessment Principles, Approaches, and Barriers
    Includes sections involving: Questions & Answers, Principles and Characteristics, Barriers and Challenges
  • Part III: Evaluation/Assessment Methods
    Includes sections involving: Portfolios, Rubrics, Written Testing, Clinical Evaluations, Objective Structured Clinical Examinations (OSCEs), and Simulated Patients

Each part and its associated sections contain objectives for that material, extra readings, and one or more exercises that are designed to help you reinforce and apply the information learned.

Part 2

Objectives

  1. Discuss the importance of assessment in higher education.
  2. Describe the purpose of student learning outcomes in the curriculum.
  3. Describe the components of student learning outcomes and the characteristics of the criteria that should be prepared to better define them.

Required Reading

  1. Trent AM. Outcomes assessment planning: an overview with applications in health sciences. J Vet Med Educ 2002;29:9-19. Read pgs. 9-11.

Background

Formal mechanisms are in place to help judge and certify the quality and effectiveness of higher education institutions (i.e., accreditation). Most academic institutions are accredited by regional accreditation agencies, such as the Higher Learning Commission (HLC), a commission of the North Central Association of Colleges and Schools (the agency that accredits West Virginia University and the other institutions in its geographical area). Regional accreditation qualifies an institution to receive federal financial aid, and it is a prerequisite in order for degree programs (e.g., medicine, pharmacy, nursing, dentistry, engineering, etc.) at that institution to be accredited by their professional organizations. Providing evidence that an institution and its programs are accomplishing their educational goals is of key importance to accrediting agencies.

Over the past several years, the evaluation of a program’s quality and effectiveness has moved from an “input-based” to a more “outcome-based” process. Outcomes are sometimes thought to be items such as the number of graduates of a program. However, when the term outcome-based is used today, it generally refers to the assessment of student learning outcomes. An outcome-based approach places student learning at the center of assuring and advancing education quality.

In the1980s, the publication A Nation at Risk focused primarily on the declining quality of primary and secondary schools. However, it engendered the report, Involvement in Learning: Realizing the Potential of American Higher Education, that identified the need for enhanced student involvement, higher expectations, and the assessment of student learning in postsecondary education. Higher education faculty are quite good at collecting data but less proficient at analyzing those data, especially as they pertain to learning. It is only through data evaluation that the final, critical step in the assessment process can be undertaken. The phrase “closing the feedback loop” is used to describe changes made to a curriculum based on what the faculty have concluded and learned from the analysis and interpretation of assessment data (Figure 1).

Figure 1 - Assessment Model

Changes in the curriculum and its delivery should be based upon the findings from solid assessment data. Institutions must work toward establishing a “culture of assessment” in which there are ongoing, open discussions among faculty about assessment and learning, a willingness to develop learning outcomes and appropriate measures of those outcomes, an interest in collecting data about outcomes, and an eagerness to use this information to make changes to improve student learning.

Despite many years of the assessment movement, few institutions systematically use assessment results to improve the curriculum and student learning. There are several reasons for this (described in greater detail in Part II of the assessment module). However, a lack of literature consensus and confusion about the definitions of many assessment related terms are important contributors. A glossary is provided (Appendix) for several assessment-related terms, including those terms used similarly in the literature. The glossary definitions were selected based upon whether they reflected those used in the majority of published literature. In order to develop an assessment plan for a program, the expectations (i.e., student learning outcomes or competencies) must first be defined clearly. Since the term “competencies” is often used in health sciences education to describe student learning outcomes, these terms will be used interchangeably throughout this module. The remainder of Part I of the Assessment Module reviews some basics about student learning outcomes, followed by an example of a process used by dental hygiene to develop a competency-based curriculum.

Student Learning Outcomes

Many faculty think of the set of courses in a program when they hear the word “curriculum.” However, the curriculum also encompasses the actual learning experiences, student learning outcomes, teaching and learning processes, student evaluation methods used, and program assessment processes employed. Assessment is critical for curriculum development. Assessment not only documents if the learning outcomes of a course or program are being met, but more importantly, it drives actual student learning itself. Why? Because students will determine, and modify, how and what they learn based upon how they are being evaluated. Development of student learning outcomes is the foundation to building courses and curricula. Student learning outcomes should be used to guide content development, the choice of instructional methodologies, and the selection of evaluation methods. Further, learning outcomes should be derived from the educational mission and goals of the program and institution, and they should be congruent with the activities and responsibilities undertaken in actual practice. Learning outcomes provide students with clear expectations of their performance (i.e., what they should be able to do or achieve) by the end of the learning experience. Learning outcomes should be developed at several levels, each of which is consistent with and reflective of the broader levels:

  • Completion of a specific lesson
  • Completion of a unit of instruction
  • Completion of a course
  • Completion of a program of study

Using an outcomes-based curricular approach, the focus is on what students will be able to do rather than what faculty will do (i.e., learning-centered vs. teaching-centered). Instruction/courses are planned around student learning outcomes. Teaching and learning strategies are then developed and courses delivered that support and facilitate outcomes achievement. Thus, the educational environment is actually created as a product of an outcomes-based curriculum.

Student learning outcomes should be explicit and measurable. They should incorporate three components:

  • Knowledge
  • Skills
  • Attitudes/values

For example, suppose the following learning outcome is from an evidence-based medicine course: Students will critically and accurately analyze a published clinical study and summarize its strengths, limitations, and applicability to practice. What is the knowledge component of this outcome? It would include all the information that students would need to know to be able to analyze and interpret a clinical study (e.g., study designs, types of statistical analyses, data handling methods, etc.). What is the skills component? It would include the actual ability to analyze, summarize and apply what is being read. What is the attitudes/values component? It would include the ability to be critical, accurate, and thorough, and to appropriately consider the needs of an individual patient or practice environment when applying the information gained.

Bloom’s taxonomy categorizes cognitive levels of learning on a continuum from simple to complex (i.e., knowledge, comprehension, application, analysis, synthesis, evaluation). Thus, student learning outcomes should also be consistent with the desired cognitive level of learning. In addition to developing professional and practice-related outcomes, outcomes should be included that describe expectations for general abilities (e.g., critical thinking, communication, ethics). Student learning outcomes should also be continually re-examined and revised as needed based on changes in practice or the profession.

For each student learning outcome developed, specific criteria (e.g., expectations, descriptions, objectives) should be prepared for two purposes: to assist the student in achieving the outcomes, and to assist the instructor in evaluating student performance. (Note: some use the term “objectives” in a manner synonymous with learning outcomes; others use “objectives” to refer to the specific criteria that better define the expectations for learning outcomes.) The criteria should have the following characteristics:

  • Criteria should describe clearly each aspect of what the students need to know and do in order to achieve the outcome.
  • Criteria should be described in sufficient detail that another instructor could use the criteria and arrive at the same conclusion about the student’s performance.
  • Criteria should be geared to the appropriate level of expertise for students as well as to the performance level needed to achieve that outcome in actual practice. For example, a learning outcome written for a medicine program might involve the ability of their graduates to prepare a comprehensive therapy plan for an individual patient. A criterion for a first year medical student related to this outcome might be to describe the various types of therapies available for a given disease state. However, a third year student might be expected to not only be proficient at this, but to also be able to analyze the advantages and disadvantages of each type of therapy for a specific patient’s unique characteristics, and to rank the therapeutic options by desirabilit

As indicated previously, learning outcomes and their associated criteria should be used to determine the specific content and instructional methods necessary to achieve the outcomes. Student learning outcomes that require higher cognitive levels of learning (e.g., synthesis, evaluation) will require different types of educational and evaluative methods compared to those that require lower cognitive levels of learning (e.g., knowledge, comprehension). For example, readings and lectures might be reasonable and effective educational strategies for helping students learn the types of therapies used for a disease. However, the use of case studies or simulated patients might be required in order for students to become proficient in selecting the best therapy for a given patient with that disease.

Glossary of Terms

Term Definition Similar Terms Seen in Literature

Assessment

The ongoing systematic process that involves the development of student learning outcomes and the collection, review, and use of information about those outcomes for the purpose of improving student learning and development and to enhance program quality

Evaluation, Student learning outcome assessment, programmatic assessment

Authentic evaluation or assessment*

Determination of knowledge or performance in a context that resembles actual practice

Competence

What students are able to achieve and demonstrate in an environment that replicates conditions found in a real-life practice setting

Performance

Course-embedded assessment

Determining the extent of student learning in a program by obtaining, analyzing, and interpreting data generated at the course level

Course-based assessment, Embedded assessment

Direct measures

Methods that require students to actually display or demonstrate the extent of their learning

Evaluation

A measure of or conclusion about the extent of student learning that can be formative or summative

Assessment

Formative evaluation or assessment*

Providing feedback for improvement and not for making final decisions, accountability, or to affect student progress through a course or curriculum

Generalizability

Global reliability that includes inter-rater reliability as well as other contributing factors

Goals

Broad or general expectations for learning concepts or student learning

Indirect measures

Obtaining reflections or opinions about student learning without actual demonstration

Performance

Routine, repeated engagement in professional activities in an actual real-life practice setting

Competence

Performance criteria

Specific descriptions and expectations of what students need to know or do to successfully achieve a learning outcome that also serve as the basis for making judgments about achievement

Performance evaluation or assessment*

Measures to determine students’ abilities and skills that are undertaken in real-life, non-simulated settings

Competency evaluation, Competency assessment

Rubric

A tool used for scoring or evaluating work that specifies the criteria used to differentiate levels of performance (e.g., what is needed to achieve a score of excellent [4] vs. good [3] vs. satisfactory [2] vs. poor [1])

Checklist, Rating scale

Student learning outcome

The specific knowledge, skills, and behavior/attitudes/values that students are expected or intended to achieve through their college experience

Objective, Goal, Outcome, Educational outcome, Ability-based outcome, Competency, Programmatic outcome

Summative evaluation

A sum total or final product measure of achievement at the end of an instructional unit or course of study that can affect a student’s progress in a course or curriculum

Summative assessment

Value-added

A positive change in learning outcomes measured following instruction or educational experiences; the difference in students’ knowledge, skills and attitudes before vs. after completing the educational experience

Learning gain, Longitudinal change

*Use of the term “assessment” or “evaluation” should depend upon the context in which it is being used (assessment = ongoing, systematic, generates findings that are used to make changes; evaluation = measure of or conclusion about learning)

From: Abate MA, Stamatakis MS, Haggett RR. Excellence in curriculum development and assessment. Am J Pharm Educ 2003;67(3):478-500 (article 89).

Part 3

Exercise 1

  1. Which of the following should be developed first when preparing a new course – the course content or the intended student learning outcomes? Explain.
  2. What are some of the intended student learning outcomes for your course/area of instruction that would support your program’s outcomes?
  3. Do you base the selection of instructional approaches and evaluation methods in your course upon desired student learning outcomes/competencies? Why or why not?
  4. According to the article by Trent:
    • How are objectives/outcomes in health care education commonly phrased?
    • How is “competency” differentiated from “performance?”