Fill - Free fillable LLU Assessment Guide 2020-01-21 Final(1) (Loma Linda University) PDF form

A S S E S S M E N T G U I D E

Office of Educational Effectiveness

Updated January 21, 2020

Publishing Note:

The Guide is published once a year and is available online. Addendums with

information pertinent to the current academic year will also be posted at:

www.llu.edu/central/assessment/assessment.page

Dustin Jones, Previous Associate Director of Public Relations Cover design

Marilyn Eggers, Office of Educational Effectiveness Director & Associate Provost Editor, designer

Janelle Carillo, Office of Educational Effectiveness Program Manager Assistant editor, designer

Developed in collaboration with Loma Linda University Assessment Committee

LOMA LINDA UNIVERSITY – Assessment Guide

Editing & Design

Co-chairs:

Laura Alipoon, Chair, Radiology Technology, School of Allied Health Professions

Donna Gurule, Associate Dean for Academic Administration, School of Public Health

Members

Adam Arechiga, Associate Dean, School of Behavioral Health

- Lisa Butler, School of Nursing

- Kurt Cao, School of Nursing

- Euni-Hwi Cho, Director of Assessment, School of Dentistry

- Kate Cockrill, School of Allied Health Professions

- Lynda Daniel-Underwood, Assistant Dean, Clinical Site Recruitment, School of Medicine

- Katherine Davis, Assistant Professor, School of Allied Health Professions

- Marilyn Eggers, Director, Office of Educational Effectiveness; Associate Provost

Hansel Fletcher, Assistant Dean, Graduate Student Affairs, Faculty Graduate Students

Mark Fletcher,

Business Analyst

- Michelle Hamilton, Assessment Specialist, School of Public Health

- Natalie Hohensee, School of Dentistry

- Kathryn Knecht, Associate Professor, School of Pharmacy

- Stanley Matsuda, Director of Education Assessment, School of Pharmacy

Brigit Mendoza, Assistant Professor, School of Allied Health Professions

John, Nassif, Manager Curriculum Services, School of Medicine

Ken Nelson, Associate Director, Office of Educational Effectiveness

Leo Ranzolin, Associate Dean, School of Religion

Karen Ripley, School of Nursing

Karen Saul, School of Behavioral Health

Ernie Schwab, Academic Dean, School of Allied Health Professions

Donna Thorpe, Associate Professor, School of Allied Health Professions

John Wical, Director, Administrative Information Systems

Resource and Contact List

Office of the Provost

•

Ronald Carter, PhD, Provost

Magan Hall 110

Ext. 87616

Assessment Committee

•

Laura Alipoon, EdD, Co-chair

Chair, Radiology Technology, SAHP

Ext. 47273

lalipoon@llu.edu

•

Donna Gurule, DrPH, Associate Dean

for Academic Administration, SPH

Ext. 44598

dgurule@llu.edu

Provides support for Deans and

Academic Deans, incorporates

information into the University Strategic

Plan.

Provides consultation for all stages of the

assessment process.

Provides consultation for all stages of the

assessment process.

Office of Educational Effectiveness

•

Marilyn Eggers, PhD, Director, Associate Provost

Magan Hall 104

Ext. 15042

meggers@llu.edu

Provides assistance with assessment,

University Institutional Learning

Outcomes, Mission Focused Learning

Outcomes, and program review.

•

Ken Nelson, MD, Associate Director,

Institutional Researcher

Library Rm 308

Ext. 44195

wknelson@llu.edu

Provides assistance with program department

school and university data e.g., admission,

enrollment, cohorts, graduation, retention,

grades, faculty, etc.

•

Kathy Davis, MS, Associate Director

(PT)

Magan Hall 101

Ext. 55896

kdavis@llu.edu

Provides consultation for all stages of the

assessment.

•

Janelle Carillo, MBA, Program Manager

Supports assessment and program review.

Magan Hall 104

Ext. 15042

assessment@llu.edu

LOMA LINDA UNIVERSITY – Assessment Guide

Table of Contents

Introduction to Guide ................................................................................................................... 3

LLU Institutional Learning Outcomes (ILOs) and Assessment Strategy.............................. 4

Mission Focused Learning Outcomes ....................................................................................... 4

Professional Institutional Learning Outcomes ......................................................................... 6

Professional ILOs in Watermark’s LiveText and via™ ........................................... 7

Program Assessment Plan............................................................................................................ 8

Program Learning Outcomes (PLOs) ............................................................................ 8

Performance Indicators for ILOs and PLOs ............................................................... 10

Curriculum Map ............................................................................................................. 12

Assessment Matrix ......................................................................................................... 14

Watermark’s LiveText and via™ .............................................................................................. 16

LLU Annual Reports ................................................................................................................... 17

Annual Faculty Report (Faculty Portfolio) ................................................................. 17

Institutional Learning Outcomes Assessment Report (AMS tab: “Learning

Outcomes Analysis) …. ................................................................................................. 20

Annual Action Plan Report .......................................................................................... 22

Office of Educational Effectiveness Contact Information ..................................................... 24

Appendices .............................................................................................................................. …25

Assessment Resources ................................................................................................... 26

LLU Assessment Guide

January 21, 2020

Professional Institutional Learning Outcomes Resources........................................ 29

LLU Assessment Glossary ............................................................................................ 33

School Assessment Specialist Position Description ................................................. 37

LLU Templates:

Curriculum Map ............................................................................................................. 39

Assessment Matrix ......................................................................................................... 41

ILO Rubrics

Academic and Professional Information Literacy ...................................................... 43

Quantitative Reasoning ................................................................................................ 45

Critical Thinking........................................................................................................... 49

Professional Critical Thinking ..................................................................................... 50

Oral Communication ................................................................................................... 53

Professional Oral Communication............................................................................. 54

Academic and Professional Written Communication .. ......................................... 56

Mission Focused Learning (MFL) Rubrics

Wholeness Rubric …… .................................................................................................. 58

MFL Resources

MFL Standards for Course Design and Teaching ..................................................... 59

LLU Assessment Guide

January 21, 2020

Introduction

The Loma Linda University Assessment Guide is designed for busy faculty in the

following ways:

• Provides essential assessment information in a quick-to-read format of brief

narratives and bullet lists.

• Demystifies how to develop or update a Program Assessment Plan.

• Gives step-by-step directions about how to fill out the three annual reports due

at the end of every October:

o Faculty Portfolio (aka: Faculty Profile and Annual Faculty Report) – all

faculty

o Institutional Learning Outcome Assessment Report – all programs

o Annual Action Plan – all programs

• Includes assessment resources in the appendices for those who want to learn

more about assessment.

• Offers a glossary for assessment and program review terms used at LLU.

• Provides templates and all the LLU Institutional Learning Outcomes rubrics for

easy reference while reading the guide. In addition, all of these resources are

posted at the Office of Educational Effectiveness’ assessment website

Important Changes

ILO Assessment for All Students: All LLU students need to be assessed on all five of

the Institutional Learning Outcomes regardless of the length of programs. For more

information see section: LLU Learning Outcomes and Assessment Strategy.

Mission Focused Learning (MFL) Outcomes: A new self-assessment Wholeness rubric

(p. 58) is now available. It can be used by students, faculty, staff, and administrators. In

addition, there are new MFL Standards for course design, development, and teaching.

Both are available in the appendices. Please see section LLU Learning Outcomes and

Assessment Strategy, for more information.

To learn more about Mission Focused Learning

and the MFL Standards for course

design please review the OEE website.

Formative Assessment: In addition to the required Summative Assessment for ILOs and

PLOs, a Formative Assessment is now required. If programs only conduct a Summative

Assessment, it doesn’t allow time to address any learning gaps that may exist with the

current students. However, by utilizing a mid-program Formative Assessment it

provides the opportunity for programs to address any discovered learning gaps while

there is still time to make changes to help current students successfully master the ILOs

and PLOs. Of course, it’s not expected that students will do as well on the mid-program

Formative Assessment as they do on the end-of-program Summative Assessment.

LLU Assessment and Program Review Resources: http://www.llu.edu/assessment/

Mission Focused Learning: https://home.llu.edu/education/office-of-provost/departments-and-

divisions/educational-effectiveness/mission-focused-learning

BACK TO CONTENTS

LLU Assessment Guide

January 21, 2020

LLU Learning Outcomes and Assessment Strategy

Institutional Learning Outcomes

Loma Linda University’s Institutional Learning Outcomes (ILOs) for students are

assessed throughout the academic degree programs within the University appropriate

for their disciplines and degrees. The Office of Educational Effectiveness works with

these programs to guide their assessment. For more in depth information about LLU’s

ILO assessment, please see: http://www.llu.edu/central/assessment

1. Critical Thinking: Students demonstrate critical thinking through examination

of ideas and evidence before formulating an opinion or conclusion.

2. Information Literacy: Students demonstrate the ability to identify, locate,

evaluate, utilize, and share information.

3. Oral Communication: Students demonstrate effective oral communication skills

in English.

4. Quantitative Reasoning: Students demonstrate the ability to reason and develop

evidence-based decisions using numerical information.

5. Written Communication: Students demonstrate effective written communication

skills in English.

Mission Focused Learning Outcomes

Loma Linda University’s three Mission Focused Learning Outcomes (MFLOs) are firmly

rooted in its mission, vision, and values

. Because Mission Focused Learning is LLU’s

culture, the University has developed a specialized assessment wholeness rubric to

ensure integration of these outcomes over time.

• Wholeness

: Students apply the University philosophy of wholeness into their

personal and professional lives.

• Values: Students integrate LLU’s Christ-centered values in their personal and

professional lives.

All Students Assessed on All Five ILOs

LLU requires that all students be assessed on all five of the Institutional Learning

Outcomes. For many programs the most effective and easy way to accomplish this is to

assess every ILO every year. Since the length of programs vary, the reporting schedule

for one to two ILOs per year is no longer in place. The very short ILO assessment report

should be entered in the Assessment Management System (AMS). Please see

Institutional Learning Outcome Assessment Report on how to submit an ILO report.

LLU Values: Compassion, Excellence, Freedom, Integrity, Humility, Justice, Purity/Self-control

http://www.llu.edu/central/values.page

Wholeness: Loved by God, growing in health, living with purpose in community.

BACK TO CONTENTS

LLU Assessment Guide

January 21, 2020

Undergraduate ILO Assessment

Assessing. LLU requires all undergraduate students to be assessed on all five of the

Institutional Learning Outcomes regardless of the length of programs. For example, a

one-year undergraduate program would need to assess all students—on all five ILOs—

in the one year. Programs longer than one year would assess all five ILOs during the

length of the program. LLU recommends using Watermark’s Livetext and via™ for

conducting assessments. For more information see LLU LiveText by Watermark.

Reporting. LLU requires all undergraduate programs to report on all five of the

Institutional Learning Outcomes for every cohort. Thus, the program would be reporting

every year on all five ILOs. This report must be submitted in the Assessment

Management System (AMS) under the tab “Learning Outcomes Analysis.” For more

information see “Institutional Learning Outcome Assessment Report” on p. 20.

Graduate ILO Assessment

Assessing. LLU requires all graduate students to be assessed on all five of the

Institutional Learning Outcomes regardless of the length of programs. For example, a

one-year post-baccalaureate graduate program would need to assess all students—on all

five ILOs—in the one year. Programs longer than one year would assess all five ILOs

during the length of the program. LLU recommends using Watermark’s Livetext and

via™ for conducting assessments. For more information see LLU LiveText by

Watermark.

Reporting. LLU requires all graduate programs to report on all five of the Institutional

Learning Outcomes for all students. This report must be submitted in the Assessment

Management System (AMS) under the tab “Learning Outcomes Analysis.” For more

information see “Institutional Learning Outcome Assessment Report” on p. 20.

Sample ILO assessment Reporting Schedule for 2-Year Programs

Year 1

Year 2

Critical Thinking

Information Literacy

Oral and Written Communication

Quantitative Reasoning

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

Professional Institutional Learning Outcomes

Rationale

In today’s world it is important for all LLU graduates to have excellent professional

skills in addition to being experts in their fields in both knowledge and skills. Certainly,

LLU graduates should have excellent critical thinking, oral and written communication,

information literacy, and quantitative reasoning skills that are appropriate to their

discipline and level. Whole patient care depends on these kinds of professionals. Thus,

professional versions of the rubrics or those from professional accreditors are

appropriate to use with students in clinical or skills-intensive programs. It is an

alternative assessment approach for clinical programs that have trouble using the

standard academic Institutional Learning Outcomes (ILOs) and rubrics to fit their

discipline/profession and level.

Professional Institutional Learning Outcomes: Professional programs and skills-

intensive disciplines may adapt and assess LLU’s ILOs and rubrics to better meet their

unique learning and assessment needs.

Alternative Assessment Process

LLU has new freedom and responsibility to define, assess, and document learning in our

own way. The following alternative approach to the LLU’s existing ILOs and assessment

tools is one of the initial ways for programs to meet their unique needs:

1. Transform the Institutional Learning Outcomes (ILOs) into Professional

Institutional Learning Outcomes (Professional ILOs). Note: The Professional ILOs

do not replace the regular Program Learning Outcomes (PLOs) that address the

specific curriculum and skills of the program’s discipline or profession.

2. Develop definitions for the ILOs that are meaningful for the program’s discipline

and level.

3. Select or develop a rubric that accurately assesses student learning for the

program’s new Professional ILO definitions. Programs may continue to use the

current LLU/AAC&U VALUE Rubrics or a rubric used within the program’s

discipline/profession. Alternatively, programs may choose to use clinical

versions of the LLU/AAC&U rubrics developed by the Learning Outcomes

Committee.

All programs have the option to choose whether to use the regular ILOs or to move to

the Professional ILO alternative assessment process. This can vary by ILO.

Currently, the following ILOs have a Professional ILO Rubric:

• Critical Thinking

• Oral Communication

• Written Communication

All degree programs must assess all students on all the ILOs during their program.

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

Professional ILOs in Watermark’s LiveText and via™

How will changing from LLU’s ILOs to Professional ILOs impact using Watermark’s

Livetext and via™ and the overall assessment reporting for LLU?

LLU ILO Rubrics: All of the LLU existing ILO and Professional ILOs rubrics will be

in Watermark’s Livetext Professional ILO Rubrics: New Professional ILO rubrics

will need to be added to Watermark’s Livetext and via™ and tagged for their

corresponding ILOs. As a result of this tagging, data reports can be run in

Watermark’s Livetext and via™ as usual even when programs use different rubrics.

Support: Workshops are available to help programs set up their new rubrics after

they either have developed their own rubrics or have selected existing rubrics from

their discipline/profession. Once the rubrics are set up and tagged properly in

Watermark’s Livetext and via™, faculty will be able to use them for assessments as

usual. Contact assessment@llu.edu to schedule a workshop.

Annual Reports: All programs will complete the annual assessment reports, as

usual, based on their assessments using either the traditional ILOs and rubrics,

LLU’s new PILO rubrics or their own Professional ILOs with appropriate rubrics.

Assessment Schedule: As noted earlier in this Guide all students need to be assessed

on all five ILOs. Programs will now be able to get a better understanding of where

their students are each year and can take appropriate actions if they are not meeting

the criteria for success. This approach is considered best practice.

Professional Programs: Programs may choose whether they use the ILOs or

Professional ILOs. They can transform the ILO into appropriate Professional ILOs

for their programs, or they can keep some of the ILOs and only use a few

Professional ILOs. No matter how a program selects ILOs and/or PILOs, there will

only be five all together with no duplicates. The final selection should be a great

match for the program’s discipline and level.

See Professional Learning Outcomes Resources, p. 29, for studies regarding each of the

five ILOs implemented in professional programs

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

Program Assessment Plan

There are three components to the program assessment plan: (1) program learning

outcomes with performance indicators, (2) a curriculum map, and (3) an assessment

matrix. These tools are used to plan and monitor the program’s curriculum, teaching,

and assessment.

WSCUC requires each of its institutions to have all of their degree programs complete

the Inventory for Educational Effectiveness Indicators (IEEI) twice in an accreditation

reaffirmation cycle: Mid-Cycle Review and Accreditation Reaffirmation. Consequently,

LLU must complete form again for our WSCUC review. WSCUC must collect and

compile all institution’s completed IEEI reports and send the consolidated report to the

federal Department of Education. Fortunately, most of the required information for this

form can be found in the Program Assessment Plan. LLU needs 100% completion of this

form.

Program Learning Outcomes (PLOs)

All programs have expected student learning program outcomes. Whether you are

reviewing and updating your program’s existing PLOs, or, if you are developing new

PLOs for the first time, the process is the same.

Program learning outcomes are the program’s expectations for student learning.

• Ask the question, “What do we want our students to learn, know, or be able to

do by the end of this program?”

• Responses to this question will guide the identification and development of the

program’s outcomes.

Professional Accreditation: Programs with professional accreditation will be guided by

their professional accrediting agency’s expectations and requirements for the

profession/discipline.

Remember: These outcomes should cover the big picture of student learning in the

program. This is not the place for specific, detailed course competencies or objectives.

All courses should address at least one of the PLOs. Below are some key definitions,

concepts, and guidelines.

Learning Outcomes

The knowledge, skill, attitudes, values, etc., that students should be able to

demonstrate by the end of the program. ~Gloria Rogers

Program learning outcomes should:

• Build on what already formally or informally guides the program.

• Be limited to 5 to 7.

• Be clear, concise, AND measurable.

• Have 1 to 3 performance indicators to measure each outcome. Each performance

indicator must be assessed.

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

Process of Developing Program Learning Outcomes

1. Review your program’s professional or programmatic competencies.

2. Condense, combine, and/or collapse the resulting summative outcomes down to

5-7 outcomes. These outcomes should cover the full scope of the program.

3. Put resulting outcomes into a standard format.

Gloria Roger’s Example – Standard Format

Learning outcome: Demonstrate ethical responsibilities.

Performance criteria:

1) Apply knowledge of professional code of ethics.

2) Evaluate the ethical dimensions of a problem in the discipline.

Bloom’s Taxonomy Resources

Bloom’s Taxonomy has good lists of active verbs that can strengthen PLO

development. Below are several resources. There are many more on the web.

All Three Domains

Blooms Taxonomy, Learning Objectives and Higher Order Thinking

https://www.unthsc.edu/center-for-innovative-learning/blooms-taxonomy-learning-

objectives-and-higher-order-thinking/

Cognitive Domain

Developing Great Objectives [Outcomes]: New Bloom’s Taxonomy (Medical

perspective)

https://www.evms.edu/education/medical_programs/doctor_of_medicine/instructor_too

ls/learning/developing_objectives/blooms_taxonomy/

How to Design and Use Learning Objectives in Clinical Teaching

Especially good for developing CLOs

https://www.pharmaceutical-journal.com/learning/learning-article/how-to-design-and-

use-learning-objectives-in-clinical-teaching/20200251.article?firstPass=false

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

Performance Indicators for ILOs and PLOs

Performance indicators describe specifically how the learning outcome will be measured.

Typically, learning outcomes can be assessed in many different ways, thus each one

needs from 1-3 performance indicators to describe specific assessments.

LLU Institutional Learning Outcomes (ILOs) Performance Indicators (PIs)

Performance Indicators are not provided for the ILOs. Programs need to develop their

own PIs for the ILOs to better meet their discipline/professional and level needs.

Value of Performance Indicators and Collected Data

1. To focus and motivate students, faculty, and staff toward achieving results

2. To communicate achievements to university and community stakeholders, and

prospective students

USAID Center for Development Information and Evaluation

Two Essential Parts of Performance Indicators

1. Content reference: Subject content that is the focus of instruction (e.g., steps of the

design process, chemical reaction, scientific method).

2. Action verb: Direct students to a specific performance (e.g., “list,” “analyze,”

“apply”)

Gloria Rogers

Example: Mission Focused Learning Outcome

1. Students apply the University philosophy of wholeness into their personal

and professional lives.

a. Demonstrate knowledge of LLU’s philosophy of wholeness.

b. Plan a personal strategy for wholeness and implement it.

Types of Measures

1. Direct – The assessment is based on an analysis of student behaviors or products

in which they demonstrate how well they have mastered.

2. Indirect– The assessment is based on an analysis of reported perceptions about

student mastery of learning outcomes.

Performance Indicator Principles

• There should be at least one direct measure for each outcome.

• Develop 1-3 recommended performance indicators appropriate for the discipline

and level for each of the five LLU ILOs and the program’s 5-7 PLOs.

Allen, M. J. (2008). “Strategies for Direct and Indirect Assessment of Student Learning.”

Retrieved on November 29, 2017 from:

http://academics.lmu.edu/spee/officeofassessment/assessmentresources/selectinganassessmentme

asure/

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

Writing ILO and PLO Performances Indicators

1. Analyze a learning outcome to determine how it is currently being assessed or

how it could be assessed in the program.

2. Develop a statement that indicates the method of assessment along with the

specific characteristics students should exhibit to show achievement.

Examples of Direct Measures

• Exit and other interviews

• Standardized exams, only if questions are mapped to outcomes

• Locally developed exams, only if questions are mapped to outcomes

• Portfolios

• Simulations

• Performance appraisal

• External examiner

• Oral exams

• Behavioral observations

Examples of Indirect Measures (self-assessments)

• Written surveys and questionnaires

• Exit and other interviews (yes, they can also be direct measures!)

• Archival records

• Focus groups

NOTE: Although the following methods are used to evaluate student learning, they are

not accepted as assessment for specific ILOs or PLOs:

• Course evaluations

• Grades

• GPAs

• Standardized, program, and course test total scores

o However, there is a way to analyze scores for specific questions in a test

that are directly linked to specific ILOs or PLOs, the resulting scores

would be acceptable as ILO or PLO assessments. Using ExamSoft is one

way to track specific learning outcomes to particular exam questions.

There should be enough questions related to the learning outcome to

assess student learning adequately.

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

Developing a Curriculum Map

Curriculum maps give programs a mechanism to organize curriculum in a logical and

reasonable manner to support ILOs and PLOs. They encourage faculty to rethink what is

taught, how learning is assessed, and a process in which to focus on the goal of

implementing the Loma Linda University ILOs as well as the program learning

outcomes (PLOs). The curriculum map ensures that the ILOs and PLOs will be

implemented into the program’s courses. Not only does this strengthen the curriculum,

but it also helps to ensure the program will not stray from the PLOs, ILOs, and LLU’s

mission.

Six Basic Steps on How to Prepare a Curriculum Map

in a Table:

1. List the five institutional learning outcomes (ILOs) and the 5-7 programmatic

outcomes across the top of the table.

2. List all of the program’s course numbers on the vertical axis of the table.

3. Review each course to determine what intentional educational strategy supports

or helps students to achieve a specific outcome. For example, where instruction

will be given and at what level.

4. Identify to what extent each course addresses the outcome in the instruction and

note it on the map.

5. Identify the assessment level, if any.

6. Repeat the last three steps for each course and outcome.

Learning Outcome Implementation in Courses’ Instruction

• Determine the extent that each outcome is implemented in each course.

• Choose the appropriate level of instruction that is most appropriate for each

course or if the program is externally accredited, use the required scale. Here is

one example that would be appropriate for all programs:

o I = Introduced

o E = Expanded

o A = Advanced

• Enter an I, E, or A in a cell whenever a course intentionally addresses a learning

outcome at one of these levels. A few courses may have more than one of these

levels. Only list the specific outcomes that are explicitly addressed in a course.

Assessment with Results Tracked over Time:

In addition to indicating the level of instruction for every learning outcome, assessment

also needs to be indicated. Most of the time there will be only one of the designations

below per outcome; however in some unusual circumstances two might be in the course.

Only one level per outcome is permitted.

• B = Baseline – Assessment at the beginning of program. Recommended; tracked

by the program. This assessment shows the level of skills or learning of students

when they enter the program.

• F = Formative – Assessment at the middle of the program. Required; tracked by

the program and the University. Mid-program assessment gives the program the

LLU curriculum map template: http://www.llu.edu/central/assessment/assessment.page

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

opportunity to make any necessary changes for the current students who were

assessed, if they did not meet the criteria for success. This is the only opportunity

for the program to fill this gap, if one is found. The summary formative report

should be included briefly in the annual ILO summative assessment report.

• S = Summative – Assessment at the end of the program. Required; tracked by

the program and the University. Shows the students’ final level of success for the

indicated learning outcome. If students did not meet the criteria for success, the

program needs to determine what changes need to be made to improve student

learning. This is closing the loop to make improvements for future students.

How to Analyze a Curriculum Map

Review the curriculum map with these key principles and then make changes as needed.

• Every course needs to have at least one ILO and one PLO.

• Every ILO and PLO needs to have at least one course.

What to do with a Course without an ILO or PLO

• Each learning outcome (ILO and PLO) must be addressed in at least one

course.

If a course does not have an ILO, review the ILOs carefully to find one to fit

the course, and add it to the curriculum map.

• If a course does not have a PLO, evaluate it carefully. Then take one of

these actions:

o Redesign the course to include at least one appropriate PLO.

o Develop a new PLO that is needed for the program and is

addressed in the course.

o Evaluate the course further to see if it should be eliminated

because it does not support any of the required PLOs.

What to do with an ILO or PLO without a Course

• Evaluate the learning outcome carefully, then take one of these

appropriate actions:

o Re-examine the outcome to see if it should be revised or

eliminated.

o Modify an appropriate current course to address the ILO or PLO.

o Develop a new course to address the orphan learning outcome.

Keep It Current: Update the program’s curriculum map

whenever there is a change in

the curriculum, so it is always current. This will make it a valuable resource for the

program.

Conclusion

Going through this development and evaluation process will provide an accurate

program curriculum map that will give a comprehensive overview of where and to what

extent the program’s courses implement and assess the five LLU ILOs and the 5-7

program outcomes.

The LLU Curriculum Map and the Assessment Matrix templates have been updated and can be

found at: http://www.llu.edu/central/assessment/assessment.page

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

Developing an Assessment Matrix

An assessment matrix is a tool to organize and track how the LLU Institutional Learning

Outcomes (ILOs) and Program Learning Outcomes (PLOs) will be assessed. The

required ILO assessment report and voluntarily submitted PLO assessment report

should be entered into the AMS.

Learning Outcomes Analysis tab. An up-to-date assessment matrix will make the

submission of the ILOs’ assessment reports much easier to complete. Please see page ___

for the LLU Assessment Matrix Template.

Two Sections

There are two sections that ask for different kinds of information:

1. Where the learning outcomes are published

2. A detailed learning outcomes assessment plan and results

First Section: Publishing Outcomes

Programs should publish their learning outcomes in all of the following locations so

potential students and current students can see the program’s commitment to what they

will learn:

• University catalog

• Program web site

• Course syllabi

• Other program materials

Second Section: A Detailed Learning Outcomes Assessment Plan

Column 1: Learning Outcomes

• University ILOs are already included in the template posted at the Office of

Educational Effectiveness assessment website

• Add the Program Learning Outcomes.

Column 2: Performance Indicators (PIs)

• Add the learning outcomes’ performance indicators for each ILO and PLO.

Column 3: Assessment Tools & Data Collection Cycles

Indicate the following for each learning outcome:

1. Existing assessment tools already in use:

▪ Student assignment, project, lab, etc.

▪ Rubric or other tool to assess the students’ work on the

assignment, project, lab, etc. If not using one of the LLU ILO or

Professional ILO rubrics, upload it in the ILO report in the AMS.

See ILO report in p. 20.

LLU Assessment: http://www.llu.edu/central/assessment/assessment.page

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

2. New assessment tools may be developed or required by either the

program or the university. Currently LLU is using Watermark’s LiveText

and via. Some programs professional accreditors require a specific

assessment tool.

3. The data collection cycle (e.g., end of every quarter, annually, every other

year, etc.)

Column 4: Criteria for Success

How will programs know if student learning is successful?

• Set the level of success for each learning outcome’s performance indicator’s

assessment across the program.

o Example 1: “85% of the students will attend at least one professional

meeting; 50% will present at such meetings.”

o Example 2: “80% of the students will achieve level 3 or higher on the

University rubric.”

• Note: Course and test grades are not considered to be learning outcomes

assessment unless specific learning outcomes are mapped to individual test

questions. Course evaluations are also not considered learning outcome

assessments but rather are indicators of student satisfaction.

Column 5: Who interprets the assessment data? What is the process?

• Document the evaluation process for the program’s assessment data (who does it

and how is it done). For example, “Course instructor(s) conducts the

assessments;” “Program faculty team assess the culminating projects/paper,” etc.

• Carefully analyze existing data collection processes. Each outcome should have

at least one direct assessment. If needed, make the necessary changes.

Column 6: Findings from Data Collection

• Analyze the collected data for each ILO and PLO.

• Look for meaningful findings.

• Was the criteria for success met? (“yes” or “no”)

Column 7: Resulting Program Changes

Finally, close the assessment loop by making necessary changes whenever the criteria

for success on a learning outcome has not been met:

• If the “Criteria for Success”(see above) has been met, then state NA. No changes

are necessary.

• If the “Criteria for Success” has not been met, note the resulting course or

program changes that either have already been implemented or will be made

soon.

NOTE: The purpose of assessment is to improve student learning. Closing the loop

when the “Criteria for Success” for a learning outcome has not been met is the perfect

opportunity to make changes to the program designed to improve student learning.

See p. 20 for the annual ILO/Professional ILO assessment report.

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

Watermark’s LiveText and via™ at LLU

Loma Linda University (LLU) has selected Watermark’s Livetext and via™

as the

campus assessment and electronic portfolio tool of choice for schools and programs.

Watermark "is an Internet-based subscription service that allows students and

instructors to create, share, and collaborate on educational curriculum"

. Once set up,

faculty and students can use it as part of regular course work for ILO or PLO and/or for

portfolio assessments.

LLU Institutional Learning Outcomes

(ILOs) rubrics have already been uploaded and

are ready to use in Watermark’s Livetext and via™. This will make assessment of ILOs

easier and faster for faculty. In addition, it is easy to set up rubrics for program and

course learning outcomes as well. Watermark’s Livetext and via™ can be used for many

other things including student e-Portfolios, Field Experience Management

(FEM) to

assist with internships, practicums, accreditation, and so on.

To get started in Watermark’s Livetext and via™, please request a faculty account by

emailing edtech@llu.edu. Once you have signed up for your faculty account, email your

school's Watermark’s Livetext and via™ administrator requesting the LLU Metarubrics

so you can begin assessing the ILOs!

To find schedules for Watermark’s Livetext and via™ training at LLU, and other up-to-

date LLU resources, please go to the LLU Watermark support page

Have questions about Watermark’s Livetext and via™? Contact your school's

assessment technology administrator:

School

Contact

Extension

Email Address

Allied Health Professions

Kate Cockrill

Kathy Davis

44068

88477

kcockrill@llu.edu

kdavis@llu.edu

Behavioral Health

Karen Saul

55463

ksaul@llu.edu

Dentistry

Christina Gutierrez

44601

cpgutierrez@llu.edu

Medicine

John Nassif

87223

jnassif@llu.edu

Nursing

Lisa Butler

44736

lsbutler@llu.edu

Pharmacy

Stanley Matsuda

87731

smatsuda@llu.edu

Public Health

Wendy Genovez

88088

wgenovez@llu.edu

Religion

Magi Armany

92505

marmany@llu.edu

LLU Support

Doris Serna

49719

edtech@llu.edu

Watermark’s LiveText and via™ - https://www.Watermark.com/

Eastern Michigan University College of Education Watermark FAQ

LLU ILOs - http://www.llu.edu/central/assessment/ilo.page

Watermark’s LiveText and via™: What We Do - https://www.Watermark.com/what-we-do/

LLU Watermark Support Page - http://www.llu.edu/central/assessment/Watermarkintro.page

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

LLU Annual Reports

Three LLU annual reports are due each year at the end of October: (1) Faculty Portfolio,

(2) Annual Action Plan, and (3) Institutional Learning Outcome Assessment.

1. Annual Faculty Report

also called “Faculty Profile” and “Faculty Portfolio.”

Who is responsible to complete: All full-time and half-time faculty

Due date: End of October each year

This report is needed so programs, departments, schools, and the University can learn

about significant faculty contributions to help administrators make informed decisions

and plans. It is also useful for meeting both WSCUC and professional/discipline-specific

accreditation requirements.

What’s in it for the faculty? There are several benefits of keeping profiles each year:

• Provides an online professional profile in the LLU Faculty Directory

for public

viewing.

• Keeps current CVs in your Faculty Profile (safe, editable, and downloadable—see

CVeditor in left menu)

• Your contributions can be recognized and mined by LLU.

Portfolio Location: To find your personal profile:

1. Go to the University Desktop on the One Portal

a. https://one.lluh.org > select Apps menu > click University Desktop

b. University Desktop direct link: https://one.lluh.org/vip/apps/university-

desktop

2. Click on Faculty Profile under Faculty Portals section

a. Faculty Profile direct link: https://myllu.llu.edu/facultyPortfolio/

Data Entry Tips: There are just a few things to remember:

• Plus sign (+) – Click on the plus sign to add a new item in the desired category.

• Pencil ( ) – Click on the pencil by the item that you want to edit.

• Item sections – Inside each category new item are several sections. Some vary

from category to category but most are similar, such as:

o Change Activity Type – A pull-down menu at the top of the item. Select the

appropriate subcategory.

o Title and Description – A textbox is given for both the title and description.

Copy text from your CV and paste into the textboxes. There are basic

word processing-like tools available when needed.

▪ Internal Audience Only: Click on the small arrow next to Internal

Faculty Profile: https://myllu.llu.edu/facultyPortfolio/

LLU Faculty Directory: http://www.llu.edu/pages/faculty/directory/index.html

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

Audience Only. If desired, enter a message for administration

only. Even if the item is marked for Public Display, the “Internal

Audience Only” note will not be visable to the public.

▪ Enter a fixed end date only when you no longer are teaching the

course. If you want the end date to be “to present,” in the drop-

down menu, scroll up on both the day and the year to select

“blank” or “None” respectively.

o Public Display: If you want the item to show up in your profile on the LLU

Faculty Directory, select “Yes.” If you don’t want it to be publically

available, select “No.” Either way, the University will be able to see it to

report internally.

o Save: Don’t forget to Save, or your work will be lost!

o Delete Item: If you want to delete an activity item, click in the box beside

“Delete this item?” at the bottom of the item entry window. Then click

Save. There is no undo for delete, so use it carefully.

Annual Faculty Report Items: There are key areas to be completed that most likely will

not need to be updated every year. They are:

• Educational History

• Employment History

Annually update the following items as needed:

• Professional Development

• Teaching LLU Courses

• Teaching (Other)

• Research and Grantsmanship

• Publications

• Presentations

• Patient Care

• Service

• Honors and Awards

• Commitment to LLU Mission

• Admission Duties

Discover My Publication: Search your name.

• It may not capture all of your publications, but it will be a helpful start.

Important Last Step Each Year: When you have completed updating your portfolio to

meet the end of October deadline, click on Sign Off Annual Report in the left menu.

Then select the academic year from the date pull-down menu at the top of the window.

It will generate a list of all the activities you entered for the selected year. After

reviewing it, respond to the prompt (see below) and then click Save portfolio status:

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

Please note: Completing this last step will allow administrators to run reports on the

faculty completion status.

Strategy to Relieve Deadline Stress: Eliminate the end of October deadline stress by

updating your portfolio every time you complete an activity; or update it monthly or

quarterly.

For Administrators who need “Faculty Portfolio” Report Permission. If you are a

program director, department chair, academic associate dean, or dean, you may need to

be set up to run reports on Faculty Portfolio data for the faculty in your area of

responsibility. Please contact the Office of Educational Effectiveness at

assessment@llu.edu or extension 15042.

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

2. Institutional Learning Outcome Assessment Report

Who is responsible: All program directors

Date due: End of October each year

ILO Assessment Report Submission

All programs will submit their current year’s ILO summative assessment report to the

LLU Assessment Management System (AMS

) by the end of October every year. The

assessment report is based on the LLU program Assessment Matrix Template and can be

found in the Learning Outcomes Analysis tab. Be sure to note if formative assessment

was done earlier and the results. See Institutional Learning Outcomes on p. 4 for more

information.

Please refer back to section “All Students Assessed on All Five ILOs.”

Choose:

• Academic Year: Select the correct academic year and outcome for the report to be

submitted.

Assessment Tools: (1) Describe the project or other culminating-type assessment given

to students. Also, (2) list the rubric used or other measurement tool perhaps from a

professional/discipline-specific accrediting agency. If the program did not use the LLU

rubric, describe how the culminating assessment was evaluated. Upload it here (click on

the folder icon).

Data Collection Cycle: Note the assessment cycle (e.g., annually or every other year,

etc.) See Institutional Learning Outcomes on p. 4 for more information. Please refer back

to section “All Students Assessed on All Five ILOs.”

Criteria for Success: LLU uses the AAC&U four-point rubrics so indicate the level of

success expected and the percentage of students who must achieve this level in order for

the program to be successful for the designated ILO. For example, “80% of the students

will achieve a three or better on the LLU rubric.”

How Is Data Interpreted? There are various ways that the data can be analyzed and

interpreted: course instructor/s, program director, and program director and faculty. It is

best practice for at least some of the ILO and PLO assessment results to be evaluated by

more than one person such as the course instructor.

Findings and Analysis: Give a brief summary of the data and what it means.

See Learning Outcomes Analysis tab: http://myllu.llu.edu/assessment/programs/

AMS: http://myllu.llu.edu/assessment/programs/ - Learning Outcomes Analysis tab

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

Success Criteria Met? This short response makes it possible to run reports on submitted

assessment data reports. There are only two options: Met, and Not Met. The description

for this item is in “Findings and Analysis.”

• Upload documents with the appropriate tables into the reporting sections they

support. Click on the folder to the right of the desired section’s name to upload

documents.

Resulting Changes: This only needs to be completed if the program did not meet the

criteria for success on the designated ILO or PLO. Describe how the program closed the

loop in order to improve student success on this learning outcome. See ”Developing an

Assessment Matrix” on p. 14.

Save: Your work will not be saved unless you click on Save. Save regularly!

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

3. Annual Program Action Plan Report

(formerly “Annual Program Report”)

Who is responsible: All program directors

Due date: End of October each year

The new Annual Action Plan Report begins with entering the program’s action plan

recommendations into the assessment management system along with the planned

solutions based on the most recent program review or professional accreditation cycle.

Each year this action plan should be updated to show the past academic year’s progress.

When the program completes a new cycle of LLU review or professional accreditation,

enter the new action plan, and then update it annually. Action plans should include all

recommendations from external reviewers during program review and document how

those recommendations have been addressed. Programs are also requested to include

action items they identify during annual assessment cycles, or from strategic planning

and reflection. The dropdown menu for the “Recommendation source “allows you to

choose how the recommendation/issue was identified.

What’s in it for LLU? The Annual Action Plan report is an online database that will

make it possible for administrators at every level to:

• Track trends across their area of responsibility.

• Identify issues to be addressed.

• Understand timelines for solutions.

• Find who is responsible for specific action items.

• Get data to make better informed decisions.

• Locate examples and data for writing accreditation and program review

documents and other reports as well.

What’s in it for You? Faithfully updating the annual program report will:

• Systematically build evidence to be used for making data-informed decisions.

• Help in writing program review and/or accreditation reports at all levels at LLU.

• Helps you to sleep better at night because you know your action plan is

updated!

Tip: Give all program faculty read-only access so they can keep track of what needs to be

done, when, how, and who is responsible for each item (keeps everyone on the same

page). It is recommended to give only one or two people edit access to prevent edit

conflicts. However, some programs divide the responsibilities among the faculty so each

designated faculty would need their own edit access. Send access requests to

assessment@llu.edu.

Annual Report: Action Plan in the AMS

The AMS Action Plan Report has two parts for each recommendation’s Issues & Goals.

Section One: Identify the nature of the recommendation/goal.

• Category: Select the category from pull-down menu for a recommendation.

Action Plan Report: http://myllu.llu.edu/assessment/programs/

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

There are nine categories in which to put recommendations and plans to solve

them. If you only have two recommendations in two categories, the rest of the

categories will remain empty.

1.1 Alignment: Vision, Mission, and Goal

1.2 Alignment: Academic and Professional Trends

1.3 Alignment: Societal and Professional Demands

2.0 Administration and Management of Resources

3.0 Faculty and Staff: Profiles; Scholarship; Achievements

4.1 Student: Student Enrollment, Retention, and Graduation;

4.2 Student: Satisfaction

4.3 Student: Accomplishments/Outcomes

5.0 Collaboration and Communication

6.0 Alumni Satisfaction

7.0 Curricula

8.0 Assessment Procedures and Tools

9.0 Others

1. Current State/Issue: Give a good description.

2. Source: Identify where the recommendation came from. For example, program

self study, external team report, etc.

3. Date Identified: Note when the source gave the recommendation.

4. Goal: Describe the ideal situation or goal after the necessary changes have been

completed to meet the recommendation.

5. Timeframe for Completion: Select the deadline for the completed goal by year,

and either month or quarter from the pull-down menu.

6. Status: Don’t forget to update the status for each goal before the end of October

deadline: Not Started (default), In Progress, Completed, and Cancelled.

7. Notes: This is the place to keep records for anything related to the

recommendation. If you cancel the recommendation, be sure to give a clear

explanation for doing so.

8. Save: Don’t forget to Save each time you add or edit the recommendation or

your work will be lost.

Section Two: After entering the basic information on Section One, complete separate

action items for each action necessary to accomplish the recommendation’s proposed

solution/goal.

• Target Goal Timeframe: This is automatically populated from the data entered

in Section One.

• Action: Describe what must be done. It may take more than one action to

accomplish a goal. Be sure to make a separate action item for each action so it can

be tracked properly.

• Completion Time Frame: Enter the time this particular action item should be

completed.

• Responsibility Level: Select the appropriate option: Program, Department,

School, or University

• Responsible Parties: Enter the name(s) of the person(s) who are assigned to

accomplish this action.

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

• Status: It is important to select the appropriate status level: Not started (default),

In progress, Completed, or Cancelled. If it has been cancelled, be sure to indicate

in the Notes area why it has been cancelled.

• Notes: Enter documenting notes on the action item or status.

• Save: Don’t forget to Save!

Important Edit Tools

• Pencil: Click on the pencil icon to edit an item.

• Folder: Click on the folder to upload files to attach to the recommendation’s

solution goal. This will be helpful to keep files accessible for whenever they are

needed including when writing interim reports or the next self study.

Important Note for All AMS Reports

• Tables: Please do not copy and paste tables into the report textboxes, because this

causes many complications in running AMS reports. Instead, whenever needed,

please upload documents with the appropriate tables into the reporting sections

they support. Click on the folder to the right of the desired section’s name to

upload documents.

Office of Educational Effectiveness

Contact OEE for help with assessment, program review, and institutional research:

• Phone: extension 15042

• Email: assessment@llu.edu

• Website: http://www.llu.edu/assessment

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

Appendices

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

Assessment Resources

Books

Allen, M. J. (2003). Assessing Academic Programs in Higher Education. Bolton, MA: Anker

Publishing Company, Inc.

Allen, M. J. (2007). Assessing General Education Programs. Bolton, MA: Anker Publishing

Company, Inc.

Angelo, T. A., and Cross, K. P. (1993). Classroom Assessment Techniques: A Handbook for

College Teachers (2

ed.). San Francisco, CA: John Wiley & Sons, Inc.

Banta, T. W., and Palomba, C. A. (2015). Assessment Essentials: Planning and Implementing,

and Improving Assessment in Higher Education (2

ed.). San Francisco, CA: Joosey-

Bass.

Banta, T. W. and Associates (2002). Building a Scholarship of Assessment. San Francisco,

CA: John Wiley & Sons, Inc.

Bresciani, M. J. (2018). Outcomes-Based Program Review: Closing Achievement Gaps In- and

Outside the Classroom with Alignment to Predictive Analytics and Perfoemance Metrics

ed.). Sterling, VA: Stylus Publishing, LLC.

Bresciani, M. J. (2006). Outcomes-Based Academic and Co-Curricular Program Review: A

Compilation of Institutional Good Practices. Sterling, VA: Stylus Publishing, LLC.

Kuh, G. D., Ikenberry, S. O., Jankowski, N. A., Cain, T. R., Ewell, P. J., Hutchings, P.,

Kinzie, J. (2015). Using Evidence of Student Learning to Improve Higher Education.

San Francisco, CA: Jossey-Bass.

Light, T. P., Chen, H. L., and Ittelson, J. C. (2012). Documenting Learning with ePortfolios: A

Guide for College Instructors. San Francisco, CA: John Wiley & Sons, Inc.

Maki, P. L. (2010). Assessing for Learning: Building a Sustainable Commitment across the

Institution. Sterling, VA: Stylus Publishing, LLC.

Maki, P. L., and Borkowski, N. A. (Eds.) (2006). The Assessment of Doctoral Education:

Emerging Criteria and New Models for Improving Outcomes. Sterling, VA: Stylus

Publishing, LLC.

Nilson, L. B. (2010). Teaching at Its Best: A Research-Based Resource for College Instructors

ed.). San Francisco, CA: Jossey-Bass, A Wiley Imprint.

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

Palloff, R. M., and Pratt, K. (2009). Assessing the Online Learner: Resources and Strategies for

Faculty. San Francisco, CA: John Wiley & Sons, Inc.

Stevens, D. D., and Levi, A. J. (2012). Introduction to Rubrics: An Assessment Tool to Save

Grading Time, Convey Effective Feedback, and Promote Student Learning (2

ed.).

Sterling, VA: Stylus Publishing, LLC.

Suskie, L. (2018). Assessing Student Learning: A Common Sense Guide (3

ed.). San

Francisco, CA: John Wiley & Sons, Inc.

Walvoord, B. E. (2010). Assessment Clear and Simple: A Practical Guide for Institutions,

Departments, and General Education (2

ed.). San Francisco, CA: Jossey-Bass.

Walvoord, B. E., and Anderson, V. J. (2010). Effective Grading: A Tool for Learning and

Assessment in College (2

ed.). San Francisco, CA: Jossey-Bass.

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

Web

LLU Assessment and Program Review - Office of Educational Effectiveness

http://www.llu.edu/assessment

National Learning Institute for the Assessment of Learning Outcomes -

http://www.learningoutcomeassessment.org/

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

Professional Institutional Learning Outcomes Resources

Written Communication

Brown, C. A., Dickson, R., Humphreys, A.-L., McQuillan, V., & Smears, E. (2008).

Promoting Academic Writing/Referencing Skills: Outcome of an Undergraduate E-

Learning Pilot Project. British Journal of Educational Technology, 39(1), 140-156.

http://0search.ebscohost.com.catalog.llu.edu/login.aspx?direct=true&db=eric&AN=EJ782

722&site=ehost-live&scope=site

Fields, T. T., & Hatala, J. J. (2014). That, That, but Not That... Using a Cafeteria Plan to

Enhance Writing Skills. Administrative Issues Journal: Education, Practice, and Research,

4(2), 3-11. http://files.eric.ed.gov/fulltext/EJ1058504.pdf

Fields, T. T., Hatala, J. J., & Nauert, R. F. (2014). Perceptions of Preceptors and Students

on the Importance of Writing. Administrative Issues Journal: Education, Practice, and

Research, 4(1). http://files.eric.ed.gov/fulltext/EJ1058482.pdf

Fillyaw, M. J. (2011). Case Report Writing in a Doctor of Physical Therapy Education

Program: A Case Study. Journal of the Scholarship of Teaching and Learning, 11(1), 139-154.

http://files.eric.ed.gov/fulltext/EJ915929.pdf

Grillo, E. U., Koenig, M. A., Gunter, C. D., & Kim, S. (2015). Teaching CSD Graduate

Students to Think Critically, Apply Evidence, and Write Professionally. Communication

Disorders Quarterly, 36(4), 241-251.

http://0cdq.sagepub.com.catalog.llu.edu/content/36/4/241.full.pdf

Oral Communication

Costello, E., Corcoran, M., Barnett, J. S., Birkmeier, M., Cohn, R., Ekmekci, O., . . .

Walker, B. (2014). Information and Communication Technology to Facilitate Learning

for Students in the Health Professions: Current Uses, Gaps, and Future Directions.

Online Learning, 18(4).

http://files.eric.ed.gov/fulltext/EJ1048370.pdf

Evans, D. J. R. (2013). Connecting with Different Audiences: The Anatomy of

Communication is Essential. Anatomical Sciences Education, 6(2), 134-137.

http://onlinelibrary.wiley.com/doi/10.1002/ase.1311/abstract

Horwitz, L. I., Moin, T., & Green, M. L. (2007). Development and implementation of an

oral sign-out skills curriculum. Journal of general internal medicine, 22(10), 1470-1474.

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2305855/pdf/11606_2007_Article_331.pdf

Junod Perron, N., Nendaz, M., Louis-Simonet, M., Sommer, J., Gut, A., Baroffio, A., . . .

van der Vleuten, C. (2013). Effectiveness of a Training Program in Supervisors' Ability to

Provide Feedback on Residents' Communication Skills. Advances in Health Sciences

Education, 18(5), 901-915.

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

http://0search.ebscohost.com.catalog.llu.edu/login.aspx?direct=true&db=eric&AN=EJ103

6064&site=ehost-live&scope=site

Managheb, S. E., Zamani, A., Shams, B., & Farajzadegan, Z. (2012). The Effect of

Communication Skills Training by Video Feedback Method on Clinical Skills of Interns

of Isfahan University of Medical Sciences Compared to Didactic Methods. Health

Education Journal, 71(5), 546-552

http://0-hej.sagepub.com.catalog.llu.edu/content/71/5/546.full.pdf

Walton, K. L. W., & Baker, J. C. (2009). Group Projects as a Method of Promoting Student

Scientific Communication and Collaboration in a Public Health Microbiology Course.

Bioscene: Journal of College Biology Teaching, 35(2), 16-22.

http://files.eric.ed.gov/fulltext/EJ889701.pdf

Yang, S.-H., Shih, C.-K., Liu, C.-H., Peng, H.-T., & Chan, W. P. (2014). Service Learning

for Medical Students: Program Development and Students' Reflections. Turkish Online

Journal of Educational Technology - TOJET, 13(1), 193-198.

http://files.eric.ed.gov/fulltext/EJ1018159.pdf

Quantitative Reasoning

Manrai, A. K., Bhatia, G., Strymish, J., Kohane, I. S., & Jain, S. H. (2014). Medicine’s

uncomfortable relationship with math: calculating positive predictive value. JAMA

internal medicine, 174(6), 991-993.

http://archinte.jamanetwork.com/data/Journals/INTEMED/930309/ild140014.pdf

McNaught, K., & Hoyne, G. (2013). Testing Program Reveals Deficient Mathematics for

Health Science Students Commencing University. Issues in Educational Research, 23(2),

180-195. http://www.iier.org.au/iier23/mcnaught.pdf

Rao, G. (2008). Physician numeracy: Essential skills for practicing evidence-based

medicine. Family Medicine, 40(5), 354.

http://www.stfm.org/fmhub/fm2008/May/Goutham354.pdf?pagewanted=all

Whiting, P. F., Davenport, C., Jameson, C., Burke, M., Sterne, J. A. C., Hyde, C., & Ben-

Shlomo, Y. (2015). How well do health professionals interpret diagnostic information? A

systematic review. Bmj Open, 5(7). doi: 10.1136/bmjopen-2015-008155

http://bmjopen.bmj.com/content/5/7/e008155.full.pdf

Information Literacy

Coleman, C. A., & Fromer, A. (2015). A health literacy training intervention for

physicians and other health professionals. Fam Med, 47(5), 388-392.

http://www.stfm.org/FamilyMedicine/Vol47Issue5/Coleman388

Judd, T., & Kennedy, G. (2011). Expediency-Based Practice? Medical Students' Reliance

on Google and Wikipedia for Biomedical Inquiries. British Journal of Educational

Technology, 42(2), 351-360.

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

http://0search.ebscohost.com.catalog.llu.edu/login.aspx?direct=true&db=eric&AN=EJ927

821&site=ehost-live&scope=site

Kaplowitz, J. R., & Yamamoto, D. O. (2001). Web-based Library Instruction for a

Changing Medical School Curriculum. Library Trends, 50(1), 47-57.

http://0search.ebscohost.com.catalog.llu.edu/login.aspx?direct=true&db=eric&AN=EJ639

624&site=ehost-live&scope=site

Robertson, D. S., & Felicilda-Reynaldo, R. F. (2015). Evaluation of graduate nursing

students' information literacy self-efficacy and applied skills. J Nurs Educ, 54(3 Suppl),

S26-30. doi: 10.3928/01484834-20150218-03

http://www.healio.com/nursing/journals/jne/2015-3-54-3-supplemental/{ee17f1d7-3419-

4aa0-8739-8a1cd7ccb9d3}/evaluation-of-graduate-nursing-students-information-literacy-

self-efficacy-and-applied-skills.pdf

Stombaugh, A., Sperstad, R., Vanwormer, A., Jennings, E., Kishel, H., & Vogh, B. (2013).

Using lesson study to integrate information literacy throughout the curriculum. Nurse

Educ, 38(4), 173-177. doi: 10.1097/NNE.0b013e318296db56

http://ovidsp.tx.ovid.com/ovftpdfs/FPDDNCIBIFKPLH00/fs046/ovft/live/gv025/0000622

3/00006223-201307000-00017.pdf

Trujillo, J. M., & Figler, T. A. (2015). Teaching and learning health literacy in a doctor of

pharmacy program. Am J Pharm Educ, 79(2), 27. doi: 10.5688/ajpe79227

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4386748/pdf/ajpe79227.pdf

Critical Thinking

Ferris, H., & O'Flynn, D. (2015). Assessment in Medical Education; What Are We Trying

to Achieve? International Journal of Higher Education, 4(2), 139-144.

http://0search.ebscohost.com.catalog.llu.edu/login.aspx?direct=true&db=eric&AN=EJ106

0624&site=ehost-live&scope=site

Grillo, E. U., Koenig, M. A., Gunter, C. D., & Kim, S. (2015). Teaching CSD Graduate

Students to Think Critically, Apply Evidence, and Write Professionally. Communication

Disorders Quarterly, 36(4), 241-251.

http://0-cdq.sagepub.com.catalog.llu.edu/content/36/4/241.full.pdf

McDonald, P. L., Straker, H. O., Schlumpf, K. S., & Plack, M. M. (2014). Learning

Partnership: Students and Faculty Learning Together to Facilitate Reflection and Higher

Order Thinking in a Blended Course. Online Learning, 18(4).

http://olj.onlinelearningconsortium.org/index.php/olj/article/download/516/121

Rowles, J., Morgan, C., Burns, S., & Merchant, C. (2013). Faculty Perceptions of Critical

Thinking at a Health Sciences University. Journal of the Scholarship of Teaching and

Learning, 13(4), 21-35. http://files.eric.ed.gov/fulltext/EJ1017052.pdf

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

Schell, R., & Kaufman, D. (2009). Critical Thinking in a Collaborative Online PBL

Tutorial. Journal of Educational Computing Research, 41(2), 155-170.

http://0-jec.sagepub.com.catalog.llu.edu/content/41/2/155.full.pdf

Walter, K. O., Baller, S. L., & Kuntz, A. M. (2012). Two Approaches for Using Web

Sharing and Photography Assignments to Increase Critical Thinking in the Health

Sciences. International Journal of Teaching and Learning in Higher Education, 24(3), 383-394.

http://files.eric.ed.gov/fulltext/EJ1000690.pdf

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

LLU Assessment Glossary

Assessment Management System (AMS)

: An online repository and reporting tool that

includes the three annual reports (see “Annual Reports”) on p. 17.

Action Plan Report: Formerly called the Annual Program Report. At the end of a

program review or professional accreditation cycle the program should develop an

action plan showing how it plans to address each of the resulting recommendations

before the next cycle’s visit. This updated report is required annually by the end of

October each year. Update the report in the AMS.

ALO

: Accreditation Liaison Officer to interface between the WASC Senior College and

University Commission (WSCUC) and the college or University.

Annual Program Report: See Action Plan Report

Annual reports: LLU has three annual reports: Faculty Portfolio (see p. 17), the

Institutional Learning Outcome Assessment Report (see p. 20), and Annual Action Plan

Report (see p. 22-23). All are due the end of October of each year.

Assessment: Processes that identify, collect, use, and prepare data that can be used to

evaluate student achievement.

Assessment matrix: A tool to organize and track how the LLU Institutional Learning

Outcomes (ILOs) and Program Learning Outcomes (PLOs) are assessed. (see p. 14)

Baseline assessment: Assessment conducted at the beginning of the program to

determine students’ entry levels on ILOs and PLOs. This assessment shows the level of

skills or learning of students when they enter the program. It is not required. (see p. 12)

Competency: Level at which performance is acceptable.

Curriculum map: A mechanism to organize the program’s curriculum in a logical and

reasonable manner to support their learning outcomes and shows where the outcomes

are taught and assessed. (see p. 12)

Direct measure

: The assessment is based on an analysis of student behaviors or

products in which they demonstrate how well they have mastered learning outcomes.

(see p. 10)

AMS: http://myllu.llu.edu/assessment/programs/

LLU ALO - Marilyn Eggers, PhD, Associate Provost: Educational Effectiveness and Extended Education

Allen, M. J. (2008). “Strategies for Direct and Indirect Assessment of Student Learning.” Retrieved on

November 29, 2017 from:

http://academics.lmu.edu/spee/officeofassessment/assessmentresources/selectinganassessmentmeasur

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

Evaluation: Process of reviewing the results of data collection and analysis and making

a determination of the value of findings and action to be taken.

Exit and other interviews: Face-to-face interviews asking graduate students to share

their perceptions about the target of study—e.g., their own skills/attitudes, skills and

attitudes of others, or program qualities. This can be done in online programs with

Zoom

Focus groups: Guided discussion of a group of people who share certain characteristics

related to the research or evaluation question, conducted by a trained moderator.

Formative assessment: (Updated) Assessment at the middle of the program. Required;

tracked by the program and the University. Mid-program assessment gives the program

the opportunity to make any necessary changes for the current students who were

assessed, if they did not meet the criteria for success. This is the only opportunity for the

program to fill this gap, if one is found. The summary formative report should be

included briefly in the annualFinstitutional ILO report along with the summative

assessment report. (see pp. 12 and 16)

Indirect measure

: The assessment is based on an analysis of reported perceptions

about student mastery of learning outcomes

Institutional learning outcomes (ILOs): The institution’s learning outcomes that all

students at all levels should be successful in by the end of the program. (see pp. 8)

Institutional research

: Provides the university community with information to support

decision-making and educational effectiveness efforts and fulfills requests for

institutional data from local, state and federal agencies.

Learning outcome: See Student Learning Outcomes (SLOs)

Mission Focused Learning Outcomes (MFLOs): Loma Linda University’s two Mission

Focused Learning Outcomes (MFLOs) are firmly rooted in its mission, vision, and

values. Because Mission Focused Learning is LLU’s culture, the University is developing

specialized assessment processes to ensure integration of these outcomes over time.

• Wholeness

: Students apply the University philosophy of wholeness into their

personal and professional lives.

Zoom: https://www.zoom.us

Allen, M. J. (2008). “Strategies for Direct and Indirect Assessment of Student Learning.”

Retrieved on November 29, 2017 from:

http://academics.lmu.edu/spee/officeofassessment/assessmentresources/selectinganassessmentme

asure/

LLU Institutional Researcher: W. Ken Nelson, MD, Office of Educational Effectiveness

Associate Director

Wholeness: Loved by God, growing in health, living with purpose in community

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

• Values

: Students integrate LLU’s Christ-centered values in their personal and

professional lives.

Objectives: Broad statements that describe the career and professional accomplishments

that the program is preparing graduates to achieve.

OEE: See Office of Educational Effectiveness

Office of Educational Effectiveness (OEE)

: This office promotes educational

effectiveness by coordinating and facilitating assessment, program review, University

accreditation, institutional research, and other areas in addition to special projects

including EXSEED

Operational (—ize): Defining a term or object so that it can be measured. Generally

states the operations or procedures used that distinguish it from others.

Outcomes: Statements that describe what students are expected to know and are able to

do by the time of graduation. See p. 8 for outcomes.

Performance criteria/indicators: Specific, measurable statements identifying the

performance(s) required to meet and assess the outcome. They are confirmable through

assessment evidence. (See p. 10)

Portfolios: Collections of work samples and reflections usually compiled over time and

rated using rubrics.

Professional ILOs: Professional and skills-intensive programs may adapt LLU’s ILO’s

and rubrics to meet their unique learning and assessment needs. Professional ILO

rubrics are being developed by the Learning Outcomes Committee and are posted at the

OEE website. (see p. 6)

Program review

: LLU has a formal program review process and guide to assist

programs in their cycle of review.

Rubrics: A rubric is a set of categories or elements that define and describe the

important components of the work being completed, critiqued, or assessed. Each

category or element contains a gradation of levels of completion or competence with a

score assigned to each level and a clear description of what criteria needs to be met to

attain the score at each level.

LLU Values: Compassion, Excellence, Freedom, Integrity, Humility, Justice, Purity/Self-control

http://www.llu.edu/central/values.page

OEE: http://www.llu.edu/assessment

EXSEED: http://www.llu.edu/exseed

Program Review: http://www.llu.edu/central/assessment/programreview.page

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

School Assessment Specialist

: Each school assigns at least one Assessment Specialist

to coordinate and support school assessment activities. These individuals have

evaluation and measurement experience and receive additional training and support

from the OEE. They are also members of the University Assessment Committee

Stakeholder: Anyone who has a vested interest in the outcome of the program/project.

Student Learning Outcomes (SLOs): Knowledge, skill, attitudes, values, etc., that

students should be able to demonstrate by the end of the program. This is a large

category term for three types of SLOs: (1) institutional (ILOs), (2) program (PLOs), and

(3) course (CLOs). (see pp. 4 and 8)

Summative assessment: Assessment at the end of the program. Required; tracked by

the University. Shows the students’ final level of success for the indicated learning

outcome. If students did not meet the criteria for success, the program needs to

determine what changes need to be made to improve student learning. This is the final

closing of the loop to make improvements for future students. (see p. 13)

Triangulate: The use of a combination of assessment methods in a study. An example of

triangulation would be an assessment that incorporated student work, surveys, and

observations.

Written surveys: Asking individuals to share their perceptions about the study target—

e.g., their own or others' skills/attitudes/behavior, or program/course qualities and

attributes.

Some of these definitions were presented by Gloria Rogers in her Faculty Workshop on Assessing Student

Learning, August 6 & 7, 2007, at LLU.

See School Assessment Specialist: http://www.llu.edu/central/assessment/assessment.page

University Assessment Committee: http://home.llu.edu/academics/academic-

resources/educational-effectiveness/committees/university-assessment-committee

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

University Assessment Committee

•

Loma Linda University

School Assessment Specialists Position Description

Each School will assign at least one Assessment Specialist with evaluation and

measurement experience to coordinate school assessment activities. The Office of

Educational Effectiveness (OEE) provides ongoing assessment support and training to

assure meaningful, coordinated assessment.

All school Assessment Specialists serve on the University Assessment Committee

and

also function as liaisons between this committee and their Schools.

The Assessment Specialist will:

1. Provide assessment support for the University, schools, and programs

o Assist programs to develop their assessment plans and processes

o Assist course directors with course-level assessments

▪ Support Watermark software program where applicable

2. Be familiar with and promote resources from the OEE

o Assessment Guide

o Program Review Guide

o Distance Education Guide

o Office of Educational Effectiveness (OEE) website

o Newsletter (view on OEE website)

o Power BI

3. Communicate clearly and regularly with schools on LLU’s assessment

announcements, expectations and deadlines

4. Provide feedback to schools and programs to assist them in closing the

assessment loop and ensure the dissemination of the results

5. Participate in school’s Program Review efforts

6. Provide assessment mentoring both in their own and other schools

UAC -

https://myllu.llu.edu/syncall/itemdetail/?communityId=3295&itemType=story&itemId=6328

Watermark (LiveText & via) - https://home.llu.edu/education/office-of-provost/departments-

and-divisions/livetext-by-watermark-and-via-by-watermark

LLU Assessment Guide - https://home.llu.edu/education/office-of-provost/departments-and-

divisions/educational-effectiveness/assessment

LLU Program Review Guide - https://home.llu.edu/education/office-of-provost/departments-

and-divisions/program-review

LLU Distance Education Guide - https://home.llu.edu/education/office-of-provost/departments-

and-divisions/online-programs/llu-distance-education

OEE - https://home.llu.edu/education/office-of-provost/departments-and-

divisions/educational-effectiveness

LLU Power BI - https://powerbi.microsoft.com/en-

us/landing/signin/?ru=https%3A%2F%2Fapp.powerbi.com%2F%3Froute%3Dgroups%252fme%2

52fdashboards%252f0ff493b9-6ab9-4d65-afc5-e65d58191f00%26noSignUpCheck%3D1

LLU Power BI – If you don’t have access, contact assessment@llu.edu

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

7. Participate in the development of assessment reports and ensure the

dissemination of these reports

8. Support and encourage the completion of the annual program reports and

ensure dissemination of the results

o AMS

▪ ILO Analysis

▪ Action Plans

o Annual Faculty Report

The Assessment Specialist may:

1. Serve on other OEE committees as needed and as assigned by schools to

formulate the assessment policies for LLU and to ensure the dissemination of

these decisions to their respective schools.

2. Participate on assessment projects that benefit the University.

3. Participate in the development and offering of assessment workshops.

4. Coordinate their school’s professional accreditation, if appropriate, as well as be

supportive in LLU’s WSCUC accreditation preparation. Ensure that assignments

for WSCUC preparations are completed at both the school and program level.

Academic Management System (AMS) -

https://myllu.llu.edu/assessment/programs/?tab=dashboard

Annual Faculty Report - https://myllu.llu.edu/profile/portfolio/

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

[Program Name] Curriculum Map

[School Name]

[Academic Year]

Loma Linda University

ILOs

1. CritThink

2. InfoLit

3. OralCom

4. QuantR

5. WrittenCom

Courses

PLOs

Courses

ILOs: LLU’s Institutional Learning Outcomes – see ILO Legend at end of document

PLOs: Program Learning Outcomes

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

PLOs

Instruction:

I = Introduced

E = Expanded

A = Advanced

Assessment

B = Baseline – Assessment at beginning of program. Recommended; tracked by the program.

F = Formative – Assessment at the middle of the program. Required; tracked by the program.

S = Summative – Assessment at the end of the program. Required; tracked by the University.

Institutional Learning Outcomes (ILOs):

Abbreviation

ILOs

1. CritThink

Critical Thinking

2. InfoLit

Information Literacy

3. OralCom

Oral Communication

4. QuantR

Quantitative Reasoning

5. WrittenCom

Written Communication

Results tracked over time

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

Assessment Matrix

[School name: Program name]

[Academic year]

Loma Linda University

Where are outcomes published? Mark all that apply.

Catalog

Program Website

Course Syllabi

Program Documents

Other (list)

LLU Institutional Learning Outcomes

Program Learning Outcomes

LLU Institutional Learning Outcomes

LLU Institutional

Learning

Outcomes (ILOs)

Performance

Indicators

Assessment

Measurement Tools

& Data Collection

Cycles

Criteria for Success

Who interprets the

assessment data?

What is the process?

Findings from

Assessment Data

Collection

Resulting Program

Changes

1. Critical

Thinking

2. Information

Literacy

3. Oral

Communication

4. Quantitative

Reasoning

5. Written

Communication

LLU template based on Point Loma Nazarene University’s Assessment Plan Matrix

Develop ILO Performance Indicators (1-3) to fit the program.

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

Program Learning Outcomes

Program

Learning

Outcomes (PLOs)

Performance

Indicators

Assessment

Measurement Tools

& Data Collection

Cycles

Criteria for Success

Who interprets the

assessment data?

What is the process?

Findings from

Assessment Data

Collection

Resulting Program

Changes

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

LLU INSTITUTIONAL LEARNING OUTCOME: INFORMATION LITERACY RUBRIC

DEVELOPED FOR ACADEMIC AND PROFESSIONAL USE

Based on the AAC&U Written Communication VALUE Rubric, value@aacu.org, assessment@llu.edu, or see sites below1.

The VALUE rubrics were developed by teams of faculty experts representing colleges and universities across the United States through a process

that examined many existing campus rubrics and related documents for each learning outcome and incorporated additional feedback from faculty. The

rubrics articulate fundamental criteria for each learning outcome, with performance descriptors demonstrating progressively more sophisticated levels of

attainment. The rubrics are intended for institutional-level use in evaluating and discussing student learning, not for grading. The core expectations

articulated in all 15 of the VALUE rubrics can and should be translated into the language of individual campuses, disciplines, and even courses. The utility

of the VALUE rubrics is to position learning at all levels within a basic framework of expectations such that evidence of learning can by shared nationally

through a common dialog and understanding of student success.

Definition

The ability to know when there is a need for information, to be able to identify, locate, evaluate, and effectively and responsibly use and share that

information for the problem at hand. - National Forum on Information Literacy

Framing Language

This rubric is recommended for use evaluating a collection of work, rather than a single work sample in order to fully gauge students’ information

literacy skills. Some of the areas that could be evaluated could include: research papers, editorials, speeches, grant proposals, marketing or business plans,

PowerPoint and other presentations, posters, literature reviews, position papers, and argument critiques—technical procedures, charting, clinical projects to

name a few. In addition, a description of the assignments with the instructions that initiated the student work would be vital in providing the complete

context for the work. Although a student’s final work must stand on its own, evidence of a student’s research and information gathering processes, such as

a research journal/diary, could provide further demonstration of a student’s information proficiency and for some criteria on this rubric could be required.

Glossary

Access the Needed Information – Use Boolean search logic (as found at https://libguides.mit.edu/c.php?g=175963&p=1158594 ) within search engines for

evidence-based, discipline-specific data-bases and other professional sources.

Evaluate Information and Its Sources Critically – Seek, recognize and use legitimate evidence-based sources.

Ethical and Legal Information – Students correctly cite and reference information in their writing to avoid plagiarizing. This includes restrictions on the access and

use of published, confidential, and/or proprietary information i.e. copyright and trademarks. Students carefully paraphrase, summarize and quote in ways that are true to

the original context.

This rubric is recommended for use in evaluating information literacy skills in both academic and professional settings.

1 AAC&U - http://www.aacu.org/value/metarubrics.cfm; LLU Office of Educational Effectiveness - http://www.llu.edu/central/assessment/index.page

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

LLU INSTITUTIONAL LEARNING OUTCOME: INFORMATION LITERACY RUBRIC

DEVELOPED FOR ACADEMIC AND PROFESSIONAL USE

Based on the AAC&U Written Communication VALUE Rubric, value@aacu.org, assessment@llu.edu, or see sites below2

Definition

The ability to know when there is a need for information, to be able to identify, locate, evaluate, and effectively and responsibly use and share that information for the problem at hand. - The

National Forum on Information Literacy

Evaluators are encouraged to assign a zero to any work sample or collection of work that does not meet benchmark (cell one) level performance.

Determine the Extent and Types of

Information Needed

Effectively determines key concepts

and defines the scope of the problem or

research question. Types of information

sources selected directly relate to

concepts or problem.

Determines key concepts and defines

the scope of the problem or research

question.. Types of information sources

selected relate to concepts or respond

to the problem.

Determines some key concepts and

partially defines the scope of the

problem or research question. (parts are

missing, remains too broad or too

narrow, etc.). Types of information

sources selected partially relate to

concepts or respond to the problem.

Has difficulty determining key concepts

and defining the the scope of the

problem or research question. Types of

information sources selected minimally

relate to concepts or respond to the

problem.

Access the Needed Information

Accesses information using effective,

well-designed search strategies and the

ability to refine the search while using

the most appropriate information

sources.

Accesses information using a variety of

search strategies with the ability to

refine the search while choosing

relevant information sources.

Accesses information using simple

search strategies, retrieves information

from limited and similar sources.

Accesses information randomly,

retrieves information that lacks

relevance and quality.

Evaluate Information and Its

Sources Critically

Effectively analyzes and applies

evidence-based information sources

directly related to the scope and

discipline of the problem or research

question, such as: relevance to the

research question, currency, authority,

audience, and bias or point of view.

Analyzes and applies evidence-based

information sources using multiple

criteria appropriate to the scope and

discipline of the problem or research

question, such as: relevance to the

research question, currency, authority,

audience, and bias or point of view.

Chooses a variety of information

sources using basic criteria, such as:

currency and relevance to the problem

or research question.

Chooses a few information sources.

using limited criteria, such as: relevance

to the problem or research question.

Use Information Effectively to

Accomplish a Specific Purpose

Communicates, organizes and

synthesizes information from sources to

fully achieve the specific purpose, with

clarity and depth.

Communicates, organizes and uses

information from sources. The

intended purpose is achieved.

Communicates and organizes

information from sources. The intended

purpose is not fully achieved.

Communicates information from

sources. The information is fragmented

and/or used inappropriately, such as:

misquoted, taken out of context, or

incorrectly paraphrased. The intended

purpose is not achieved.

Access and Use Information

Ethically and Legally

Demonstrates thorough knowledge and

application of the ethical and legal

restrictions on the access and use of

published, confidential, and/or

proprietary information. Uses the

designated format correctly including

appropriate citations.

Demonstrates knowledge and

application of ethical and legal

restrictions in the access and use of

published, confidential, and/or

proprietary information. Uses the

designated format correctly including

appropriate citations.

Demonstrates partial knowledge and

applicaton of the ethical and legal

restrictions in the access and use of

published, confidential, and/or

proprietary information.

Demonstrates limited knowledge and

application of the ethical and legal

restrictions in the access and use of

published, confidential, and/or

proprietary information.

2 AAC&U - http://www.aacu.org/value/metarubrics.cfm; LLU Office of Educational Effectiveness - http://www.llu.edu/central/assessment/index.page

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

LLU INSTITUTIONAL LEARNING OUTCOME: QUANTITATIVE REASONING RUBRIC

DEVELOPED FOR ACADEMIC AND PROFESSIONAL USE

Based on the AAC&U Quantitative Literacy VALUE Rubric, value@aacu.org, assessment@llu.edu, or see sites below

The VALUE rubrics were developed by teams of faculty experts representing colleges and universities across the United States through a process that examined

many existing campus rubrics and related documents for each learning outcome and incorporated additional feedback from faculty. The rubrics articulate fundamental

criteria for each learning outcome, with performance descriptors demonstrating progressively more sophisticated levels of attainment. The rubrics are intended for

institutional-level use in evaluating and discussing student learning, not for grading. The core expectations articulated in all 15 of the VALUE rubrics can and should be

translated into the language of individual campuses, disciplines, and even courses. The utility of the VALUE rubrics is to position learning at all levels within a basic

framework of expectations such that evidence of learning can by shared nationally through a common dialog and understanding of student success.

Definition

Quantitative Literacy (QL) – also known as Numeracy or Quantitative Reasoning (QR) – is a "habit of mind," competency, and comfort in working with

numerical data. Individuals with strong QL skills possess the ability to reason and solve quantitative problems from a wide array of authentic contexts and everyday life

situations. They understand and can create sophisticated arguments supported by quantitative evidence and they can clearly communicate those arguments in a variety of

formats (using words, tables, graphs, mathematical equations, etc., as appropriate).

Quantitative Literacy Across the Disciplines

Current trends in general education reform demonstrate that faculty are recognizing the steadily growing importance of Quantitative Literacy (QL) in an

increasingly quantitative and data-dense world. AAC&U’s recent survey showed that concerns about QL skills are shared by employers, who recognize that many of

today’s students will need a wide range of high level quantitative skills to complete their work responsibilities. Virtually all of today’s students, regardless of career choice,

will need basic QL skills such as the ability to draw information from charts, graphs, and geometric figures, and the ability to accurately complete straightforward

estimations and calculations.

Preliminary efforts to find student work products which demonstrate QL skills proved a challenge in this rubric creation process. It’s possible to find pages of

mathematical problems, but what those problem sets don’t demonstrate is whether the student was able to think about and understand the meaning of her work. It’s

possible to find research papers that include quantitative information, but those papers often don’t provide evidence that allows the evaluator to see how much of the

thinking was done by the original source (often carefully cited in the paper) and how much was done by the student herself, or whether conclusions drawn from analysis

of the source material are even accurate.

Given widespread agreement about the importance of QL, it becomes incumbent on faculty to develop new kinds of assignments which give students

substantive, contextualized experience in using such skills as analyzing quantitative information, representing quantitative information in appropriate forms, completing

calculations to answer meaningful questions, making judgments based on quantitative data and communicating the results of that work for various purposes and

audiences. As students gain experience with those skills, faculty must develop assignments that require students to create work products which reveal their thought

processes and demonstrate the range of their QL skills.

This rubric provides for faculty a definition for QL and a rubric describing four levels of QL achievement which might be observed in work products within

work samples or collections of work. Members of AAC&U’s rubric development team for QL hope that these materials will aid in the assessment of QL – but, equally

important, we hope that they will help institutions and individuals in the effort to more thoroughly embed QL across the curriculum of colleges and universities.

AAC&U - http://www.aacu.org/value/metarubrics.cfm; LLU Office of Educational Effectiveness - http://www.llu.edu/central/assessment/index.page

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

Framing Language

This rubric has been designed for the evaluation of work that addresses quantitative literacy (QL) in a substantive way. QL is not just computation, not just the

citing of someone else’s data. QL is a habit of mind, a way of thinking about the world that relies on data and on the mathematical analysis of data to make connections

and draw conclusions. Teaching QL requires us to design assignments that address authentic, data-based problems. Such assignments may call for the traditional written

paper, but we can imagine other alternatives: a video of a PowerPoint presentation, perhaps, or a well designed series of web pages. In any case, a successful

demonstration of QL will place the mathematical work in the context of a full and robust discussion of the underlying issues addressed by the assignment.

Finally, QL skills can be applied to a wide array of problems of varying difficulty, confounding the use of this rubric. For example, the same student might

demonstrate high levels of QL achievement when working on a simplistic problem and low levels of QL achievement when working on a very complex problem. Thus,

to accurately assess a students QL achievement it may be necessary to measure QL achievement within the context of problem complexity, much as is done in diving

competitions where two scores are given, one for the difficulty of the dive, and the other for the skill in accomplishing the dive. In this context, that would mean giving

one score for the complexity of the problem and another score for the QL achievement in solving the problem.

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

LLU INSTITUTIONAL LEARNING OUTCOME: QUANTITATIVE REASONING RUBRIC

DEVELOPED FOR ACADEMIC AND PROFESSIONAL USE

Based on the AAC&U Quantitative Literacy VALUE Rubric, value@aacu.org, assessment@llu.edu, or see sites below

Definition

Quantitative Literacy (QL) – also known as Numeracy or Quantitative Reasoning (QR) – is a "habit of mind," competency, and comfort in working with numerical data. Individuals with strong

QL skills possess the ability to reason and solve quantitative problems from a wide array of authentic contexts and everyday life situations. They understand and can create sophisticated arguments

supported by quantitative evidence and they can clearly communicate those arguments in a variety of formats (using words, tables, graphs, mathematical equations, etc., as appropriate).

Evaluators are encouraged to assign a zero to any work sample or collection of work that does not meet benchmark (cell one) level performance.

Interpretation

Ability to explain information

presented in mathematical forms (e.g.,

equations, graphs, diagrams, tables,

words)

Provides accurate explanations of

information presented in mathematical

forms. Makes appropriate inferences based

on that information. An example would be to

accurately explain the trend data shown in a graph

and make reasonable predictions regarding what

the data suggest about future events.

Provides accurate explanations of

information presented in mathematical

forms. An example would be to accurately

explain the trend data shown in a graph.

Provides somewhat accurate

explanations of information presented

in mathematical forms, but

occasionally makes minor errors

related to computations or units. An

example would be to accurately explain trend

data shown in a graph, but may miscalculate

the slope of the trend line.

Attempts to explain information

presented in mathematical forms, but

draws incorrect conclusions about what

the information means. An example would

be to attempt to explain the trend data shown in

a graph, but will frequently misinterpret the

nature of that trend, perhaps by confusing

positive and negative trends.

Representation

Ability to convert relevant information

into various mathematical forms (e.g.,

equations, graphs, diagrams, tables,

words)

Skillfully converts relevant information into

an insightful mathematical portrayal in a

way that contributes to a further or deeper

understanding.

Competently converts relevant

information into an appropriate and

desired mathematical portrayal.

Completes conversion of information

but resulting mathematical portrayal is

only partially appropriate or accurate.

Completes conversion of information

but resulting mathematical portrayal is

inappropriate or inaccurate.

Calculation

Calculations attempted are essentially all

successful and sufficiently comprehensive

to solve the problem. Calculations are

presented elegantly, clearly, concisely.

Calculations attempted are essentially

all successful and sufficiently

comprehensive to solve the problem.

Calculations attempted are either

unsuccessful or

represent only a portion of the

calculations required to

comprehensively solve the problem.

Calculations are attempted but are both

unsuccessful and are not comprehensive.

Application / Analysis

Ability to make judgments and draw

appropriate conclusions based on the

quantitative analysis of data, while

recognizing the limits of this analysis

Uses the quantitative analysis of data as the

basis for deep and thoughtful judgments,

drawing insightful, carefully qualified

conclusions from this work.

Uses the quantitative analysis of data

as the basis for competent judgments,

drawing reasonable and appropriately

qualified conclusions from this work.

Uses the quantitative analysis of data

as the basis for judgments, lacking

inspiration or nuance leading to

marginal conclusions drawn from the

work.

Uses the quantitative analysis of data as

the basis for tentative judgments, and is

hesitant or uncertain about drawing

conclusions from this work.

Assumptions

Ability to make and evaluate

important assumptions in estimation,

modeling, and data analysis

Explicitly describes assumptions and

provides compelling rationale for why each

assumption is appropriate. Shows

awareness that confidence in final

conclusions is limited by the accuracy of

the assumptions.

Explicitly describes assumptions and

provides rationale for why assumptions

are appropriate. Shows awareness that

final conclusions are limited by the

accuracy of the assumptions.

Partially describes assumptions with

incomplete rationale..

Attempts to describe assumptions

without rationale..

Communication

Expressing quantitative evidence in

support of the argument or purpose of

the work (in terms of what evidence is

used and how it is formatted, presented,

and contextualized)

Uses quantitative information in

connection with the argument or purpose

of the work, presents it in an effective

format, and explains it with consistently

high quality.

Uses quantitative information in

connection with the argument or

purpose of the work. The data is

presented in an effective format.

Uses quantitative information, but

does not effectively connect it to the

argument or purpose of the work.

Presents an argument for which

quantitative evidence is pertinent, but

does not provide adequate support.

(May use quasi-quantitative words such

as "many," "few," "increasing," "small,"

and the like in place of actual quantities.)

AAC&U - http://www.aacu.org/value/metarubrics.cfm; LLU Office of Educational Effectiveness - http://www.llu.edu/central/assessment/index.page

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

LLU INSTITUTIONAL LEARNING OUTCOME: PROFESSIONAL CRITICAL THINKING RUBRIC

Based on the AAC&U Critical Thinking VALUE Rubric

The VALUE rubrics were developed by teams of faculty experts representing colleges and universities across the United States through a process that examined many existing campus

rubrics and related documents for each learning outcome and incorporated additional feedback from faculty. The rubrics articulate fundamental criteria for each learning outcome, with

performance descriptors demonstrating progressively more sophisticated levels of attainment. The rubrics are intended for institutional-level use in evaluating and discussing student

learning, not for grading. The core expectations articulated in all 15 of the VALUE rubrics can and should be translated into the language of individual campuses, disciplines, and even

courses. The utility of the VALUE rubrics is to position learning at all undergraduate levels within a basic framework of expectations such that evidence of learning can be shared nationally

through a common dialog and understanding of student success.

Loma Linda University (LLU) is a largely graduate and professional health sciences institution. Dr. Lynda Daniel-Underwood developed a clinical version of the AAC&U Critical Thinking

Rubric for the School of Medicine. The LLU Learning Outcome Committee took Dr. Daniel-Underwood’s version of the rubric she developed for the MD program and broadened it to be

applicable for practice in a variety of settings to include programs with clinicals, practica, and field experiences.

Definition

Critical thinking is higher level reasoning using professional judgment with appropriate and reliable sources to make evidence-based decisions.

Framing Language

This Professional Critical Thinking Rubric is designed to be transdisciplinary, reflecting the recognition that success in all disciplines requires inquiry and analysis that share common

attributes. Further, research suggests that successful critical thinkers from all disciplines increasingly need to be able to apply those competencies in various and changing situations

encountered in diverse environments. This rubric is designed for use with many different types of environments, and the suggestions here are not an exhaustive list of possibilities. Critical

thinking can be demonstrated in situations that require students to complete analyses of complex professional experiences and issues.

Notes on Uses of the Rubric

• Zero Score: Evaluators are encouraged to assign a zero to any work sample or resolution of a problem that does not meet level one performance.

• Contextualizing the Rubric: Programs may further modify this rubric to fit their unique disciplines’ needs.

Glossary

The definitions that follow were developed to clarify terms and concepts used in this rubric only.

• Sources: Reliable books, statements, people, etc., supplying appropriate information

• Problem: A practical situation within the discipline/profession that needs resolution.

• Professional ILO

: Professional programs and skills-intensive disciplines may adapt and assess LLU’s ILOs and rubrics to meet their unique learning and assessment needs.

• Reliable: Peer-acceptable, trusted standards and sources

AAC&U Critical Thinking Rubric was initially revised by Dr. Lynda Danial-Underwood, LLU School of Medicine, for the clinical setting and was further revised by LLU Learning

Outcomes Committee; AAC&U - http://www.aacu.org/value/metarubrics.cfm - value@aacu.org; LLU Office of Educational Effectiveness -

http://www.llu.edu/central/assessment/index.page - assessment@llu.edu.

Based on Dictionary.com:

http://www.dictionary.com/browse/source?s=t

LLU’s Professional Institutional Learning Outcomes:

Based on Dictionary.com: http://www.dictionary.com/browse/reliable?s=t

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS

LLU INSTITUTIONAL LEARNING OUTCOME: PROFESSIONAL CRITICAL THINKING RUBRIC

Based on the AAC&U Critical Thinking VALUE Rubric

Definition

Critical thinking is higher level reasoning using professional judgment with appropriate and reliable sources to make evidence-based decisions.

Evaluators are encouraged to assign a zero to any work sample or resolution of a problem that does not meet level one performance.

Identify the presented

problem(s)

Identifies problem(s) accurately,

independently, and with expertise.

Identifies problem(s) accurately and

independently.

Identifies problem(s) without accuracy or

broad focus but seeks input.

Inaccurately identifies the problem

nor seeks input when appropriate.

Gain new information

Gathers sufficient and appropriate

information from reliable sources.

Gathers appropriate information from

reliable sources.

Gathers insufficient information from

standard and unreliable sources.

Gathers unreliable information

from various sources.

Define key

components within

context of the

presented problem(s)

Defines, formulates and prioritizes

individualized multiple solutions to the

presented problem(s) based on prior

knowledge and the evidence.

Defines and formulates a solution to

the presented problem(s) based on

prior knowledge and the evidence.

Defines and formulates an ineffective

solution to the presented problem(s).

Defines and formulates an

inappropriate solution to the

presented problem(s).

Integrate knowledge

and expertise for

decision-making

Articulates the complexities of the

presented problem(s) and uses prior

knowledge and skills to fully assess

the complexities of the proposed

solution.

Articulates the complexities of the

presented problem(s) and uses prior

knowledge and skills to assess the

appropriateness of the proposed

solution.

Articulates some of the complexities of

the presented problem(s) to partially

assess the proposed solution.

Articulates a simplistic approach to

the presented

complex problem(s).

Solve problem safely

and implement

effectively

Effectively resolves the presented

problem(s) safely using approved

standard protocols and suggests new

solutions based on sound evidence.

Safely resolves the presented

problem(s) and effectively uses

approved standard protocols,

devising individualized solutions

when appropriate.

Ineffectively resolves the presented

problem(s) or incompletely uses standard

protocols.

Unsafely addresses the presented

problem(s).

AAC&U Critical Thinking Rubric revised by Dr. Lynda Danial-Underwood, LLU School of Medicine, and the LLU Learning Outcomes Committee; AAC&U -

http://www.aacu.org/value/metarubrics.cfm - value@aacu.org; LLU Office of Educational Effectiveness - http://www.llu.edu/central/assessment/index.page - assessment@llu.edu.

LLU Assessment Guide

January 21, 2020

BACK TO CONTENTS