Clinical Competency Assessment Redesign Project

Background 

From 2015-2018, I co-chaired the University of Michigan School of Dentistry’s Competency Assessment Team (CAT). The charge of this team was to redesign the method for assessing clinical competencies in the DDS program. Formerly, each discipline (such as endodontics, prosthodontics, orthodontics, etc.) set its own assessment guidelines, schedule, and grading scale. This led to two issues: 

  • Students reported being more stressed about navigating their requirements than about treating patients.
  • Students were treating patients as simple discipline-specific cases – “an orthodontics case” or “a root canal” or “an upper denture” – rather than as whole humans with complex needs. 

The first step to move the school closer to its vision of true “whole-patient” care as to bring together all the discipline-specific courses together into one Comprehensive Care course in the early 2000s. This created one massive 32 credit course, delivered over six semesters, called Comprehensive Care. During this course, students provide oral health care to patients in a school clinic under the supervision of faculty members in each of the eleven disciplines within the school.

By 2015, it was clear that the next step of integration was needed to reduce student stress and confusion around requirements. The next phase of the program, where I began my involvment, aimed to standardize assessment across disciplines. This had three steps:

  • Using a common grading scale across all disciplines
  • Using a common grading process across all disciplines
  • Posting all requirements, instructions, and grades in one electronic system

My role

As co-chair of the Competency Assessment Team, my purview was to map the entire clinical curriculum (no single document at the school existed to describe all the requirements), propose a model for an integrated electronic grading system, select and customize software, train faculty and students in its use, and oversee the rollout and initial revisions to the system. Simply mapping the existing curriculum and processes took nearly six months. Standardizing rubrics and developing a new basic workflow took another six months, and software selection and development took approximately two years. Nearly 20 systems were investigated, and in the end, a combination of existing technologies was used to grade the clinical competencies.

Outcomes

In the spring of 2018, we rolled out the new system to the DDS and DH programs. In the first semester, more than 5000 grades were entered into the system. The number of electronic systems students needed to consult was reduced from 8 to 2 seamlessly integrated systems, and eliminated the need for a paper record to check their grades. Faculty and students rapidly developed new ways of working with grade data, and in the first semester, additional tools were built to facilitate grading and reporting. 

 Sample materials I developed

Online MBA Program Development – 2019

Video floating an image of instructor Brian Flannagan over an interior scene in the University of Michigan Ross Business School as his main points scroll into the distance.
This video floatis an image of instructor Brian Flannagan over an interior scene in the University of Michigan Ross Business School as his main points scroll into the distance.

The University of Michigan’s Ross School of Business set out to create the most engaging and visually stunning online MBA program in the world – and succeeded.

This program mirrors the on-campus MBA experiential opportunities such as intensive residency weekends and the challenging Multidisciplinary Action Projects with real companies. In addition, faculty wanted to replicate their engaging class sessions online – and they didn’t want to look like a talking head on a webcam while they did it. So, this project engaged a truly astonishing crew of designers, developers, and filmographers to create instructional videos and synchronous class experiences of a quality rarely seen in education. Videos were shot in studios, on-site within the business school, and on location in a variety of businesses to bring students into the workplace.

My roles in this project were varied. I provided instructional design on four courses, helping faculty decide how to convey their content in a meaningful way. I developed a number of tracking tools and workflows to help us organize the astonishing number of pieces to each course. The simplest course I managed had 21 videos and 21 Storyline interactives, all of which I proofed 3-6 times. My most complex task was to help set expectations and facilitate communication among our team of 30+ education and creative professionals comprising internal staff, university personnel, and staff from three separate contracting firms.

Dental Hygiene eLearning Program Development – 2007-2010

DH master's students_ClassOf2016_Web
MS Dental Hygiene Class of 2016, with their computers

In 2006, department chair Laurie McCauley charged Wendy Kerschbaum and Anne Gwozdek with developing an online dental hygiene degree completion program to better equip dental hygienists for the next decades of the profession and to increase sagging program enrollment.

Wendy and Anne contacted me to help them shape every aspect of the program, from the highest-level planning through the creation of 11 individual courses. I also helped them develop a curriculum-wide portfolio that keeps students engaged with and oriented to program competencies throughout the 2-year program. When the degree completion (BS) program was complete, we then developed a 2-year online master’s degree program based on the same model.

Importance and Impact of Project or Work

Active learning, online

A major goal of this curriculum redesign was to “raise the bar” of student learning. Results from the first 10 years of the program show that students are, indeed, learning more material at a deeper level than students in traditional face-to-face classes. We achieved this by:

  • increasing student responsibility for reading and understanding course texts,
  • creating assignments that ask students to apply what they’ve learned in realistic scenarios and in the field with community partners, and
  • developing challenging grading rubrics to guide student performance.

A portfolio to keep an eye on

The portfolio asks students to critically analyze what they’ve learned at the end of each course. This metacognition gives students a chance to place each course’s content into he context of the program as a whole and to orient themselves before beginning the next course. The DH portfolio has been held up at the campus and national level as an exemplar of this much-touted teaching technique.

Program successes

  • Students and faculty report that students learn more and do more active learning in the online vs. face-to-face bachelor’s degree program.
  • Success of the BS program led to the development of the MS program, especially after multiple BS graduates complained to faculty that the master’s programs they entered were not as rigorous as the bachelor’s program they had just left.
  • Of the first six cohorts of the BS program (first group graduating in 2009):
    • 25% are enrolled in grad school or professional school/or have graduated
    • 23% are working at community-based clinics and/or involved with PA 161 Programs
    • 33% have received awards/publications
    • 28% hold leadership positions in professional associations

Contributions

My role was primarily that of consultant and facilitator. I asked my colleagues to describe their goals and make their implicit assumptions explicit. I reviewed drafts of syllabi and assignments, offering suggestions for changes to increase student learning or decrease faculty workload. Faculty always had control over course content and the final say in the content and format of assignments.

Working closely with administrators and faculty, I helped shape the development of:

  • High-level program goals – specifically, explaining how to achieve academically rigorous online education
  • The overall structure of the program – including the Portfolio
  • Eleven individual courses – from course outline through the writing of individual rubrics
  • Co-taught final portfolio course – leading students through developing a final presentation portfolio
  • Faculty development – in online teaching techniques, giving feedback, and inter-rater reliability in grading portfolio entries

Publications related to the program

Press

See a selection of articles about Emily’s efforts.

Publications

Research and Publications

I design every major curricular project with a comprehensive evaluation plan, which allows for program assessment and dissemination of best practices. Here are a number of publications based upon that research.

The Transformation Rubric for Engaged Learning

As we started doing focus groups with our graduating cohorts and two-years-out alumnae in the Dental Hygiene Online Degree Completion Program, we kept hearing students say “this program changed my life.” All the faculty and directors of the program had taught in traditional dental hygiene programs for years, and no one had ever heard students us these terms to describe their education. We had already documented the academic rigor of the program, but we felt it was important to capture the transformative nature of the program.

Quantifying transformation

In the dental world, if it can’t be measured, it can’t be reported. So we set out to design a reproducible method for coding any type of student-generated text (written reflections, exit interview transcripts, etc.) for type of change (e.g., leadership, professional skills, clinical skills) and depth of change (i.e., transformative or not transformative). This would not only let us explain the degree to which our program transformed students’ lives, but could also be used by other programs to evaluate their impact.

Over time, programs could potentially compare the impact of one intervention to another and weigh the relative benefits and costs.  For example, by analyzing student exit interviews over several years, a department might be able to decide eight-week internships are nearly as beneficial as twelve-week internships, or they might realize a reflective portfolio is worth every minute faculty and students spend reviewing the reflections together.

Related Link

Transformation Rubric for Engaged Learning: a Tool and Method for Measuring Life-Changing Experiences.

Help Yourself Garden

Why a garden for a School of Dentistry?

There was a beautiful space for a garden directly across our patient parking area from the main doors of building. It had great soil, southern exposure, ADA compliant garden bed, and sprinklers. Why not build a vegetable garden there? So in 2013, I secured permission from various university entities and began the Help Yourself garden.

Each year, the garden has a theme, with various sections of the long, narrow bed marked with explanatory signs. Themes so far have included:

  • continents, displaying plants in the continents of origin
  • recipes from around the world, with a blog of recipes
  • colors, with rainbow-hued vegetables
  • “Garden of Science,” with signs explaining plant families, fractals, Gregor Mendel’s pea plant-based genetic experiments, and a Three Sisters Garden
  • “Five Senses” garden, with plants to touch, taste, smell, see, and even hear

Staff, students, faculty, patients, and passersby visit the garden at a rate of about 50 people a day to enjoy:

  • the stress-busting joy of interacting with nature right outside the door
  • bites of fresh produce
  • education about how to garden
  • collegial interactions across traditional group boundaries
  • rest, play, and creativity, not bounded by the usual millimeter-specific strictures of dental education

Garden links

Pathways Program

pathwaysMap
Map of original Pathways program curriculum, showing differences between the three tracks

The University of Michigan School of Dentistry Pathways Project is a co-curricular program that allows students to explore areas of interest within dentistry. Unlike the four-year core curriculum, in which all students take the same courses at the same time, Pathways allows for student-directed experiences within three distinct “paths”: research, leadership, and healthcare delivery. The program includes faculty mentoring and culminates in an individual or group project one to seven semesters in duration. At inception, all 400+ DDS student participate in Pathways across their four years of dental school.

Importance and Impact of Project or Work

There is no set curriculum or work plan for a dentist once she’s running her own practice. However, due to the intensive nature of the dentistry curriculum, traditionally, the first two years of the program are rigorously scheduled with no time for exploration and no room for student self-direction. Not only is this a disservice to students who will have to self-direct during their careers, it’s stifling to students to want to excel and begin making a difference in people’s lives.

Projects in the first few years of the program have made huge impacts, including:

  • Providing oral health care to veterans without insurance
  • Providing oral health care, clean water, and computers to students at rural schools in Kenya
  • Developing expertise in the latest digital dentistry techniques
  • Research on brain and motor behavior adaptations in congenital orafacial anomalies
  • Studies of gender in patient-dentist relationships
  • Explorations of the efficacy of various techniques for teaching oral health care to children

Students do significant good in the world through their Pathways projects!

Roles and Tasks Accomplished

I served several roles in the Pathways program: project manager, curriculum designer, and program evaluator. The most challenging part of my involvement with the program was the timing: the core group of faculty planners had had six months to take the project from concept to enrollment. I was brought in over a year after the first students began the program. Therefore, this was a difficult case of “building the bridge as we walked on it” for an exceptionally complex program. Specific accomplishments with which I assisted the group include:

  • Compiling and refining existing goals for the three paths into a cohesive set of Pathways Program Goals
  • Finalizing activities and policies for all four years of the program for each of the three paths – including consensus-building among six co-directors and two administrative staff in less than an hour per week
  • Writing the policy manual for students, faculty, and administrators
  • Needs analysis and software selection, customization, and implementation of software to track students’ progress through the program
  • Student and faculty training sessions in person, in print, and via online videos
  • Developing the program evaluation plan
  • Collecting and analyzing program data and writing initial program evaluation reports

Related Presentation

Pathways ADEA 2014

 

Three Functions of Instructional Technologies

threeTypesLearningTech
Learning technologies can deliver content, assess knowledge or – most importantly – foster engagement with the material and with other people

Learning technologies can be broken into three functional categories: content delivery, assessment, and engagement. There are ways to do each function either analog or digitally, and there are numerous digital tools each category (and some technologies span categories).  Knowing what you want to use technology for is a crucial first step to selecting a particular technology to use.

Content delivery

This is the subject matter we are trying to teach. Content can be delivered via spoken words, written words, and still or moving images. Common technologies include lecture, images, and videos.  It is crucial to recognize that content delivery is not sufficient to bring about learning.

Assessment

Assessment verifies students have learned something.  Assessment strategies boil down to two basic categories – written work (tell me what you know) and performance of a task (show me that you can do something). Common technologies include online testing software and recording a live performance.

Engagement

Learning happens when students engage with learning material and find a place for it in their broader understanding.  Engagement is encouraged when students compare/contrast new material with what they already know, apply what they’ve learned, and help other people understand the material.  If your students aren’t learning your content well, consider enhancing your engagement strategies first. 

Related Presentation

ADEA 2014 Three types of instructional technologies

Educational Portfolio Program Development

protfolioDiagram
Portfolios have more utility in conjunction with higher-order learning.

I have been designing reflective student portfolio programs since 1996, beginning with the Kalamazoo College Portfolio – which was, to our knowledge, the first required reflective electronic undergraduate portfolio in the US. I have since consulted with dozens of colleges and universities about starting their own portfolio programs.

Description of the Project

“Portfolio” is a defined differently by nearly every person who uses it. There are three main types of student portfolios:

  • Assessment Portfolios are collections of student work, and their primary purpose is to grade the student on their performance on course objectives. These are usually shared with course instructors.
  • Reflective Portfolios ask students to analyze their own actions, competence, thoughts, and feelings. They are usually shared with a small group of people, such as instructors, advisors, or peer facilitators.
  • Self-Presentation Portfolios are polished works targeted at an external audience, often with the hopes of helping a student get a job. These are seen as a detailed supplement to a resume and may contain examples of work.

Many portfolios incorporate elements of each of the types described above – but the most effective portfolio programs focus heavily in one or two areas. It’s just too hard to do everything well.

Importance and Impact of Project or Work

Portfolios have been a “hot item” in education for more than two decades. They have great potential to impact student learning, critical thinking skills, and confidence, but a significant amount of instructor and administrative effort is required to reap substantial benefits. There are times when portfolios are not appropriate – fore example, a portfolio is overkill for the assessment of straightforward, knowledge-level understanding. You don’t need a portfolio to tell you if a student has mastered algebra or has memorized vocabulary words.

My professional stance is that a bad portfolio program is worse than no portfolio at all, because it can waste precious time and cause significant frustration if it is perceived as a complex series of “hoops to jump through.” However, a well-designed portfolio program can have a transformative impact on student learning and confidence. With the depth and breadth of my experience, I am able to help academic programs:

  • clarify their needs, goals, and specific outcomes for a portfolio program;
  • decide which – if any – type of portfolio will meet their goals within their specific circumstances;
  • develop the program, including faculty and student buy-in and training materials.

Related Link

U Michigan Dental Hygiene Degree Completion Portfolio