C3DO 2023 Updates

The NBOME Board of Directors has accepted recommendations from the Core Competency Capstone for DOs (C3DO) Task Force for a new cohort of colleges of osteopathic medicine (COMs) to participate in Phase 2 pilots for C3DO. Learn more.

The C3DO Case Development Committee met in our Philadelphia offices on October 20-21. Physicians from across the country convened onsite to review case performance from Phase 1 and develop additional case materials and post encounter assessments for Phase 2. Participants remarked:

“I enjoyed meeting others in the osteopathic profession and learning about what goes into case and testing development.”

“This was a fantastic experience!”rnrnu003ciu003e“Great couple of days working on and editing cases. This is extremely valuable work by all involved and I can’t wait to see the final outcome.”

Additional opportunities will be available to participate through the year. If you are interested in participating, the best way to begin is by joining the NBOME National Faculty and indicating your interest in C3DO.

Proposals from COMs interested in participating in the C3DO Phase 2 pilots were due November 3. De-identified proposals were shared with members of the C3DO Task Force in advance of their November meeting.

The Task Force met on November 28. Members received an update on Phase 1 pilot scoring trends and correlations and the October Case Development Committee meeting. They also continued review of student feedback from Phase 1 to address concerns regarding value, timing, student preparation, and student and COM feedback in anticipation of Phase 2 pilots in 2024.rnrnThe Task Force reviewed the evaluation criteria for the Phase 2 proposals, discussed and debated points of difference among the proposals, and highlighted their relative strengths.

After the meeting, Task Force members individually evaluated each proposal, noted whether they would recommend the submitting COM for inclusion in the Phase 2 pilots, and ranked the submissions for inclusion. Information from the survey will be compiled and recommendations presented to the NBOME Board of Directors at their December meeting.

The NBOME will release the Request for Proposals (RFP) for Phase 2 of the C3DO pilots on Friday, September 29. The RFP is a guide for COMs considering submitting a proposal to be included in Phase 2, with details on the assessment and instructions for applying. The RFP reflects lessons learned throughout Phase 1, including use of an online tool for applications. Applications will be accepted through November 3, 2023.

The RFP was sent to COM deans along with a registration link for an optional informational webinar. For information about the RFP or webinar, please reach out to your dean.

The C3DO Advisory Panel will meet on October 10. This is a panel made up of representatives of multiple stakeholder groups (AACOM [BOD, AOGME, ECOP, COSGP], AOA, AAOE, Licensing Board members, residents, students, and others). See March update for list of members. They will review the progress and reports from the C3DO Task Force and Pilots and advise the NBOME Board of Directors.rnrnAlso in October, a group of osteopathic physicians, including representatives from COMs that participated in Phase 1, will be convening in Conshohocken, PA for a C3DO case development meeting, creating and revising cases for use in the upcoming Phase 2 pilots.

The C3DO Task Force met on Tuesday, August 29. The Task Force reviewed preliminary data regarding both feasibility and reproducibility across COMs. In addition, they reviewed student survey feedback from participants in the Phase 1 pilots. Some examples:

798 students completed the post-activity survey, for a 98% response rate. They rated elements of the C3DO assessment and offered a total of 1,865 comments. 92% of students indicated that the C3DO provided at least a partial assessment of their skills (with additional questions asking about individual skill performance), and 89% agreed that the cases presented appropriate clinical challenges.

Sample comments include:

“I think because the cases were so diverse and many of them required you to think about their health holistically, and OMM was integrated in the case, I really did feel like I was able to integrate my osteopathic training.”

“Overall, this was a great experience and very comfortable being in my home institution.”

“Feel that this format was a formal and very thorough assessment of my clinical knowledge and skills up until this point in clinical training. Highly recommend.”

The NBOME staff is creating a brief video highlighting the student experience—watch for it coming soon!

The Task Force reviewed a draft request for proposals (RFP)—the guide for COMs thinking about submitting a proposal for their inclusion in the Phase 2 pilots. The RFP is being crafted to reflect lessons learned from Phase 1 and to make the process more user-friendly, including streamlining applications by using an online tool to apply. The 2023 RFP will be released on September 29, and proposal responses will be accepted through November 3, 2023.

Initial analysis of pilot studies in 2023 at six colleges of osteopathic medicine (COM) campuses show promising results regarding the feasibility of on-campus standardized assessment of fundamental osteopathic clinical skills of osteopathic medical students. Learn more.

All four C3DO pilot schools have now administered the pilot assessment of a core competency capstone (C3DO) prototype at 6 campuses nationwide. NBOME staff visited the participating colleges of osteopathic medicine (COMs) during their final preparations and the first days of their sessions to assist the COMs as needed and to observe their processes. The NBOME gathered a wealth of information during these visits as well as from exam data.

Our goals were to identify opportunities to assess the feasibility of clinical skills assessment at the COMs while ensuring a national, standardized experience for fair evaluation of all osteopathic medical students. Our deep gratitude to all four schools for their hard work on the C3DO pilots and their gracious hosting of NBOME representatives during those visits.

The NBOME is now turning to analysis of the vast stores of information gathered during the pilot, from site visit reports, examination data, analytics from online training courses for pilot COM staff and standardized patients (SPs), and feedback received through surveys of COM staff, SPs, and students. Information from all of these sources will be joined with lessons learned over the past several months of pilot preparation to inform the C3DO Task Force and Advisory Panel in the coming months, as they consider next steps in C3DO development.

Additional announcements about the future of the pilot program as well as the current COM Dean clinical skills attestations from the NBOME Board of Directors are expected in late July.

The Core Competency Capstone for DOs (C3DO) task force met virtually on May 11. The task force reviewed and discussed the feedback from the C3DO Advisory Panelists’ April meeting. Additionally, task force members reviewed data from the first pilot at RVUCOM, which completed initial runs in April with great success in learning more about opportunities and challenges of administering this type of activity at colleges of osteopathic medicine.

Data included survey results from staff on the pilot implementation and form creation processes – how they put together the case combinations for the students to experience and trained their local staff. Task force members discussed using feedback from these initial pilot locations for the remaining schools in this first phase, and allowed for adoption of additional tools to iteratively improve the activity.

The task force reviewed students’ feedback on participation in the activity, including their impression on the types and distribution of the cases, and whether they felt the pilot adequately assessed their osteopathic clinical skills. Students favorably reacted to an updated post encounter exercise which included clinical decision making items pertaining to the case they saw (as opposed to a SOAP note).

Some comments from student feedback included:

“I appreciated the preparation I made for this as it really helped push my training and understanding.”

“After going through 2 years of clinical rotations, this was an opportunity to demonstrate what I have learned. I felt like my training as a medical student trained me well and this was my chance to show it.”

“think the assessment itself was reasonable but the preparation material could be more specific in explaining what is expected of us–will we be using aspects of different screening exams together etc.”

“There is enough variety to touch on most fundamental osteopathic clinical skills.”

“I would recommend giving a better understanding of the schedule and the expectations.”

“Need OMT to be more apparent.”

Pilots continue through the late spring: MUCOM is administering their pilot sessions now and CUSOM and MWU-AZCOM are beginning their sessions in early June. The task force will meet again in early summer to review additional data.

April was a busy month for the C3DO pilot project. The RVU Utah and Colorado campuses completed their main pilot testing dates and we have received great constructive feedback from staff, administration, standardized patients (SPs), and students. Thank you to all who participated. Marian University College of Osteopathic Medicine, Campbell University School of Osteopathic Medicine, and Midwestern University – Arizona College of Osteopathic Medicine continue preparations for their pilots this spring.

We have already made some improvements for the colleges still to come in the pilot, with added support on training and administration through the use of time-based checklists. We have placed training surveys directly in the learning modules to make completion easier for COM staff and SPs. We have developed robust orientation materials for colleges to inform students about the project.

A centralized group of OMT physician examiners chosen from across the country have been oriented and trained to provide evaluation of the performance of OMT on 3 cases per session. We believe that this model will provide an independent assessment of these skills during the patient encounters.rnrnThe C3DO Advisory panel met on April 12. Please see the last update for the names of these individuals and their represented organizations. This diverse group of individuals represents organizations from across the profession of osteopathic medicine and the learning continuum (including students and residents) as well as patient safety and advocacy. Each person had the opportunity to identify benefits, unintended consequences, and suggestions for the program. They will continue to provide advice regarding the project, the exam prototype, and feedback on reports generated from the C3DO Task Force.

NBOME leaders are meeting again with the state medical and osteopathic medical licensing community on May 3-May 6 for further input on this initiative at the national Federation of State Medical Boards (FSMB) Annual Conference and annual meeting of the American Association of Osteopathic Examiners (AAOE).

The C3DO pilot is in development at our various sites. We appreciate the collegial spirit and value the feedback and expertise of our COM partners.

The pilot format will include 8 standardized patient encounters (14 minutes each); scored elements of history and physical examination (checklist); OMT (centrally scored; distributed physician examiner pool); and communication and respectfulness evaluation (global rubric). There will not be a SOAP note component, but students will instead answer two questions following their encounters: the first an open-ended differential diagnosis (or problem list) format pertinent to the SP presenting problem, and the second an extended multiple choice question aimed at key features of the diagnosis or treatment for the patient they encountered.

Pilot Goals

Feasibility: Can the COMs administer a centrally developed examination? What is the experience like for them? The NBOME will use measures such as surveys of staff, faculty, Standardized Patients (SPs), and students, among other measures. As the participating COMs have selected uses for this investigation that align with their own curriculum, it will be investigated how this can integrate into their current processes.

Standardization: Can the resulting examination be standardized across COMs? The NBOME will analyze student performance evaluations, SP performance, and administration.

Advisory Panel

After an extensive application cycle with nominee solicitation from multiple stakeholder groups (AACOM [BOD, AOGME, ECOP, COSGP], AOA, AAOE, Licensing Board members, residents, students, and others), the chosen members of the C3DO Advisory Panel were notified on March 1, 2023. Their first meeting will be in April 2023, and they will review the progress and reports from the C3DO Task Force and Pilots. Based on the pilot research and the current performance assessment environment, the panel will also advise the NBOME Board of Directors. David Kuo, DO, has agreed to chair this panel.

Members

• Sharon J. Obadia, DO, AACOM BOD
• Italo Subbarao, DO, MBA, AACOM BOD
• Andrea P. Mann, DO, AACOM BOD
• Heath A. Parker, DO, AACOM BOD
• Christine M. Khandelwal, DO, AAOE
• William B. Swallow, DO, AAOE
• David E. Tannehill, DO, AAOE
• Peter F. Bidey, DO, MSEd, ACOFP
• Gautam J. Desai, DO, ACOFP
• Laura M. Rosch, DO, CS, MS, ACOI
• David L. Tolentino, DO, ACOI
• Mary K. Carneval, DO, ACOS
• Alyssa R. Pastorino, DO, ACOS
• Robert Dolansky, DO, AOA
• Carolyn Quist, DO, AOA BEL
• Selena Raines, DO, AOA BEL
• David O. Shumway, DO, HBS, AOA BEL
• Janice A. Knebl, DO, MBA, AOA COCA
• Brian Bowles, AOSED
• Edoardo G. Frezza, OMS – through June 2023, COSGP
• Nichelle Chastain, OMS – added October 2023, COSGP
• Rebecca Giusti, DO, NF At-large
• Natalie A. Hyppolite, DO, MBS, NF At-large
• Maureen P. Barnes, ARM, CPHRM, Patient Safety
• Bruce Anderson, Patient Safety
• Lawsen Parker, OMS, SOMA
• Sherman Gorbis, DO, Educator
• Amanda Cain, DO, Educator
• Suzanne L. Rogers, DO, Educator
• Darren Cohen, DO, Educator
• Brysen Keith, DO, Resident

Next Steps

Pilots begin in mid-April at the COMs’ selected times. We have been able to accommodate all COM requests for their preferred schedule to align with their existing activities.

NBOME staff and consultants spent the first half of February preparing examination materials, creating training protocols and materials, and working closely with the pilot COMs. NBOME is providing cases for the pilots. Pilot COMs will be able to select among these cases to create exam forms (the combination of cases students see in an exam session) that fit their needs while matching the C3DO blueprint, designed to align with the COMLEX-USA Master Blueprint Clinical presentations.

NBOME staff and consultants have been meeting with representatives of the COMs, keeping the Physician Leads engaged with C3DO development and beginning work with SP training teams from the COMs that are administering the first pilot sessions. Training is competency-based, designed to ensure standardized performance across the COMs while letting the COMs draw on their expertise as they work with their SPs.

On Thursday, February 16, Jeanne Sandella, DO, VP for Professional Development Initiatives and Communications, and Amy Lorion, Senior Director for Professional Development Initiatives, presented at AACOM’s Community Forum Meeting. She shared updates and answered questions from the AACOM Board of Deans regarding the C3DO pilots.

The C3DO Task force met again on January 19, 2023. They discussed scoring elements: assessment of OMT; communication, interpersonal skills, and professionalism; history and physical examination. Also under discussion were form requirements-the types of clinical skills cases that should be seen by each candidate. Core NBOME staff and consultants have worked tirelessly to put the Task Force recommendations into place. Some of these activities include: building the exam-day structure; creating exam forms and selecting cases; refining the assessment tools for OMT and for communication, interpersonal skills, and professionalism; and refining standardized patient portrayal protocols. The first pilot exams will be held in mid-April, and pilot sessions will be held through June according to the schedule preferred by the COM.

The goals of this first set of pilots are to examine the feasibility of training, administration, and scoring at each site and to evaluate the standardization of the examination across the sites. The four selected COMs are excellent partners; all have clinical skills testing programs with experienced and dedicated staff and reflect different regions of the country and different sized classes.

A call for members of the C3DO Advisory panel, a larger oversight group formed of broader representation including from specialty colleges, student and patient advocacy groups, and other representation of the profession (e.g., AACOM/AACOM Groups, AOA and COCA, AAOE/Licensure Boards, other GME groups) was sent out earlier this month. Input from across the profession remains integral to the C3DO initiative.

Over the past two weeks, NBOME staff and consultants held meetings with representatives from each of the four COMs participating in the 1st round of C3DO pilots. They discussed pilot planning within the context of each COM’s particular needs and circumstances. This strategy allows for collaboration with the schools as the details of the first pilot are finalized. These discussions will be shared with the C3DO task force at their next meeting on January 19. Future meetings with the COM staff will focus on supporting their efforts on local implementation of the pilot plan.