C3DO 2024 Updates

The call for proposals from COMs interested in participating in Phase 3 of the C3DO pilot project closed last week. Our appreciation to those COMs that attended the request for proposals (RFP) informational webinar and/or reached out to the NBOME with questions, and especially to all those that applied. If you have any questions about this process, please reach out to C3DO@nbome.org.

COMs selected for inclusion in Phase 3 will be joining 8 COMs (10 campuses) that participated in Phase 1 and/or Phase 2 and have signed on to return for Phase 3.

In October, the NBOME facilitated a case development workshop, with experts from across the country—most from C3DO pilot COMs—convening at the NBOME Philadelphia offices develop new C3DO cases and review existing ones. The workshop gave the participants the opportunity to share their experiences with the C3DO pilot as well as to create new content for Phase 3.

In the coming weeks, the NBOME will hold meetings with both the C3DO Advisory Panel and the C3DO Task Force. The task force will be discussing pilot schools scheduled for phase 3, as well as reviewing new data and suggestions from Phase 2 COMs who have completed administrations.

Reminder if you are a member of any of the following groups: osteopathic undergraduate medical education faculty, graduate medical education faculty, or state licensing board representatives! It is the final days to complete the stakeholder survey, open through November 20. If you have not received the survey, please reach out to the NBOME at C3DO@nbome.org. Results will be presented to the task force and NBOME Board of Directors at future meetings.

Given the continued success and improvements that have been realized with the C3DO Project, the NBOME Board of directors has endorsed a Phase 3 pilot in 2025.

The NBOME will release the Request for Proposals (RFP) for Phase 3 of the C3DO pilots to Deans of colleges of osteopathic medicine on Monday September 30. The RFP is a guide for new COMs considering submitting a proposal to be included in Phase 3, with details on the assessment and instructions for applying. The RFP reflects lessons learned throughout Phases 1 and 2. Applications will be accepted through November 4, 2024.rnrnPhase 3 COMs will pilot automated transfer of C3DO exam data and encounter videos directly from COMs’ exam software programs to the NBOME’s C3DO software. Software vendors that we have partnered with are hard at work building these transfer systems, contributing to feasibility for Phase 3 COMs.

The RFP will be sent to COM deans along with a registration link for an optional informational webinar, which is scheduled for October 14, 2024. For information about the RFP or webinar, please reach out to your dean.

To ensure the continued participation and engagement of other stakeholder groups in defining osteopathic clinical skills assessment and standards achievement, the NBOME will also conduct a survey to Undergraduate Medical educators, Graduate medical education representatives, and state licensing board members. If you are a faculty member or state licensing board member or representative, please reach out to the NBOME at C3DO@nbome.org and we will forward your organization a link for this survey for its faculty/members. This survey will be released the week of October 7.

In August, three COMs continue to administer their Phase 2 C3DO pilot sessions, with one more, A.T. Still University-Kirksville College of Osteopathic Medicine, preparing to begin their sessions next month. This summer, the NBOME has been supporting these schools, finishing performance analysis for the four pilot COMs that had already completed their Phase 2 sessions, and analyzing all the data coming in from all of our pilot partner COMs. We are grateful for the work of all these partners—totaling 11 COM campuses to date!

The NBOME is preparing for a likely Phase 3 beginning in April 2025. The NBOME team is using lessons learned to further improve the assessment and its processes, most significantly in the areas of data exchange and management. Feedback from faculty, committee members, and student participants informs decisions around case content, evaluation tools, administration, and scoring. We continue to outreach for organizational feedback from the licensure community, UME and COM community and AACOM, and the GME/teaching hospital community, as well as specialty societies and student and resident leadership groups.

The C3DO Task Force will meet in September to discuss Phase 3 goals and to finalize new stakeholder surveys. These surveys will help inform future direction based on changes in the educational environment following the discontinuation of the Level 2-PE.

For example:

• How has education and assessment of clinical skills transformed at COMs and in GME since 2020
• Has the perception about the importance of attaining competency in these skills evolved over the past 5 years?
• Have the skills of our students and residents changed over time?

We are creating a request for proposal for a potential Phase 3 of additional pilot studies, which will be shared with COM Deans in late September in order to gather information from COMs interested in being considered for participation in the further development and evolution of C3DO. Stakeholder surveys, pilot progress and outcomes, and external factors are reviewed by the NBOME Board of Directors regularly and future directions in osteopathic clinical skills assessment will be announced following a review of Phase 2 final report at their June 2025 meeting.

Clinical skills verification currently remains in place for COMLEX-USA through the graduating class of 2027 using a dean’s attestation that each graduating student has demonstrated the osteopathic clinical skills necessary for graduation, with clarification for the graduating classes of 2028 and beyond expected by July 2025.

The C3DO Task Force met on June 3 to review Phase 2 progress so far and discuss possible future uses of the assessment. Staff updated the Task Force on pilot progress in Phase 2. In June, two pilot COMs completed their Phase 2 administrations, Campbell University School of Osteopathic Medicine and University of New England College of Osteopathic Medicine, and one pilot COM began theirs, Rowan-Virtua School of Osteopathic Medicine (Rowan-Virtua SOM). Rowan-Virtua SOM will be administering the C3DO through March 2025. The final Phase 2 pilot COM, A.T. Still University-Kirksville College of Osteopathic Medicine, is gearing up for its administrations, slated for September 2024.

Much of the meeting was devoted to discussion regarding recommendations for future use of the C3DO: as a licensure requirement, as a clinical skills verification tool, and other programmatic uses. Task Force members discussed the strengths and weaknesses of the options and the input that has been received from the Advisory Panel and the NBOME Liaison committee. Finally, they suggested additional data that would help support decision-making for the future direction of the program, including stakeholder surveys and additional information from pilots.rnrnThe NBOME Board of Directors also met in June to review and discuss next steps for a potential Phase 3 pilot. We will begin planning for Phase 3 this summer. More information will be released soon, starting with existing partner colleges.

The NBOME is grateful for all of the work these and the other pilot COMs are doing to make the pilot a success. Clinical skills verification currently remains in place for COMLEX-USA through a dean’s attestation that each graduating student has demonstrated the osteopathic clinical skills necessary for graduation through 2027.

C3DO Phase 2 continues to expand. Three more colleges of osteopathic medicine (COMs) launched their pilots this month—all three in a row, with start dates of April 10, April 12, and April 15—for a total of four pilots currently running. Two other COMs begin administrations in May.

A member of the NBOME team visits COMs participating for the first time in Phase 2. This provides the COMs with an onsite resource and allows the NBOME team to see in person how each COM makes the C3DO their own, nesting the standardized assessment within their local setup and protocols. The COMs do amazing work and it’s been extremely helpful to see and learn from their approaches to the C3DO.

The NBOME and participating COMs are also sharing experiences—with the osteopathic medical education community and globally. The NBOME presented on the C3DO at The International Meeting on Simulation in Healthcare (IMSH) in January and the Ottawa Conference on Clinical Competence in February. Attendees at last week’s American Association of Colleges of Osteopathic Medicine (AACOM) Educating Leaders conference had the opportunity to hear from the Phase 1 COMs about their experiences participating in the C3DO pilots and why they decided to continue with the program.

The COM perspective is also featured in the Annual Report 2023: SOARING. If you missed those presentations, catch our upcoming presentations at the Association of Hospital Medical Education (AHME) in May and at the Association of SP Educators (ASPE) in June.

C3DO Phase 2 testing began in March and three colleges of osteopathic medicine (COMs) are preparing for an April start to testing. Pilot COMs testing later in the year are in various stages of preparation depending on their testing schedule. As a COM prepares for the C3DO, the NBOME staff walks them through selecting cases and reviewing forms for their C3DO pilot testing sessions.

The forms, or groups of cases they will use, should be reasonably equivalent across COMs but can vary based on COM resources. The NBOME has developed standardized online training for proctors, standardized patients, and physician examiners, with check-in meetings to get feedback from users. The NBOME also uses these check-ins to get feedback on how the protocols are employed in each school, and how each COM informs and coordinates student participation. Phase 2 materials include a video example of C3DO with sample post encounter questions – complete with scoring rationales.

The C3DO Task Force met on March 20. Members received a brief update on Phase 2. Afterward, the Task Force engaged in discussions on quality assurance strategies for the first three goals of Phase 2.

Goals of Phase 2:

• Can we create a standardized distributed osteopathic clinical skills assessment (added cases and questions from Phase 1)?
• Is that assessment feasible in implementation and administration across various COMs, in various models (with significant updates from Phase 1)?
• Does that activity produce scores that are “reasonable” given other known measures?
• How can C3DO provide value to various stakeholder groups?

The first three goals are extensions of goals from Phase 1, focusing on changes made from the Phase 1 administrations. Ensuring we are evaluating the quality of the assessment will provide important information to the C3DO Advisory Panel and then to the NBOME Board of Directors.

The second half of the meeting was a discussion of the fourth goal—the “value” of the C3DO activity to various stakeholder groups. Members contributed their thoughts on how C3DO may provide added value to groups such as students, faculty, and administration at COMs, state licensing boards, residency program directors, patients, and others. Members asked insightful questions and provided valuable guidance regarding what this activity can and does measure and for what potential purpose.

These discussions contribute to the growing body of information to be considered by the NBOME Board of Directors regarding this pilot program of a distributed, standardized model for osteopathic clinical skills assessment.

The word of the month of February was “preparation.” With one C3DO pilot COM slated to begin testing in March and three following shortly after in April, the NBOME team spent February supporting the COMs as they trained exam staff and standardized patients and prepared their testing centers for the pilot.

This was also a time of student preparation, as the COMs provided their students with resources that included an orientation guide and a sample encounter video with practice questions. February also saw the training of physician examiners, the rollout of new and updated training materials, and work toward process improvements for data transmission. Pilot COMs that begin testing later in the year have also started preparations, eager to learn from their early-testing colleagues as we work together to refine the C3DO assessment.

On January 19, 2024, the NBOME held a kickoff meeting for lead physicians and staff from the eight colleges of osteopathic medicine (COMs) participating in phase 2 of the C3DO pilots. The workshop was designed to introduce the teams to the phase 2 project plan, starting with a summary of phase one and lessons learned.

Important updates to the program that were a direct result of feedback from schools and student participants in 2023 included enhanced case development and item editing, simplification of training materials for standardized patients, improvements in data exchange, expanded student informational materials, and the creation of a comprehensive COM “toolkit,” as a few examples.

Sessions included how to create an exam form (the set of cases a candidate would see on a given testing day); review of Phase 2 administrative and IT elements; an introduction to the NBOME’s online Learning Center and the courses it hosts for C3DO standardized patients (SPs) and staff; discussion of OMT in the C3DO; an overview of SP training; review of resources and timelines; and a discussion about scoring considerations.

Physician leads from the three COMs that had also participated in Phase 1 joined with NBOME staff to share insights from Phase 1 as well as recommendations regarding communicating with and preparing students to participate.

On January 22, each lead received their C3DO “toolkit,” allowing them and their teams to dive more deeply into the topics covered at their own pace and begin their preparations. The next step for each COM, depending on their testing dates, is a kickoff meeting for the COM’s core team. Pilots begin in March 2024 for returning colleges, and April 2024 for colleges participating for the first time, and run throughout the year.