Preparing for an ACGME Site Visit (outline and more detailed versions)UAMS Documents You will Need for Your Site VisitHelpful Hints for the Site VisitDescribing the Internal Review Process for the ACGME Site VisitWhat Happens During the ACGME Resident Interview |
Preparing for an ACGME Site Visit
Source: ACGME Bulletin 1999
Accreditation
a peer review process, a fundamental part of medicine and medical education
self-regulating
safeguards the quality of physician education
program is judged by the Program Information Form (PIF), site visitor report and supplemental data (e.g., resident surveys/interviews and operative logs)
Program Director Preparation
Start early
Review Program Requirements and Institutional Requirements
Read the most recent accreditation letter- note citations and concerns
Identify and realize improvement opportunities
Good communication with residents and faculty
Convey the attributes of the program to the site visitor in a clear and accurate way
Must be well organized and complete
Data must be accurate, consistent, current
Do not submit un-requested materials such as manuals or reprints
Call the RRC Executive Director if there are questions or confusion regarding preparation
RRC member spends 40+ hours before each meeting reviewing PIFs
Do an excellent job the first time and avoid having to submit further progress reports
Preparation includes efforts by program director, program coordinator, faculty, residents, institutional officials
Residents and faculty should review the PIF- can point out errors, benefit from becoming familiar with the description of the program
The best way to prepare is to have on-going communication, evaluation and responsiveness with faculty members and residents
Day of Site Visit
# residents = number of names on resident list
months/FTEs at each institution = number of blocks on block diagram
# of resident evaluations = frequency of resident reviews
institution referenced in narrative is not mentioned elsewhere
faculty list/faculty credentials = faculty CVs attached
Site visitor will see:
affiliation agreements
a sample resident contract
resident and institutional policies
written goals and objectives
sample evaluation forms
completed evaluations for residents and faculty
operative logscall schedules
residents’ files
due process policy
written curricula
conference schedules
attendance records
data specific to your discipline
Resident Interview by Site Visitor
Area where, based on the review of the PIF, the site visitor thinks the program may not be in compliance
Areas where he/she senses a discrepancy between what is reported in the PIF or by the program director and/or faculty and what may actually occur
Sample questions:
Are you aware of resident work hour rules?
Does your institution abide by these rules?
Describe how your educational process is evaluated
Have you seen and did you receive a copy of the program’s educational goals and objectives? When?
What are the reasons you chose this program?
Do you evaluate the faculty, the rotations and the educational program? How?
How and how often are you evaluated?
How are you supervised? Describe. Is this too much, too little, just right?
If you or a colleague had stress or other difficulties, how would you get assistance? Would it be confidential?
Explain your call system
If you had to use the due process procedure, where would you find it and who would you contact?
How did you get selected to meet with me?
Were you selected by your peers?
Were you prepared by your program director or faculty for this interview?
Were you told what to say?
Do you have an institutional closure/reduction policy?
Do you have a policy on moonlighting?
Who serves on the institution’s GME Committee?
How are the residents selected?
Do you have time to study?
Do you think you are getting a good education?
Do you participate in the care of all patients?
Do you have enough inpatients and out patients?
When do you evaluate the overall training program?
Does someone represent you on the educational committee?
How often does the program director or his/her designee review the formal evaluations with you?
Do you get to see your evaluations from the faculty?
Give an example of a concern that was acted upon by the program director and faculty
Do you get at least 1 day in 7 free from hospital duties?
Is there a balance between service expected from a resident and the quantity and quality of time residents are able to devote to their education?
Do you get enough operative/clinical cases so that you may increase your skill and competency?
What type of continuity of patient care experiences do you have?
Are you educated in an atmosphere of professionalism and quality?
Describe the educational climate of your program and the institution.
Is the program and institutional commitment to education?
What are the strengths of this residency education program?
What are the weaknesses of the program or what could be improved about this program?
Role of the Residency Review Committee (RRC)
(excerpt from presentation, ACGME Mastering the Accreditation Workshop,
February 2002)
RRC
Develops data collection tools such as PIFs and operative logs
Ensures quality resident education
Usually meets twice per year
(schedule is posted on ACGME website)
Before the RRC Meeting
After the site visit, the PIF and site visitor's report is submitted to the RRC
RRC staff assigns one or two reviewers to each program that has been reviewed by a site visitor
Program files are distributed to RRC members 4-8 weeks prior to the RRC meeting
File includes PIF, site visitor's
report and program's accreditation history
Program Review at RRC Meeting
RRC member reviews PIF, site visitor's report and program's accreditation history and prepares written evaluation
Written evaluation includes - brief history of the program, significant changes since last review, steps taken to address previous citations, inconsistencies or omissions in the material provided for review, areas of non-compliance, program's strengths/weaknessesRRC member presents comments to the full committee followed by discussion by the full committee
RRC member recommends accreditation action
Full committee reaches consensus regarding-
Accreditation status
Length of review cycle
Area of non-compliance
Areas in which education may be improved
Commendations to program for significant improvements
Preparation for an ACGME Site Visit
Key aspects of the preparation effort that result in a good site visit
include an early start; a thorough review of the program requirements; an
in-depth reading of the most recent notification letter; ongoing efforts to
identify and realize improvement opportunities; good communication with
residents and faculty; conveying the relevant attributes of a program in a clear
and accurate way to the site visitor and, through the Program Information Form (PIF)
to the RRC. The site visit process really begins immediately after the last RRC
review. If you were not part of the last visit, and you cannot find a copy of
the most recent notification letter, contact Dr. Heard or the ACGME to get a
copy. Look at citations and concerns identified in the last accreditation
action, and review what your program has done to address them. If the answer is
"nothing or not much" change your strategy quickly. Repetitive
patterns of non-compliance are not tolerated well by the RRCs. Ideally, mid-way
between the last visit and the tentative next site visit date there should be an
internal review of the program to assess progress in areas of concern and
explore the program’s strengths in areas needing improvement. The start of the
preparation of the visit may be a good time to call the executive director of
your RRC (identified on the website www.acgme.org)
he or she will give you accurate information based on a wealth of experience and
with other programs in similar situations.
The RRC will judge your program against the standards published in the requirements, which can be downloaded from the ACGME website. Start with a thorough reading. You may find that you disagree with some of the requirements. Make a note to contribute actively to the next generation of requirements. At the same time do not expect that because you do not agree with a particular requirement your program will not be held accountable for complying with it. Next, download the PIF from the website. Remember that the site visitor’s role is to clarify and verify the information you have put in the PIF. Answer all the questions. The most common reason for an adverse action that is subsequently reversed is an incomplete or poorly prepared PIF. Keep your audience in mind. The average RRC member spends 40 hours before each RRC meeting reviewing programs for the meeting. These are busy physicians, committed to their work, but not appreciative of a sloppily presented program. It is preferable to do the best job the first time.
Establish a time table to keep this project on track. Although you will need a team to get the information together, you should thoroughly review the finished product and be very familiar with it. On the day of the site visit, you will be expected to answer questions about the source of the information in the PIF. The internal review panel will review and critique your PIF prior to it being sent to the ACGME. Proofing the content for discrepancies is very important. Some of the most frequently found inconsistencies in the PIF:
Don’t prepare for the site visit alone. Engage help from others- the program coordinator, other faculty, residents, the associate dean, the internal review panel. Parcel out pieces of the PIF to be completed. Make the PIF comprehensive, succinct, clear and accurate. residents and faculty will be able to point out discrepancies and factual errors. An added and more significant benefit is that they benefit from having read the narrative and will be familiar with it on the day of the site visit. It is important that residents and faculty know and understand the various terms used by the site visitors- core curriculum, socioeconomic issue, policy on resident reduction or closure of a program, non-competitive guarantee agreement. The approximate date of the site visit is indicated in the most recent notification letter which is sent to the program director about 3 months in advance of the specific date. The site visitor is also listed. You can read a biographical sketch of the site visitor through the ACGME website. It takes approximately one year or more to do the preparation for the site visit.
A suggested time line for preparation is:
UAMS Documents You Will Need for Your Site Visit
Contact Ann D. Norwood to obtain information about:
1. Institutional Letter of Report for UAMS College of Medicine (COM) (the sponsoring institution)
2. Progress in correcting citations from the last Institutional Review
3. Description of the UAMS COM GME Committee internal review process
When preparing the PIF:
DO
|
DON’T
|
When preparing for survey day:
What happens next:
There are two types of site visits- those conducted by the ACGME field staff and those conducted by specialists. In either case, the site visitor’s primary responsibility is verifying the information you have provided. To that end, he or she also will conduct interviews with administrators, faculty and residents.
After completing the site visit, the surveyor submits a written report to the RRC Executive Director, who prepares the program file for evaluation. The site visitor does not participate in the final accreditation decision of the RRC beyond providing this report.
Notification of the outcome:
Most RRCs meet twice annually. The schedule of RRC meetings is published in the ACGME Bulletin. The RRC staff will know the date of the meeting at which your program will be considered. You will be notified of the RRC’s decision by letter.
Helpful Hints for the Site
Visit
from ACGME Bulletin July 1999
The following may be valuable to both new and established program directors. First, in ensuring the program's compliance with the requirements and in preparing for the site visit, it is vital that the program director know the program requirements. The Executive Director for the appropriate RRC should be contacted for clarification where the language or the intent of the requirement is not clear to the program director. Program directors may also want to consider developing a checklist to document the program's status for each of the program requirements.
In preparing for the site visit, the program director should take ownership and responsibility for the Program Information Form (PIF). While delegation of sections of the PIF to members of the teaching staff, residents and others is important and necessary, the program director him-or herself must be familiar with the entire document to facilitate a review and discussion with the site visitor. When questions arise about the PIF, many can be answered by the ACGME help desk, which can be reached via electronic mail at helpdesk@acgme.org or by calling at 312/464-5393. Questions related to the content of the PIF should be directed to the office of the Executive Director or the Accreditation Administrator of the appropriate RRC. The RRC's are responsible for developing, maintaining and periodically revising these forms.
The primary task in completing the PIF is to prepare an accurate description of the program that demonstrates compliance with the requirements. This cannot be stressed enough, because a comprehensive and detailed description of the program may not be adequate from an accreditation perspective if it fails to describe how the program meets the requirements. It is important to remember that the RRC does not review 'the program' per se, but rather a description of the program. Similarly, the ACGME field representative during the site visit reviews the description of the program that is presented in the PIF and expanded on during the site visit. the site visitor prepares his or her report on the basis of this description and the degree to which the information presented in the PIF could be verified during the site visit. Some program directors may use a mock survey to prepare for the actual site visit. If possible, this mock survey should involve staff from other programs in the institution as well as colleagues from other institutions.
On the actual site visit day, it may help to know that when the field representative meets with the Program Director, the initial discussion will focus on (1) how the program has addressed the list of previous concerns (if any) identified by the RRC at the program's last review; and (2) any major changes in the program that have occurred since that last site visit. Examples of major changes include changes in program sponsorship or administration, affiliations, faculty, facilities or rotations. Being well prepared for these questions can add to the program director's confidence during the remainder of the visit. To learn more about the accreditation process, new program directors, especially, should note the next meeting dates of their professional societies and attend all sessions that are specific to program directors in their specialty or subspecialty. Basic information and periodic updates on the ACGME accreditation requirements and process are also presented at an ACGME-sponsored conference entitled "Mastering the Accreditation Process," which is held at least once per year. The conference introduces program directors to the accreditation process and provides specialty-specific updates on the accreditation process.
Program directors are urged to use the resources of the ACGME offices. In addition to the Executive Directors and Accreditation Administrators who staff the individual RRCs and the Institutional Review Committee (IRC), the ACGME also has staff knowledgeable in computer applications, scheduling of site visit and all aspects of interface with the ACGME Field Staff, the appeals process, institutional review, how to obtain program consultants, and other useful information.
Documenting
the Internal Review Process During the ACGME Site Visit
from ACGME Bulletin July 1999
Program directors and institutional officials regularly pose the question of how they should document the presence of a functional internal review process during the site visit. The following summarizes the suggested approach for documenting the existence of an internal review process in the context of both a program and an institutional site visit.
The matter of internal reviews is an important issue. The ACGME Institutional Review Committee (IRC) and the Residency Review Committees (RRCs) are committed to ensuring that sponsoring institutions have and internal review process in place and operating. For this process to be effective, institutions and programs must believe that they will not incriminate themselves by candidly exploring the strengths and weaknesses of their programs. To ensure that programs can do this, the internal review documents must be private.
During the site visit of a residency education program, the ACGME field representative will NOT ask to see the actual report prepared following the internal review of the discussion of the internal review in the minutes of the Institutional GME Committee (GMEC). The exclusive purpose of these documents is use by the program and sponsoring institution in identifying and addressing deficiencies that may impact residency education. To be useful, they must be candid documents that freely discuss the positive and negative aspects of the program.
The ACGME site visitor will NOT ask to see the minutes of the GMEC for verification that individual residency programs have undergone an internal review. The ACGME field representative WILL ask to see evidence that an internal review of the program has taken place. This documentation will usually be a written synopsis of the review that discusses when it occurred, who was involved and the process used.
If the program director is unable to produce the type of documentation, the field representative will report this fact. He or she will not request to see information about the internal review from the full report or from the minutes of the GMEC. However, if the program opts to produce these documents as their evidence that an internal review has occurred, the field representative can look at them. The members of the ACGME field staff have been instructed that review of these documents must be limited to assuring that an internal review has occurred and that an appropriate structure and process were used. No other information that could be gleaned from the review of these institutional documents will be included in their site visit report.
What Happens
During an ACGME Resident Interview
by Ingrid Philibert, ACGME Staff
from ACGME Bulletin October 1999
Any discussion of what transpires in the resident interview that is part of the ACGME site visit needs to be prefaced with two fundamental statements about the ACGME’s accreditation process. First, the accreditation process assesses whether the residency education program meets the ACGME/RRC requirements. Thus, virtually all questions asked during the resident interview have their foundation directly in the requirements- are in essence a translation of the requirements into a question format. Second, the role of the ACGME site surveyor is to clarify and verify the information the program has provided in the Program Information Form (PIF), and a significant portion of this corroboration occurs in the interview with the residents. Questions the site visitor asks the residents will focus on two areas: (1) area where, based on the review of the PIF, the site visitor thinks that the program may not be in compliance with the requirements; and (2) areas where he or she senses a discrepancy between what is reported in the PIF or by the program director and/or faculty, and what may actually occur. This means that beyond a small number of general questions (a limited selection is shown below), the questions change for each program.
A Sampling of the Questions Posed during the ACGME Site Visit Resident Interview
At the start of the resident interview, most ACGME surveyors provide a mini-overview of the ACGME and the GME accreditation process, to put the resident interview into perspective. During the interview, the format of the questions comprises a mix of closed questions, such as "Are you aware of resident work hour rules? Does your institution abide by these rules:" and open-ended questions, such as "Please describe how your educational progress is evaluated?"
Two important ‘capstone’ questions, generally asked at the conclusion of the resident interview, are "What are the strengths of this residency education program?" and "What are the weaknesses of the program?" This offers residents an opportunity to comment on the program from their perspective, and can identify areas where residents have concerns that did not emerge in the review of the PIF.
For internal medicine programs (core and subspecialties), use of the computer-assisted Accreditation Review (CAAR) System causes the questions asked in the interview to be focused on areas identified as potential concerns in the CAAR surveys. CAAR involves the completion of a written survey by the residents approximately 6-18 months prior to the actual site visit. The tabulated results are provided to the site surveyor, who uses this information to tailor the questions for the resident group (some of whom may have completed the questionnaire and some of whom may not). In addition, some ‘generic’ questions and the open-ended questions about programs’ strengths and weaknesses are also asked.
The information shared by the residents during the interview remains confidential. The site visit report will never state the residents’ names, it merely reports whether residents raised a given issue unanimously, whether a group of residents reported it, or whether a single resident or a minority made a statement. Confidentiality is important, because it enables the residents to comment frankly on their educational program.
Through its field representatives, the ACGME interviews between 300 and 400 residents per week. Over a given year, just over one-fourth of the nation’s residency programs are site visited and a representative selection of their residents comment on the positive and negative aspects of the residency program. This makes the resident interview an important source of information about how residents in the United States perceive their educational program. The potential of this information source in obtaining feedback on and improving the quality of graduate medical education has hot been fully realized. A current ACGME pilot effort seeks to extract this information, aggregate it, and provide it in summary form without institutional identifiers to the graduate medical education community, to permit them added insight into how resident s view their educational programs.
Last edited: 01/02/07