One of the primary objectives of the Casualty Actuarial Society is to establish and maintain high-quality standards for membership. The value of the CAS credential comes in large part from our offering an educational process that provides the fundamental knowledge and problem solving skills that our employers and other principals expect. The CAS learning objectives are outlined in our Syllabus of Basic Education; candidates demonstrating sufficient mastery of those learning objectives are identified through a rigorous examination process.
The CAS Board oversees this strategic initiative for quality education. It endorsed the implementation of the 2011 revisions to the CAS examinations. These changes were intended to enhance the educational process by arranging topics in a more cohesive manner so that related and complementary topics are tested in the same exam and by introducing online testing of certain materials. While requiring leadership to maintain high standards of excellence in the educational process, the CAS Board has, on several occasions, encouraged leadership to pursue strategies that would reduce travel time for candidates. A proposal to increase the frequency with which CAS Exams 5 and 6 are offered will be considered by the Board in March.
Implementation of these basic education initiatives is the responsibility of the Executive Council. The Vice President – Admissions, together with her committees, recommends, implements and monitors syllabus and testing frameworks that ensure a high-quality educational experience.
We recognize the significant investment that candidates and their employers have made in the CAS educational process. The CAS and its committees work diligently and tirelessly to merit that trust. The low percentage of passing candidates for the Fall 2011 Exams has caused many to question the processes used by the CAS in its examination process. For those, like me, who have been somewhat removed from the process, let me first describe the current process. Then, I will outline the steps that are being taken by CAS leadership in response to the exceptional results of the Fall 2011 exams.
The Current Process
The process used by the Examination Committee on its upper-level exams has been in place for several years. The process utilizes a content-based pass mark process. It encompasses subject matter experts who are trained as exam item (question) writers, exam graders, and Examination Committee leadership in designing meaningful tests of syllabus materials against the learning objectives and evaluating how well candidates have demonstrated a mastery of these objectives.
The process used by the Examination Committee to establish a pass mark begins as item writers develop questions; a preliminary estimate of the expected standard for that question is prepared by the item writers and submitted to the Exam leadership.
Between the times the exam is compiled and the exam is administered, a panel of experts (the Pass Mark Panel) convenes to conduct a comprehensive review of the exam. Their process begins with a review of the definition of the Minimally Qualified Candidate (MQC); the Pass Mark Panel then reviews each question to determine the minimum standard for each question and each question part, giving consideration to the importance of the question to actuarial practice (as reflected in the weights published in the syllabus for that examination) and the difficulty of the question itself. The execution of the Pass Mark Panel review may lead to refinements in the definition of the MQC. A preliminary pass mark is developed from the standards developed for each question to reflect the overall expectations for the MQC.
A third estimate of the appropriate standard for each question comes from the graders. After the exam is administered, the responses to each question are assigned to two graders who have the original grading rubric. As they grade, however, they also consider other answers that they encounter as potentially valid and expand and refine the grading rubric. With benefit of the definition of the MQC developed by the Pass Mark Panel, and having observed actual candidate responses and expanded the grading rubric, graders are asked to provide their recommendations for the minimum standard for their question(s).
At this point, Exam leadership has three estimates of the standard for each question: one from the item writers, one from the Panel, and one from the graders. Through discussions with all constituents, Exam leadership works to reconcile differences, if any, in the three perspectives and to develop a consensus view standard.
Since graders have observed actual candidate responses to a question, their perspective is important in building the consensus view. Where aggregate candidate responses are low relative to expectation, additional reviews are conducted to evaluate whether the question itself was flawed or could be interpreted in multiple ways. The consensus standard for a question explicitly reflects such evaluations.
Once all papers have been graded and a pass mark established from the consensus standard, the Examination Committee reviews those papers closest to the proposed pass mark in their entirety. In this review, they carefully evaluate the candidate scores by question for consistency in grading. A final adjustment to the pass mark may result from this review.
At the end of the day, the Examination Committee has as its sole objective the implementation of an objective, content-based approach to assessing whether candidates have demonstrated the qualifications outlined in the learning objectives. It does not apply a “quota measure” in determining the pass mark. If 100% of the candidates sitting for an examination demonstrate the expected knowledge, then 100% of the candidates pass the examination; if only 20% of the candidates sitting for an examination demonstrate the expected knowledge, then 20% of the candidates pass the examination. Typically, however, the expected knowledge encompasses 67 to 74 percent of the available points on an exam and the application of our standards has resulted in an effective pass ratio of 30 to 45 percent.
Given the preliminary results for the 2011 Fall exams, the Examination Committee undertook additional analyses beyond the reviews outlined above to ensure that consistent standards were being applied and that the passing grade was appropriate.
The Examination Committee also invested considerable energy in exploring possible changes to the demographics of the candidate population that might explain the “outlier” results.
The Committee noted that, for Exams 6-U.S. and 6-Canada, there was a higher proportion of the candidates who were “first-time sitters” for a CAS upper-level exam; they observed that candidates at this stage of their careers may need help in preparing for the transition from multiple-choice questions to the essay-type format associated with the CAS upper-level examinations. They noted that the increasing diversity of our candidate population, including many for whom English is not the native language, may create additional challenges in an essay-type environment. In response to these observations, I have asked the Vice President – Admissions to begin exploring ways to better support our candidates to successfully complete these examinations. I expect that the Executive Council will act on her recommendations this year.
Exam 8 was the first offering of this newly designed three-hour examination. While the exam material was largely derived from the old Exam 9, the shorter exam time, the reduction in the number of questions, and the decision to not test all syllabus materials on each exam may have affected candidate performance. The Committee noted that candidates fared more poorly on questions involving new syllabus materials and on materials that were tested with questions that were asked differently than in prior exams.
Notwithstanding the efforts of the Examination Committee to ensure that its processes are fair and consistently applied, I was surprised and disappointed with the results of the Fall 2011 examinations.
In my role as president, I have had occasion to meet and interact with many CAS candidates and Associate members; I have been impressed with their capabilities and with their commitment to the casualty actuarial profession. The results of the Fall 2011 examination are inconsistent with this observation.
Particularly for Exam 8, most candidates understand and are firmly entrenched in the CAS examination process; most are committed to doing whatever is necessary to successfully complete their examinations. In my mind, the CAS needs to better understand why only 22% of the candidates sitting for the exam (23.9% effective pass ratio) met the requirements.
With the agreement of the Vice President – Admissions and the Chair of the Examination Committee, I have directed CAS staff to engage a professional education consulting firm to perform an independent review of our process. A 2001 audit report of the CAS examination process and procedures led to the implementation of the following: (1) development and publication of the learning objectives for each exam syllabus, (2) training of exam item writers, and (3) a content-based pass mark process. The objective of this review will be to evaluate our current processes against best practices for adult professional education. Upon receipt of the consultant’s report, the Executive Council will consider the consultant’s observations and recommended improvements for implementation as soon as practical thereafter.
I believe that all of the Admissions committees, from Syllabus to Examinations, are committed to executing high-quality and appropriate standards for membership. However, we recognize that any process can be improved. As actuaries, we are committed to a standard of excellence that requires us to reach beyond ourselves to explore best practices and to embrace changes that will further enhance our profession (credit rush at https://www.dresshead.com). It is with that objective in mind that the external review of our current education and examination processes is being initiated.
We encourage you to leave a comment on how the CAS might improve its education and examination processes. I look forward to your thoughts and suggestions.
Updated February 23, 2012:
CAS to Make Fall 2011 Exam Responses Available to Identified Candidates