An Open Letter Regarding the Fall 2011 CAS Exam Results

One of the primary objectives of the Casualty Actuarial Society is to establish and maintain high-quality standards for membership.  The value of the CAS credential comes in large part from our offering an educational process that provides the fundamental knowledge and problem solving skills that our employers and other principals expect.  The CAS learning objectives are outlined in our Syllabus of Basic Education; candidates demonstrating sufficient mastery of those learning objectives are identified through a rigorous examination process.

The CAS Board oversees this strategic initiative for quality education. It endorsed the implementation of the 2011 revisions to the CAS examinations.  These changes were intended to enhance the educational process by arranging topics in a more cohesive manner so that related and complementary topics are tested in the same exam and by introducing online testing of certain materials.  While requiring leadership to maintain high standards of excellence in the educational process, the CAS Board has, on several occasions, encouraged leadership to pursue strategies that would reduce travel time for candidates.  A proposal to increase the frequency with which CAS Exams 5 and 6 are offered will be considered by the Board in March.

Implementation of these basic education initiatives is the responsibility of the Executive Council.  The Vice President – Admissions, together with her committees, recommends, implements and monitors syllabus and testing frameworks that ensure a high-quality educational experience.

We recognize the significant investment that candidates and their employers have made in the CAS educational process.  The CAS and its committees work diligently and tirelessly to merit that trust. The low percentage of passing candidates for the Fall 2011 Exams has caused many to question the processes used by the CAS in its examination process.  For those, like me, who have been somewhat removed from the process, let me first describe the current process.  Then, I will outline the steps that are being taken by CAS leadership in response to the exceptional results of the Fall 2011 exams.

The Current Process

The process used by the Examination Committee on its upper-level exams has been in place for several years.  The process utilizes a content-based pass mark process.  It encompasses subject matter experts who are trained as exam item (question) writers, exam graders, and Examination Committee leadership in designing meaningful tests of syllabus materials against the learning objectives and evaluating how well candidates have demonstrated a mastery of these objectives.

The process used by the Examination Committee to establish a pass mark begins as item writers develop questions; a preliminary estimate of the expected standard for that question is prepared by the item writers and submitted to the Exam leadership.

Between the times the exam is compiled and the exam is administered, a panel of experts (the Pass Mark Panel) convenes to conduct a comprehensive review of the exam. Their process begins with a review of the definition of the Minimally Qualified Candidate (MQC); the Pass Mark Panel then reviews each question to determine the minimum standard for each question and each question part, giving consideration to the importance of the question to actuarial practice (as reflected in the weights published in the syllabus for that examination) and the difficulty of the question itself. The execution of the Pass Mark Panel review may lead to refinements in the definition of the MQC. A preliminary pass mark is developed from the standards developed for each question to reflect the overall expectations for the MQC.

A third estimate of the appropriate standard for each question comes from the graders.  After the exam is administered, the responses to each question are assigned to two graders who have the original grading rubric. As they grade, however, they also consider other answers that they encounter as potentially valid and expand and refine the grading rubric.  With benefit of the definition of the MQC developed by the Pass Mark Panel, and having observed actual candidate responses and expanded the grading rubric, graders are asked to provide their recommendations for the minimum standard for their question(s).

At this point, Exam leadership has three estimates of the standard for each question: one from the item writers, one from the Panel, and one from the graders. Through discussions with all constituents, Exam leadership works to reconcile differences, if any, in the three perspectives and to develop a consensus view standard.

Since graders have observed actual candidate responses to a question, their perspective is important in building the consensus view.  Where aggregate candidate responses are low relative to expectation, additional reviews are conducted to evaluate whether the question itself was flawed or could be interpreted in multiple ways.  The consensus standard for a question explicitly reflects such evaluations.

Once all papers have been graded and a pass mark established from the consensus standard, the Examination Committee reviews those papers closest to the proposed pass mark in their entirety.  In this review, they carefully evaluate the candidate scores by question for consistency in grading.  A final adjustment to the pass mark may result from this review.

At the end of the day, the Examination Committee has as its sole objective the implementation of an objective, content-based approach to assessing whether candidates have demonstrated the qualifications outlined in the learning objectives. It does not apply a “quota measure” in determining the pass mark.  If 100% of the candidates sitting for an examination demonstrate the expected knowledge, then 100% of the candidates pass the examination; if only 20% of the candidates sitting for an examination demonstrate the expected knowledge, then 20% of the candidates pass the examination.  Typically, however, the expected knowledge encompasses 67 to 74 percent of the available points on an exam and the application of our standards has resulted in an effective pass ratio of 30 to 45 percent.

Given the preliminary results for the 2011 Fall exams, the Examination Committee undertook additional analyses beyond the reviews outlined above to ensure that consistent standards were being applied and that the passing grade was appropriate.

The Examination Committee also invested considerable energy in exploring possible changes to the demographics of the candidate population that might explain the “outlier” results.

The Committee noted that, for Exams 6-U.S. and 6-Canada, there was a higher proportion of the candidates who were “first-time sitters” for a CAS upper-level exam; they observed that candidates at this stage of their careers may need help in preparing for the transition from multiple-choice questions to the essay-type format associated with the CAS upper-level examinations. They noted that the increasing diversity of our candidate population, including many for whom English is not the native language, may create additional challenges in an essay-type environment.   In response to these observations, I have asked the Vice President – Admissions to begin exploring ways to better support our candidates to successfully complete these examinations.  I expect that the Executive Council will act on her recommendations this year.

Exam 8 was the first offering of this newly designed three-hour examination.  While the exam material was largely derived from the old Exam 9, the shorter exam time, the reduction in the number of questions, and the decision to not test all syllabus materials on each exam may have affected candidate performance. The Committee noted that candidates fared more poorly on questions involving new syllabus materials and on materials that were tested with questions that were asked differently than in prior exams.

Next Steps

Notwithstanding the efforts of the Examination Committee to ensure that its processes are fair and consistently applied, I was surprised and disappointed with the results of the Fall 2011 examinations.

In my role as president, I have had occasion to meet and interact with many CAS candidates and Associate members; I have been impressed with their capabilities and with their commitment to the casualty actuarial profession.  The results of the Fall 2011 examination are inconsistent with this observation.

Particularly for Exam 8, most candidates understand and are firmly entrenched in the CAS examination process; most are committed to doing whatever is necessary to successfully complete their examinations.  In my mind, the CAS needs to better understand why only 22% of the candidates sitting for the exam (23.9% effective pass ratio) met the requirements.

With the agreement of the Vice President – Admissions and the Chair of the Examination Committee, I have directed CAS staff to engage a professional education consulting firm to perform an independent review of our process.  A 2001 audit report of the CAS examination process and procedures led to the implementation of the following: (1) development and publication of the learning objectives for each exam syllabus, (2) training of exam item writers, and (3) a content-based pass mark process.  The objective of this review will be to evaluate our current processes against best practices for adult professional education. Upon receipt of the consultant’s report, the Executive Council will consider the consultant’s observations and recommended improvements for implementation as soon as practical thereafter.

I believe that all of the Admissions committees, from Syllabus to Examinations, are committed to executing high-quality and appropriate standards for membership.  However, we recognize that any process can be improved.  As actuaries, we are committed to a standard of excellence that requires us to reach beyond ourselves to explore best practices and to embrace changes that will further enhance our profession (credit rush at https://www.dresshead.com).  It is with that objective in mind that the external review of our current education and examination processes is being initiated.

We encourage you to leave a comment on how the CAS might improve its education and examination processes.  I look forward to your thoughts and suggestions.

Updated February 23, 2012:
CAS to Make Fall 2011 Exam Responses Available to Identified Candidates

avatar

About Pat Teufel

Patricia A. Teufel was the 2011-12 President of the CAS.

36 Responses to An Open Letter Regarding the Fall 2011 CAS Exam Results

  1. avatar Kelly Wargo, FCAS says:

    I appreciate your letter and efforts to improve the process, Pat. Two thoughts have been running through my minds on this topic.
    1) Maybe we should give the students a few examples of what “full credit” looks like on typical questions on the exam. It feels like the bar is moving up but are we helping students understand how to better prepare?
    2) I understand that there has been an effort to increase the level of questions from memorization to higher level understanding and applying concepts on the syllabus. Higher level questions require more thought and more time to analyze. I wonder if we have overlooked the time component.
    I just hope we haven’t frustrated potentially-awesome students to the point where they have given up on CAS exams…

    • avatar Pat Teufel says:

      Kelly: Thanks for your comments. In addition to an analysis of their papers, the CAS posts sample responses to all exam questions. With respect to the effort to move to a higher level question on the exams, the Fall exams were not affected. Those changes are scheduled to begin with the 2012 exams. The Admissions committees are working diligently to ensure that, as we move to the higher level testing, we will have adequately prepared item writers, graders and Exam committee leadership — as well as candidates.

      • avatar Michelle Cui says:

        Kelly, thank you for asking these critical questions. And Pat, thank you for the reply. My question, as a follow up, to the above Q&A is this: is CAS going to further candidates’ understanding on the new Bloom’s Taxonomy by offering additional guidance?

        The two short articles published on this subject by CAS so far are brief. In my opinion, both has not adequately addressed what candidates shall expect from 2012 and future exams under the new direction, and more importantly how they shall better prepare themselves for the change. While it is helpful to know that CAS is diligently working on a smooth transition into the new approach, it would be more helpful for candidates to receive adequate communication and guidance on the changes that are coming.

  2. avatar Kirk Leesman says:

    Pat,

    First, I would like to thank you for taking the time to write a response to the Fall 2011 results and the efforts that are being made to review the exam process.

    What struck me when reading through “The Current Process” was that the MQC is developed to a great extent individualy per question, but in what you describe I don’t see that the MQC is ever reviewed when looking at the questions in aggregate. As a near Exam US 6 passer who felt prepared for the exam and usually does not have a major issue with time, I felt time pressured throughout my sitting . This certainly affected my ability to answer the questions individually. I know of several canadites that if not having to leave questions blank all together, had to write less or had less time to review their answers than they would have liked. Would this not bring down the total scores in aggregate, but not be reflected by in the individual MQCs for each question under “The Current Process”?

    Kirk

    • avatar Pat Teufel says:

      Kirk: Your observation is an accurate one (to my understanding). While the process focuses on the individual questions, the graders have the chance to weigh in once they observe the individual candidate responses for a questions. They often adjust the MQC for the question, based on these candidate responses. Near the end of the process, the Exam Committee does review the candidate papers in their entirety; we perhaps should give greater weight to the “top down” view.

  3. avatar christopher burgess says:

    Thank you for listening to the candidates concerns and not dismissing/blaming us.
    Some suggestions maybe for the consultants:
    1. Let candidates get their answers back. Or even allow a request for 2 or 3 questions as long as they waive the right to challenge them. This would really help students learn what they are writing in the moment and grow from it, especially now that answers aren’t straightforward or lists found in the source material so you have to create an answer that doesn’t exist anywhere.

    2. Please include Exams 6-U.S in the review if it wasn’t. The increased number of new candidates being an explanation does not seem supported by the statistics.

    3. When looking at Exam 6U, unlike exam 8, look at the impact that some of the easiest material was removed and got replaced with a significant amount of material which seems to be ignored in explanations. The new MCQ appears to have been set at a score higher than most FCAS/ACAS scored when they passed previously. MQC set approx. above historical 75 percentile so most prior sittings would have had similar pass % in the 20’s with that pass mark. And this doesn’t take into account how previously lists/memorization was enough where now there is almost as much memorizations still and you need to not only remember it but combine it with multiple concepts in order to answer questions.

    • avatar Pat Teufel says:

      Thanks for taking the time to write. The audit of the examination process will apply to all CAS upper-level exams (as the same process is used for all). Your points about time pressures for the exams are relevant and we will definitely consider your views in our deliberations.

  4. Pingback: CAS seeks new review of exam process « Actuarial Opinions

  5. avatar Steve Winstead, ACAS says:

    Pat, thanks so much for the letter and the willingness of the CAS to reexamine its process.

    I feel that, for an examination system to be most effective, a failing candidate should be given some sort of feedback or at least have an adequate resource to identify how they can improve their score for the next sitting. Those who simply say “study harder” are essentially shrugging off the efforts of those who genuinely want to do better. While the score analysis sheet sent with a failing grade report is not particularly useful, I at least understood why I failed Exam 9 in the Spring last year, and I intend to modify my approach to achieve better results this next sitting. For Exam 8, however, I could not have been more prepared. The meaningless score analysis only served to add to my frustration when I compared it to the sample answers and asked myself “that looks a lot like what I wrote, why didn’t I get full credit?”. It will be a challenge this Fall sitting to do better, because just “studying harder” clearly isn’t going to work.

    I know many people share my opinion, and it seems that confidence in the exam system is diminishing, even among those without a vested interest in the most recent results. I have little doubt that there were well-qualified candidates for the Fall 2011 exams who received failing grades. I am confident however that the CAS will take appropriate measures to improve the process, and its openness to doing so is to be applauded.

    • avatar Rick Kam says:

      When you said “appropriate measures to improve the process”, does it mean 1) that the upcoming 2012 exams will serve candidates more fairly, or 2) that the “well-qualified candidates for the Fall 2011 exams who received failing grades” will actually get passing grades?

      • avatar Steve Winstead says:

        Although it would be nice, I certainly wouldn’t ask or expect the exam committee to change the pass mark. I know they’ve already spent considerable time and effort in establishing that metric, and as much as I disagree with the outcome, their position is defensible. I expect this to be a learning experience for all parties, and the process to be improved in the future.

        Of course, if the audit confirms that the pass mark truly should have been adjusted given all of the considerations, I would hope that the committee can retroactively acknowledge the success of those who “should have passed”.

        • avatar Alex Morrone says:

          While I fully understand the time that goes into creating that metric if after an analysis of the exam it is deemed that the passmark was set too high, even 1%, that it should be adjusted.

          The time and dedication that goes into each sitting is a huge sacrafice. To opt to do nothing and force a large group of CAS members to put in another 400 hours to attempt this exam again is extremely unfair to those people.

  6. avatar James Miller says:

    People have already mentioned the time component and the change in the type of questions being asked but I wanted to add to the discussion this simple example I came up with for my manager when he asked for my thoughts on this:
    ……
    Say there is an exam of 10 questions worth 10 points each. Each person who developed the exam questions says that the minimum qualified candidate (MQC) standard is 7 points out of the 10 on each of their questions. So the CAS would set the pass mark @ 7*10= 70 pts out of 100. However, that assumes that a MQC would have enough time to answer every part of every question. Let us assume that a MQC would in fact only be able to attempt 90% of the points on the exam in the limited time. That means a MQC would score .9*.7*100 = 63 points. Now after looking at the low pass rate using the 70 pt pass mark, the exam committee may lower the pass mark to some number below 70pts, but I would argue they may not lower it enough. Let us say they lower it to 68 pts. That still leaves the qualified candidates who scored 63-67.75 pts with a 5 instead of a pass.
    Or to put it in actuarial terms, they seem to set the pass mark based solely on severity without any acknowledgement of frequency beyond some tweaking of the pass mark after grading is complete (or almost complete). Of course they may have assumed that the “frequency” was that a MQC would get to attempt every point on the exam. This may have been the case in the past, but now that they have eliminated multiple choice from the upper level exams as well as moving away from questions that ask for simply regurgitating lists of information towards more open-ended and integrated questions, I don’t think this is a reasonable assumption. And don’t get me wrong, I think the newer more open-ended and integrated questions are a better test of a candidate’s knowledge of the material and should be used; but then the pass mark also needs to be adjusted or the time allotted for taking the exam needs to be increased.
    ….

    Is there a way someone could go back and calculate the average percentage of points left blank on the essay portions of exams over the last 5-10 years and see if there is a steady trend upwards over the last several sittings and a possible jump for this fall sitting? Any other thoughts on other means to measure if time pressure has increased over the recent sittings and especially in Fall 2011?

  7. avatar Jeremy Fogg says:

    Thanks for your response, Ms. Teufel. I appreciate your openess on this topic and willingness to receive feedback. I hope my feedback below is constructive and at least provide discussion points for the independent review. They are not intended to be critical of those who invest their time and energy to ensure the integrity and quality of the Society and the profession.

    I am in a unique position as a non-traditional actuarial student; so my apologies in advance for the lengthy presentation, but I believe that each component presented is related and important.

    My background includes extensive graduate work in educational psychology and curriculum & instruction in the area of mathematics education. I am well versed in Bloom’s Taxonomy (in both its classical and revised versions) and have done research in the area of developing critical thinking skills (including problem solving) through curriculum and evaluation.

    In the description of the development of the MQC, it appears that the focus is on what knowledge a candidate would possess, but fails to take into consideration the time that it may take to present that knowledge. It appears that the expectation of the MQC is the ability to present the requisite knowledge in roughly the same time as a well-seasoned actuary. But this time gap will grow tremendously once you start assessing a candidate’s knowledge base with the higher levels of Bloom’s Taxonomy. Research has indicated that the best way to assess the higher levels of Bloom’s Taxonomy is through projects and open-ended assignments–reflecting the need for a considerable amount of time by the student–and not through timed-exams. In situations that require some form of timed examination, the exam should be focused solely on the specific task at the higher level being assessed. I think replicating SOA’s use of Modules could be utilized to both develop and assess these higher levels of Bloom’s Taxonomy. Under the current process, only the assessment is done with no consideration on how to develop them. (And for what it’s worth, doing a project and/or open-ended assignment does develop those higher order processes when the individual is working to do just that.)

    I think one of the biggest issues that creating this gap between students and the CAS leadership is an issue of trust. While many candidates are expressing distrust of the leadership with regards to the evaluation of the Exams (I do not believe that there’s any complaints about the questions themselves in most cases), there is an implicit expression of distrust expressed by the CAS towards the students. I’m not sure why students are not allowed to get their submitted answers back. I’m not sure why the grading rubric itself is not released for the Exams. Sample student answers–which may contain errors–do not inform a student as to what exactly is important in providing a response that gains full credit (or even maximizes partial credit). It provides no feedback on what a non-passing candidate needs to do to improve–especially if they do not have their own submitted answer from which to compare. It seems to me that the reason that neither of these actions are done today (or in the past) is that the CAS leadership does not trust the candidate pool to use these valuable sources of information in appropriate ways to improve their ability to communicate their knowledge to the CAS. Given that the proverbial wheel appears to be reinvented each sitting (that is, new problems are constructed each time), I do not see what is lost by releasing the full grading rubric occassionally for each Exam being offered (and student responses on those sittings where the rubric isn’t released). Then students will have a better understanding of how best to prepare for the Exams, but this does require that the CAS will trust students to learn the requisite knowledge and not just mindlessly “regurgitate” material. (I do believe that the work force will weed out this latter group, if they even exist.)

    Finally, I’m not sure what will actually be gained by an independent review by an educational consulting firm unless the individual(s) doing the review is well versed in both the educational theory and actuarial processes (including the additional topics required by the Exams, like the accounting material on Exam 6). I do not believe that the crux of this issue lies in the methodology or in its mechanical implementation. I believe it lies in the judgment that is exercised when the subject matter experts (the item writers, the MQC panel, etc.) are developing the measuring stick that does not account for all relevant variables appropriately. An educational expert will likely not have an idea of how long would it take an actuarial student to address a particular topic given the time that an expert (one removed from the examination process) would be able to address the problem. To put this another way, how many of the CAS membership who have been away from the exam process for more than 5 years (excluding those who develop study materials for an Exam) would be able to measure up the established MQC for a randomly assigned Exam under the same conditions of the students sitting for that Exam?

    Again, thank you for your consideration of our input to the overall process.

  8. avatar Pat Teufel says:

    Thanks for your thoughtful comments, Jeremy. Implementation of the higher level Bloom’s taxonomy on the examinations is not scheduled to begin until this year. I very much appreciate your thoughts on leading edge educational processes.

  9. avatar Howard Mahler says:

    Unfortunately, these things happen.
    Some of the solution is just continuing the hard work of the Exam Committee, with clear communication of the goals.

    Emphasize the review of the overall length and difficulty of the exam.
    (These can be two separate things.)
    I do not know if you still have a single person, called in my day the consultant for that particular exam although they were another CAS volunteer, who looks over the entire draft exam for overall length and difficulty.
    (This was someone separate from the Vice-Chairman and Chairman of the whole Exam Committee.)

    I think a goal should be that the students have enough time to demonstrate everything they know. Time pressure should not be an issue for most students on most exams.

    I put partial blame on the obsession of the CAS Board with the overall total hours of exams and the number of exams.

    For any given set of syllabus readings there is a range of reasonable lengths of exams. However, that can be stretched beyond the breaking point. Were we trying to test 4 or more hours of syllabus material on a 3 hour exam? If so the exam committee is stuck between a rock and a hard place.

    I think you need to carefully review all the exams and in some cases trim the syllabus and/or increase the hours.

    Finally, you should review (again) the Institute of Actuaries Exam Process, if you have not done so recently, to see if there are any ideas we can adapt to improve our process.

  10. avatar Peng Lee says:

    Pat, thank you for the time you’ve spent to address the results of this past sitting.

    Unless I am mistaken, hasn’t the CAS has already implemented the higher levels of Bloom’s Taxonomy into the exams?

    (December 2011 Future Fellows)
    http://www.casact.org/newsletter/index.cfm?fa=viewart&id=6276

    • avatar Jeremy Fogg says:

      There is a difference between asking a different type of question within a given level of Bloom’s Taxonomy (namely, asking questions where more than one answer/method can be applied) and asking questions at a higher level.

      I think the statement that Ms. Teufel made is that the proportion of questions at a given level have not changed in recent sittings; but this will change with the 2012 sittings, especially for the Fellowship Exams.

  11. avatar Ron Bertrand says:

    Christopher Burgess’s point about releasing candidates’ responses back to them is a valuable one. Even a few questions would be a tremendous help. Students need a better understanding of why they are losing credit on their responses; otherwise it is very difficult to improve. Are they not writing enough, making careless errors, writing illegibly, not fully understanding the material? Too many students continue to fail exams because they continue making the same mistakes and don’t realize it.

    • avatar Lilly Li says:

      Can’t agree more. Releasing candidates’ responses quesitons, Even a few, back to them is extremely valuable. So that candidates would not repeat their mistakes.

  12. avatar Kostis Poulogiannis says:

    Dear Ms Teufel,

    First of all I believe that actuaries must be creative problem solvers, so any attempt to change your examinations from memorizing to higher level understanding is a good start, after all in real life this will be asked from us. It takes some time for a canditate to adapt to the upper level exams compared to the early exams. This move could be more “smooth” if also the early exams had (to the degree that is feasible) the same philosophy. Also testing the upper level exams twice a year compared to one would probably reduce the travel time.
    Second I would like to share my thoughts from a non US based student.
    You write, correct, that a student that english is not his native language has more hard time with the upper level exams. This in fact is more true regarding exam 6 which is nation specific. For a person that probably will never work in USA/Canada but wants to earn the CAS qualification exam 6 is a big drawback.
    I really think that a nation specific exam must be applicable to someone that wants to work as an actuary in USA/Canada. The fundamental actuarial knowledge is the same in every part of the world and the exams must focus on this. On the other hand each country has its own laws and regulations (that an actuary must be familiar with) and having a US/Canada specific exam for the ACAS qualification puts a foreing canditate in distress .For instance the UK qualification for P&C field (and in fact for any field) doesn’t require a nation specific exam and only if someone wants to work as an actuary in the UK must pass a nation specific exam.
    I really believe that if CAS exams for the associate level didn’t have this nation specific exam (you could replace it with some other) then CAS would rapidly increase its non US members.

  13. avatar Gareth Kennedy says:

    Pat

    Thank you for your openness with candidates and taking the time to seek our feedback. In the continued interest of openness with candidates, will the CAS publish the independent consultant’s report, or at least the executive summary with the key findings and recommendations? Will the CAS engage a different independent consultant from that used in the 2001 that recommended the content driven pass mark?

    Some simple internet research indicates that the Angoff method for setting pass marks (the basis of the MQC approach) has limitations. Our own Statement of Principles for loss reserving indicates we should ordinarily “examine the indications of more than one method”. Should the CAS apply the same principle when setting a pass mark and have multiple methods with a final selection based on the judgment and experience of the exam committee?

  14. avatar Richard Newell, ACAS says:

    Historically the actuarial exams have always seemed like a race against the clock – for me anyway. How many times have we left an exam knowing we could have completed a few more questions with another ten or thirty minutes? I know people that believe they could have passed the exam last year with a bit more time, and that the other replies are on to something regarding exam length and not factoring the time element needed to present a complete set of MQC-worthy responses.

    Does the exam committee sufficiently account for this element? My knowledge of the exam committee responsibilities is limited, but perhaps exam writers and/or graders should be assigned a limited portion of the syllabus (i.e. from Exam 8 they would be very familiar with all of section A. Portfolio Theory and Equilibrium in Capital Markets from the syllabus) and then assess their own performance on this section before the exam on the questions they did not write. How long did the questions take you to answer? How did you score? Even in this environment answering an hour or less of questions or less may not be representative of the pressure involved with being MQC-familiar with the whole syllabus.

  15. avatar Andrew Brown says:

    I would first like to ask all of the fellows to state black-scholes formula. Now everyone who was unable to regurgitate this information should retake exam 3F because they are no longer capable of passing the exam. This is only one example of irrelavent memorization that takes priority over learning how and when to use the formula or method. It would be easy to make a list of thousands of formulas and methods that need to be memorized. Before the invention of books this would have been important. However, books have been around for a long time and we now have a better tool for finding this information. Has anyone used wikipedia? In order to fix this problem we need to focus on learning the general principles and putting them into practice even in unique situations. Quite often the standard formulas and methods used in actuarial science fail to capture reality, but if actuaries really understood the general principles they could modify existing methods to get more accurate results.

    Another problem is each exam is completely independent of most of the other exams. There has been very little emphasis on learning to use the methods of the preliminary exams to solve the actuarial problems on the upper exams. My suggestion would be to start the process by defining risk management and insurance and then quickly moving into basic ratemaking, reserving, accounting, asop(in the form of an online course), Etc. Following this we could start learning to apply financial, statistical, mathematical, and computer science techniques to these important insurance problems.

    The biggest weakness of skills I see among actuaries is computer programming skills. These skills are necessary to build the models that the most profitable insurance companies are using. I would strongly advise adding a VEE in computer programming.

    In summary, remove irrelevant memorization and outdated material from the exams in order to make room for deep understanding of general principles. Teach connections between the material on the exams. Introduce insurance first so students can have the opportunity to learn to apply the technical skills to insurance. Reading and writing speed should not be tested on the exam, rather put challenging questions on the exam and give plenty of time to solve them. Finally, add some training in computer programming.

    I could write a large book on the problems with the actuarial exams. However, I would enjoy helping to redesign the education process if the CAS is really interested in fixing it.

  16. avatar Michael Woods says:

    Thanks for the letter Patricia. I am happy to see that the CAS is evaluating the exam process. There were certainly a few factors which could have skewed the exam results for Fall 2011. As you mentioned, one of these factors was that the Exam 6U candidate pool included a number of first-time upper-level exam takers. As a recent entry into the upper-level candidate pool, I can attest that upper-level exams are indeed a different beast. I would be interested in how the results look after removing the effect of the “first timer” students. This 6U sitting was unique since the exam had not been administered for one-and-half years due to the exam transition. Because of this, there were an unusually large number of candidates taking the exam and there were many different paths a student could have taken to end up in this sitting. From my estimations though, there were four main groups of students taking the 6U exam:

    Group 1: Students that had previous sat for 7U (in Spring 2010) but did not pass. These students had probably passed old exam 5, and had taken a few upper levels between sitting for 7U and the new 6U.
    Group 2: Students that finished their preliminary exams in Spring 2010 and began taking upper level exams in Fall 2010. Therefore they had taken two upper level exams before.
    Group 3: Students that had finished their preliminary exams in Fall 2010 and began taking upper level exams in Spring 2011. They had taken one upper level exam before.
    Group 4: Students that had finished their preliminary exams in Spring 2011 and took 6U as their first upper level exam.

    Assuming that each group was about equal in size, this would equate to “first timers” accounting for about ~25% of the candidate pool for 6U. Using this estimate to remove “first timers”, while also assuming that all “first timers” failed, we would get a 6U pass ratio of 194 / (1008-.25*1008) = 25.7%. This leads me to believe that the pass ratio might still be abnormally low even if we take into account the “first timer” effect. Obviously this estimation is not exact, since I had to make some reaching assumptions to perform this calculation, but perhaps the CAS could look into this further and perform a calculation of this sort using actual data to determine how much “first timers” actually did skew the results.

    To improve the CAS education process, I believe two initiatives are in order. First, the MQC standard needs to be reevaluated and either improved or abolished. Secondly, I believe the transparency of the grading process needs to be increased so that candidates may have better expectations of how to answer exams questions to receive full credit.

    Since implementing the MQC standard, I believe there have been around ten sittings in total. Two of these sittings, Fall 2008 and Fall 2011, had abnormally low pass ratios. Combining all upper level exams, the pass ratios for these sittings were 30.3% and 20.6%. Using the exam history available on the CAS website that goes back to 1990, there does not appear to be a sitting which compares to either of these sittings. In addition, these abnormal sittings were preceded by sittings that had normal pass ratios. The sitting before Fall 2008 had a pass ratio of 46.3% and the one before Fall 2011 was 40.3%. This means that the drop in pass ratios between sittings was 16.0% and 19.7%. I bring this up because due to the fact that students spend years in the exam system, this means that the student pool from sitting-to-sitting is relatively stable, with something around 70%-90% of the same students sitting for the exams each time. It is hard to believe that the same students, taking similar exams, and with proven track records of passing exams in the past, could have such wildly varying results from sitting-to-sitting. This leads me to believe that the grading process or pass mark setting approach has changed more than the students. In addition, the absence of a sitting with an abnormally high pass ratio leads me to believe that the MQC standard may be biased low.

    In addition to an evaluation of the exam grading process, I do believe that there could be improvements on the student side too if communication by the exam committee was more robust. There appears to be a lot of students receiving failing reports with low scores on questions which they believed they aced, with many students believing they put an answer exactly as the answer key purported the correct answer to be. Of course, students can sometimes be selective in their memories, and believe they put a better answer than they actually did. So I’m sure this phenomenon explains part of it. But there still does appear to be confusion among students about what type of answer receives full credit. There have already been a number of articles that have been helpful in helping students take CAS exams more successfully, such as the release of candidate papers in 2008, the importance of adverbs article, and the Bloom’s taxonomy article. However I think there are still future articles which could be written to help illuminate the grading process further and help students understand how to answer a question for full credit. Many students receive failing reports and learn that they received 100% credit on only a few (or zero) questions, even questions they believed were simple, so it seems there might be a gap between what is expected by the exam committee in answers and what the students believe is a good answer.

    Learning how to write an answer in a way that receives full credit wouldn’t normally be such a quandary on most exams, but it is for CAS exams due to the immense time constraints that students face. We have to constantly strike a balance between being thorough and being quick. Writing one extra sentence on each question could mean not finishing in time, and even by practicing conciseness, many students are still unable to finish. So an article that could inform students on the types of answers that are required to receive full credit would be beneficial since we would be more able to demonstrate our knowledge to the exam committee in the short time allotted on upper level exams.

    An example of an article that I would like to see is one where a few types of questions are picked from an exam and each question represents a different question type. By question type, I mean a computationally-intense question, a Bloom “evaluation” question, a Bloom “synthesis” question, etc. Then for each question type, show an answer that would get 0.5 points credit, 1.0 point credit, 1.5 points credit, etc… The author could then describe the reasons for only getting partial credit and reasons for getting full credit. Perhaps important omissions and errors could be pointed out. For example, the computational question may require more work to be shown, with most students not showing enough calculations written out to receive full credit. Or a synthesis question may require more reasoning in its answer than regurgitation from a particular article.

    I suggest this type of article because one of the most informative comments I read about the exam grading process was a similar type of “answer critique” written by Arlie Proctor on the Actuarial Outpost web forum. On the forum, a student had complained about receiving low credit on a question, even though he knew had had arrived at the correct numerical answer. Arlie was kind enough to look into the matter and with the student’s permission, posted his answer online and described why he did not receive full credit. As it turned out, he did not show enough calculations. The student did write out entire triangles of numbers, but failed to write out an equation for how the numbers in the triangles were derived. Therefore much credit was taken off. Since reading this, I have been sure to answer my questions in a different way, showing a sample calculation for each step along the way. I wonder though, if other students are aware of such nuances. Computational questions will often say “show all work”, but a student’s definition of “show all work” and the exam committee’s definition of “show all work” can be different, so it was nice to see this clarification. I believe more clarifications of this type could help students better demonstrate their knowledge on an exam. Unfortunately, since students are not allowed to see their answer sheets, this type of communication is lost and many failing students are stuck studying for the next exam without any guidance in how to improve for the next sitting.

    • avatar Benjamin Singhurst says:

      Thank you for that example in defining what “show all work” means, Michael. I agree that it would put candidates on a far more level playing field if we were given even the faintest glimpse into how we are being evaluated. Further, of the many different model solutions I have read, many seem inconsistent with what is required to be shown in order to receive full credit. If this relates to the number of total possible points for a given question, then that should be clarified up-front.

  17. avatar Andrew Wu says:

    Hi Patricia,

    First, I really appreciate your letter to explain last year’s CAS exam performance. To be a candidate seeking the membership in such a successful society, I have been a bit upset (both to myself and the society). I have switched my credential from SOA to CAS, just because I really like the traditional P&C work relatively to traditional life work, but it definitely doesn’t mean I like the CAS exam system, it’s complete opposite.

    I never blame the exam result or system before like this time. Any hard exam paper will have some people pass and any easy exam paper will still have people fail. I keep all the time to question myself why I can’t be the one to on the passed list. From recent year CAS initiating the exam structure update, I can see CAS want to reduce the travel time for the candidates which is such a great improvement to us. But it’s not the truth from we can see in 2011. When 2008 fall sitting the passing ratio was lower than everyone’s expectation, CAS started to release some candidates’ actual exam papers and write an article to explain the reason. How about this year? Both the exam committee and president write the article to give some explanations.

    After I read the exam committee’s paper, I feel really sad. Honestly, from the 1st day when I study for CAS exam and doing all the past exam preparations, I can’t say the exam is hard, CAS exam is not Olympic style competition, most type of questions are tested on our comprehensive knowledge in the P&C field, how proficient we are in gaining the P/C industry knowledge. But when I saw the statement “With respect to these standards, the 2011 exam was determined to be relatively easier than 2010, and so the pass mark was higher in an effort to achieve a common standard, thus resulting in a disparity in pass ratios between 2010 and 2011 greater than these 9 pp.”, I believe at least for the candidates who wrote the exam 9 and 8 in 2011 should be similar population as before because most of them are not writing their 1st written answer based exam (compare the multiple choice style), I trust the student quality (talent vs “bad”) are at the same level, but look at the top level people, the ratio dropped from 11% to 6% or 75 to 25, do you still think it’s a relatively easy paper? If the passing score is set at 70% or below, I think lots people agree it’s an easy one, but when lots of new style questions in one exam sitting paper, do you think you should carefully adjust your comment (I didn’t mean score)?

    The most important part I want to discuss here is about the whole appeal process. Right now from both letters, I do find that CAS is very confident with their grade processing and to be more transparent to every candidate, I think there should some improvement in the CAS appealing process as well. Under the current policy, I think there is almost no way to successfully submit an appeal. I remember in my previous exam sitting, every time when I got a 5 and after I carefully checked the posted solution, I want to do appeal right away. I never got a score below 5 and every time my 5 is very close to a passing score, I used the minimum score I got from CAS to calculate my minimum mark, it’s always very close. The concern is I do have some questions (with big mark), in my own memory I am very sure I got 100% correct, but turns out to be 50%-74% or sometimes even 0%, I feel so mad about it and want to do the appeal immediately, but after I read the CAS policy, I gave up my thoughts, it’s a mission impossible. They ask us to provide the alternative solution from other article which can also show your solution is correct, without this type of proof, all the appeal will be ignore, my god, it’s really a mission impossible. Personally, I don’t know how other people find the way, but for my opinion it’s impossible, why? Every time after exam once CAS posts the exam questions online, I will do a self-mark and I check with some actuarial post people with the solutions every others got. I think most of the questions has only one solution or most candidate do the same way, there is no alternative way to do it, and my solution is exact the same as CAS posted solution (at least in my mind), so under this type scenario, how can I do? If the question I think I am very confident to get 100%, but turns out to be 75%-99%, I can accept, but for 50-74% or even 0%, it’s unaccepted. Can I easily on the appeal letter said please check my question #? I believe from the CAS website, it mentions that there are 2 ways of appealing, one is for admin check and the other is for alternative solution. So go back my case, I don’t want to always blame on the admin check which should be very small probability to occur. For the 2nd way, I can’t provide the alternative solution as my solution is same as CAS. The best would be for us to get back our answer sheet (copied version) for those selected questions we feel it’s mis-scored. But it will be mission impossible for CAS to release those to us. 2008’s actual candidate paper posted on CAS website was a big surprise to me. To be more transparent to CAS grading process, why CAS can not release information belonging to candidate? If we can get back the question answers we doubt, it would be lots better to prepare for the next sitting, turns out to know our true mistake and better understanding CAS grader performance.

    Thanks for reading my long and not good writing in a rush mood letter, hope you can understand.

  18. avatar Khiem Ngo, F.S.A says:

    I am encouraged that you are looking into ways to improve the examination process. The exams are meant to pass qualified candidates. It’s hard for me to believe that the quality between candidates, especially for the upper level exams, can differ so much from year to year given the large number of exam takers.

  19. avatar Lilly Li says:

    Thanks Patricia for evaluating the exam process!

    I have one suggestion for improvement of the exam process. Can the exam problem writer work on the problems themselves before testing candidates? For example, Problem 35 (2011 Fall exam6U) did not give complete factors for the calculation and required candidates to make assumptions. Model answer just accepted one assumption and did not give credit to other assumptions. If the problem writer worked on the problem first, he/she should have found that omission and effectively reduced the confusion candidates might have during extremely stressful exam process.

  20. avatar Phillip Dlugosz says:

    Hi, I think it would be helpful if the exam committee released sample exam problems that cover new material, and that build in the new expectations regarding Bloom’s taxonomy.
    I think this would be especially helpful for the May 2012 exam 7 sitting.
    Thanks,

  21. avatar Jim Klann FCAS says:

    I believe we can all agree that the amount of study, and degree of knowledge, required to pass an exam should be similar from one sitting to the next. The question is, how do we achieve this?

    The Exam Committee appears to believe it can achieve it by an abstract, a priori assessment of question difficulty before candidates write the exam. This is impossible with material as complex as our syllabus. There needs to be more room for analysis after the exams are returned and graded.

    When candidates achieve lower than usual scores, the question will always be, and should always be, “Were candidates more poorly prepared, or was the exam more difficult?” If the former, the pass rate should go down; if the latter, it shouldn’t.

    No one will ever be able to answer this question definitively, but I believe we could at least attempt analysis. For example, one could compare scores achieved on similar questions, or parts of questions, versus prior exams. Yes, the EC tries to avoid repeating questions. But inevitably, at least a few questions are closely parallel to questions asked in prior years. If the score on a comparable question goes down, this is fairly strong evidence that the candidate pool is weaker. (Since candidates have the earlier question available in study materials.)

    Then, too, one could compare the overall scores of repeat test takers. This removes bias due to different candidate pools and differing numbers of new test takers. If the average score of repeat test takers declines, this is pretty strong evidence that the exam got more difficult, since one would not expect to see a sudden deterioration in the study habits of a large group of repeaters.

    In addition, one could surely analyze blank responses. Blank responses on easy questions, or on higher numbered questions (since many candidates work front to back) are an obvious indication of time pressure, which in turn is an indication of exam difficulty.

    I see no evidence in Ms. Teufel’s letter that the Exam Committee performed any of this analysis. If they did, they chose not to share the results with us. Instead we are offered assurances that the a priori assessment of question difficulty MUST be correct, so the candidate pool MUST have deteriorated, and we are offered speculation as to why this happened. As an analytical professional I am not convinced.

  22. avatar Hemanth Thota ACAS, MAAA says:

    Pat,

    I appreciate you taking the time to discuss this and look for ways to improve the process. Of all the things that bothered me about the exam committee’s letter published in January 2012, the biggest concern was ignoring the fact that adjustments had to be made before comparing the new Exam 8 (3 hour exam) with the 2010 version (4 hour exam). I’m quoting the relevant paragraph from Daniel Roth’s letter below –

    “The committee notes that length of exam will have an influence on these results. Clearly variations in exam length make it difficult to compare the results between groupings. However, the committee is not aware of significant differences in length within these groupings that would suggest the items noted above are not properly comparable. For example, the 2010 Exam 7US/2011 Exam 6US group consists of similar 100-point, 4-hour
    exams.”

    I might be missing something but I do not see anything here that talks about the shorter Exam 8 and the adjustments the committee made to their passmark setting process to account for this. I find it very surprising that an issue like this can be ignored by the exam committee (assuming that is what really happened).

  23. avatar Rex Goodman says:

    As a candidate who sat for my first upper-level exam this past sitting of 6U, I am pleased to hear that the exam frequency is being considered in the coming months. Not necessarily for the outcome of shorter travel time though.

    I strongly feel that the exam 6U syllabus inherently assumes prior knowledge of the exam 5 syllabus simply due to the fact that 6U is the old 7U. Until the past sitting of 6U, candidates had studied exam 5 material some 800 hours or so before sitting for the old 7U. Understanding the basics of ratemaking and reserving (Exam 5) seem to be an essential building block when trying to understand how insurance regulators view a company’s rates and reserves (Exam 6U).

    For this reason, I hope the CAS exam committee will strongly consider offering exam 5 twice per year.

  24. avatar Joseph Gravelle says:

    Pat,

    I am thankful of the steps that the exam committee has made to become more transparent, however there are many more steps the committee could take to improve its transparency. A short list would include the following:
    a) Releasing the definition of minimally qualified candidate that the Pass Mark Panel uses to determine the minimum standard for each question
    b) Releasing the minimum standard for each question
    c) Releasing the candidate’s actual raw score for each question
    d) Releasing the mean candidate score for each question
    e) Releasing the pass mark
    f) Releasing the grading rubric
    The first five of these items could be provided to students within a matter of days if not hours, as they all should be available electronically, and the grading rubric should only take an additional day to scan any hand written rubrics and compile a single pdf.

    It is my understanding that there are only two graders per question, and that these two graders are responsible for that questions rubric. I believe there could be great value in a larger panel (6+ members) reviewing the rubrics (earlier in the grading process), and random samples of partial credit scores from candidates near the pass mark (later in the grading process). This would help to ensure the underlying goal of “assessing whether candidates have demonstrated the qualifications outlined in the learning objectives” is met.

  25. avatar Peter Johnson says:

    Please consider my comments below in your development of the new and hopefully improved examination process.

    In short, I feel there needs to be a degree of consistency from one exam sitting to the next. I believe the following to be some of the key drivers of this inconsistency:
    1. Variation in exam length and the misconception of exam writers of what a reasonable length is;
    2. Variation in exam difficulty;
    3. The constant transition of exam committee members (and the skill levels of the exam writers/graders that transition in and out);
    4. Variation in the chosen focus of material from one exam sitting to another;
    5. Variation in the quality, length, and ambiguity of actual examination questions;
    6. Length of exam questions; (a reasonable question should be as brief as possible and shouldn’t have an entire page of text);
    7. The lack of understanding by candidates and organizations that produce study material in determining what is the exam writers feel is important. (The knowledge statements are more often than not, an insufficient guide in determining what a candidate should master – Richard Goldfarb may be able and willing to shed some light on this notion with regard to Exam 9)
    8. Changes in syllabus material.

    The number of variables in play makes it nearly impossible to fully understand where the appropriate pass mark should be. In addition, after considering the larger number of exam takers (i.e. the law of large numbers), it is difficult to imagine that the qualifications of the average candidate for a given exam change drastically from time to time.

    Specific to Exam 6 (US), the syllabus is massive and the number of possible list-type questions exceeds a 500, in my opinion. It would be very beneficial to come up with a way to point candidates in the correct direction, aside from using only broad knowledge statements. Also note that the removal of material from the old Exam 7 (new Exam 6) to the online Course 2 didn’t reduce the number of possible list-type questions due to the additional material added.

    Releasing exams of candidates who scored a 4 or 5 is a good start, but won’t fix the problem of inconsistency stated above. Please consider releasing exam answers immediately after candidates take an exam.

    Although it is extremely difficult to admit one’s mistake, everyone does understand that mistakes do happen and it seems the most appropriate thing to do is to make a correction to the pass marks for both Exams 6 (US) and 8. I understand that it has never been done, but in this case it only seems fair to make such an adjustment. With the previous 6 pass ratios for Exam 6 (US) being: 47.4%, 48.4%, 41.6%, 48.5%, 51.6%, and 46.6% it hardly seems feasible to accept a 20% pass ratio. Myself and the other exam takers I’ve spoken with had anticipated that the pass ratio would be bumped up after reading the first portion of your letter, but this clearly didn’t end up being the case.

    40% to 45% seems to be an acceptable passing level all things considered. The CAS is being unconscionable by keeping the mark so low. I won’t even explain the impact this choice has on that 15%+ portion of people that would have passed if the exams were more consistent with prior exams (based on the reasons listed Exam 6 (US) was certainly inconsistent). As mentioned above the average preparedness of candidates based on the historical information given to them should not change that much from sitting to sitting. You’ve been through the process, so I’m sure you can rationalize the effect this can have on and individual’s career and his or her personal life. You are running the risk of sabotaging the entire system and having highly qualified candidates switch their professional career track. Now you also need to consider the risk of people moving to the general casualty track being offered by the SOA.

    After a recent recruiting trip to a school in the Big Ten Conference, with a large and highly reputable actuarial program, a colleague of mine came back saying there was a reduced interest in the CAS track after seen what happened this last fall with the low pass marks. This is alarming and needs to be addressed.

    In addition you now have a back log of highly qualified candidates retaking the exam and competing with the next generation of exam takers. By not making an adjustment you have now created a undesired scenario that will take several years to work itself out.

    Going forward please consider setting a minimum pass level of 40% in addition to moving towards a more consistent exam from sitting to sitting. My rationale above should easily justify it.

Comments are closed.