So what do we know? Subject matter "experts" design the questions in a committee dynamic. Item analysis is most likely used to rate the efficiency with which each item discriminates between respondents in the highest and lowest percentiles of overall test performance. In other words, to the extent that the rate of a correct response for item #12 among respondents whose performance on the overall test places them in the highest third is significantly greater than the rate of a correct response for this item among respondents whose performance on the overall test places them in the lowest third, then item #12 must be "valid." In my opinion, this is a hair-splitting procedure that becomes necessary only to validate hair-splitting questions. If a question is obviously good, with one obviously true option and three obviously false options, then no item analysis is needed to validate the question. Why not rely on the sheer number of experts on the committee and their sheer level of expertise to determine whether a question is valid? (1) They fear that in designing a test to trip up respondents, they are also likely to trip themselves. (2) Since the expertise of each committee member is limited to a particular area, they are as green to the other areas as many of the respondents. So will a crash course in EPPP booklets really make up for a failure to acquire expertise in other domains? Of course not. The EPPP materials are reference material and have little if any educative value. The EPPP is not unlike the review guides we give to our Psych 101 students before we test them on their memory for the bold-faced terms in their textbooks. The only difference is that to avoid the appearance of a Psych 101 exam, ASPPB constructs items with a cosmetic semblance of power and precision. To avoid failing an inordinate number of psychologists, which would raise questions about the exam, someone provides commercially available study materials that by and large contain the answers. A thin veil and a hefty price tag separate the EPPP study materials from your average crib notes or -- GASP -- cheat sheet. I have to wonder about the nature of a strategic partnership between ASPPB (who manufacture the exam) and AATBS (who sell the study materials). What does AATBS know? Why would they allocate so much space in the study materials to Feminist Psychology?

Despite our meticulous and unscholarly efforts at standardization, the curriculum of most programs simply do not survey the minutae presented on the exam. I reviewed a number of the practice tests that accompanied the materials. (Incidentally, the items on the practice exams designed for the computer were noticably less difficult than the items on the practice tests in the booklets). While I am prohibited by copyright law from providing examples of the practice questions, one question required the test-taker and aspiring therapist to be intimately familiar with the genetic transmission of color-blindness. Like a resident borrowing a cup of sugar from a neighbor, Psychology co-opts substantial material from biological sciences to bolster its apparent legitimacy. One sex-related question concerning the effects of age on the time required for a man in his 60s to lose an erection following orgasm has relatively little clinical value. And many of the items include hedging or qualifying phrases like "while this research is mixed" or "while this research is inconclusive," indicating that test-takers are expected to identify as the best or correct answer the option that represents an inconclusive or contested finding.

I confess to being somewhat bemused that this exam should have any reliability. Having spoken with a few former test-takers, and having observed the progress of one test-taker over the course of multiple practice tests, I can tell you that the variation in test performance across trials is vast enough to warrant concern. Even when we look at the individual content areas, it is not uncommon to find that the percentage of items answered correctly would vary from 90 (practice test 1) to 59 (practice test 2) to 85 (practice test 3) to 62 (practice test 4). If you examined the percentages correct across content areas, you would swear these were for four different test-takers when in fact they were the same test-taker across four trials. There is no question that how one performs depends entirely on the crop of questions. The EPPP should convince any reasonable person that Psychology, while masquerading as a singular field of study, is really 10 or 12 distinct entities. EPPP test-takers bare the burden of managing this false impression to the world.

The observed test-taker offered an alternative theory that would explain a great deal of test construction. "Maybe the test is designed to measure commitment and motivation -- how willing you are to study hard for the exam. These are the people you want in this field. Maybe that's all they (test construction specialists) aimed to predict. Maybe it's the procedure." The role of the test would then be analogous to a rite of passage or initiation, something we can file away under the heading "There is a difference between knowing the path and walking the path." The test items may be poor. The knowledge represented may be of little value. The test-taker may be likely to forget everything he or she learned within four months of having crammed for the exam. But a passing score speaks volumes about that person's commitment to the goal of licensure. So many students get burned out or go limp after receiving their doctorates, and end up procrastinating their licensure preparations. (Unlike the field of medicine, where graduates have their licenses at the time of graduation, graduates of doctoral programs in clinical, school, and I/O psychology, who require licensure for employment, are not eligible to sit for their most comprehensive examination for months after their doctorate is conferred). And these graduates quickly learn the EPPP is not the kind of test for which you can study for two months, put the study materials down for two months, and then resume studying again. The facts the licensure candidates are expected to learn are guaranteed a level of attrition in memory. The loss from interference (new material pushing out old material) is so great a threat that one cannot risk losing remaining memories to decay (i.e., time). To make matters worse, most licensure candidates are working 20-40 hours a week while they are studying. In fact, graduates of clinical programs in Psychology are required to log in so many hours of SUPERVISED POSTdoctoral residency before they qualify for licensure. Many women split the available study time with the responsibilities of raising a child (okay, so perhaps I could forgive all the questions about a Feminist Psychology that makes it into the curriculum of so few programs).

For the procrastinators who stagger their test preparations across many months (and in some cases, years), the testing board is not shy about advising candidates that the study materials (those expensive books I mentioned earlier), which are revised annually, are "now outdated." Of course, my response to this would be a sarcastic "yeah, right," but they know perfectly well how to play on, and take advantage of, the anxieties associated with preparing for such an important exam. The role of this exam is roughly equivalent to that of tenure for university professors. You worked hard to get where you are. You made many sacrifices. You've incurred and mounted expenses across various categories -- financial, physical, psychological. But nothing you've accomplished previously -- not your 4.0 GPA -- not your 1600 GRE scores -- not your wildly popular dissertation -- your publications and speeches -- glowing evaluations from your supervisors -- none of this means anything if you do not pass this exam. You know that. So does the psychology board. And that is why you will gladly pay through the nose for a new set of preparation guides once the year is up. It is also why some candidates even pay to attend licensing exam preparation workshops. It's absurd. None of the material on the exam should be treated like insider "club" information privy to those for a fee. Unfortunately, much of it is. Much of it is material you are unlikely to have ever encountered in your coursework or practical experience. Much of it is trivia, fine technical data strewn across an unmanageable range of sub-fields and fiefdoms. The psychology board wants to know just how badly you want to be a member of the club.

Sure, as students, these test-takers were required to rote-memorize for course exams statistics and findings they could care less to remember let alone have the capacity to retain, but each course presents its own unique minutae, and the probability that the minutae to which any one test-taker was exposed over the course of the past decade will be the minutae he or she is expected to master for his or her moment of truth (and I use the term 'truth' here in the non-factual sense of the word) is low-to-negligible (leaving post-doctoral psychologists with an unenviable task of cramming for yet one more final exam). The EPPP study regimen is less a review of forgotten material than a crash course in numbers and facts they've never seen before. It is less a director's cut of your psychology curriculum than a Psychology edition Trivial Pursuit. The EPPP is the last chance to weed out any divergent thinkers and any imposters who may have conned and deceived their way to a 4.0 GPA, to an exemplary performance on an inhouse doctoral competency exam, through two practica and an internship. In the process of protecting the public (actually, more like their own culture) from the free-thinkers and imitators (the disingenuous of the conformists), the EPPP unwittingly trips up competent and knowledgeable psychologists and keeps them from becoming licensed therapists, and the EPPP study materials is the mechanism needed to save them. (Each state imposes its own pass-fail standard for this national exam. For example, the cut-off in the state of Virginia is 70% [140 of 200]. This means that the State of Virginia will allow you to miss 60 of the 200 questions! If this isn't a glaring admission of the failures of this exam, I don't know what is). As if the test isn't sufficiently exhausting, the EPPP throws in an additional 20 experimental items to be evaluated for use in future versions of the test. The 20 items are randomly mixed into the item pool, and the test-takers are unable to distinguish between the 200 items that will count toward their score and the 20 items that will not.

So item analysis is used not to create a test of valid items, but to create a test of tenable hair-splitters. The item analysis identifies those items that efficiently discriminate those who think alike from those who do not (who were seduced by another foil). (Again, this is all well and good if we are talking about questions where one option is true and the others false. But where we require test-takers to split hairs outside their area of expertise, we are seeking only those who think like we do or who think with one mind. The EPPP is another in a series of mechanisms in the course of professional training and evaluation used to weed out those who think a little differently. At the nexus of these mechanisms and their regulating agencies is a culture of conformity. The item validation process tends to certify as "valid" those questions on which the community of psychologists converge. With so many "best possible answer" questions, the EPPP is as much, if not more, a test of convergent thinking (i.e., like-mindedness) than competency. I guarantee that the average test-taker will not be able to, in significantly greater than 25% of the cases, the correct answer from the incorrect answer 2 years after answering these same questions correctly. The test measures how well one was able to cram hundreds of pages of useless information into their memories within a relatively short period before the date of examination.