Metaphor from film The Matrix Metaphor

I had only dimly apprehended the relevance of the film for my work when I received an amusing e-mail from a facetious visitor to my web site:

This is your last chance. After this there is no turning back. You take the blue pill, the story ends, you wake up in your bed and believe whatever you want to believe. You take the red pill, and I show you how deep the crap goes...Remember, all I'm offering is the truth, nothing more..."

Why do I find the Matrix metaphor so poignantly appropriate? Because the rules that govern psychology's academic and professional communities mass-produce, disseminate, and apply our knowledge of the human condition. Psychology presents itself as the mirror we hold up to our faces and it presumes to instruct us as to how we should view our individual selves, our society, and how we should make sense of our experiences. When you understand the rules that govern their brand of science, you will ultimately view it in the same vein as the protagonists in The Matrix viewed the programmed architecture and design of that prison that presents us with an illusion of freedom, of reality, of worth. These sentiments were reflected in my response to the tongue-in-cheek assessment of my web site:

"Ah, yes. Everything I need to know I learned from The Matrix, right? It is difficult for me to guage your real feelings and motives. You have to understand, most of these people are not ready to be unplugged. And many of them are so inured, so hopelessly dependent on the system that they will that they will fight to protect it. Are you listening to me? Or were you looking at the woman in the red dress? You can understand my trepidation. Everyone we have not unplugged is potentially an agent, an instantiation of the norms that bureacratize knowledge production, homogenize communities, deindividuate students, and make of the 'organized body of knowledge' a sea of thrashing sharks. So if you are not one of us, you are potentially one of 'them.' We do not know much, but we do know it was 'they' who scorched Psychology, because they do not understand the difference between 'knowing the path' and 'walking the path.'"

Perhaps the next time a denizen of the psychological community demands I provide psychometrically sound evidence for my arguments, I should respond by saying, "The image translators (i.e., pseudo-scientific paradigm) work for the construct program (i.e., institutionalized psychology)." This statement would not be without its merits. Allow me to introduce you to the Architects of University Psychology, to the tools that keep us in a false consciousness. All I offer is the truth. And I can only show you the door. You have to walk through it yourself. But be warned that when you awaken, you may experience a stinging in your eyes. Contrary to what you think at the moment, you've never used them.

Item Analysis

*The following is taken from a May, 2001 interview.

EHRENFELS: “Professors use a statistical technique – or should I say they have their research assistants use a statistical technique – known as ‘item analysis.’ This is only one of dozens of statistical formulas employed in research, but I would like to make an example of this one in particular because it is used in grading as well as in research and I feel it captures the essence of their M.O. A questionnaire or an exam contains multiple questions – what are known as ‘items.’ Many questionnaires are even subdivided into scales, groups of items that are purported to tap different aspects of your personality or knowledge. For example, the Graduate Record Examination and Scholastic Aptitude Test has a group of items constructed to measure your Verbal skills and another group for your Mathematical skills. Item analysis yields a coefficient which is a measure of the extent to which one item correlates with all the other items in the scale collectively – or if there are no subscales – in the questionnaire collectively.”

MOYER: “So – in other words – and let’s use the example you used in our earlier conversation – let’s say we have a Shyness scale – your answer to the question ‘on a scale from 1 to 7, rate the extent to which you prefer privacy to social gatherings’ is correlated with your combined answer to all the other questions in the scale to determine the extent to which this item is associated with the other items.”

EHRENFELS: “Yes. Naturally – this is not done on a person-by-person basis, but the analysis is performed once you have a number of people who have taken the questionnaire. We MAY find that despite the fact that particular question seems to be like the others in the Shyness scale – seems to measure the same thing – Shyness – we may find that people who tend to rate themselves higher on this particular item – end up scoring lower on the scale as a whole. So in effect, by keeping this particular question in the scale, we deflate or underestimate the true score on the scale. So we discard the item.”

MOYER: “Is there a cut-off you use. You have a coefficient for each item in the scale, right?”

EHRENFELS: “Correct.”

MOYER: “So then how you do know when the item is good, or not good? What does it take to keep it? Or to throw it out?”

EHRENFELS: “There are no hard and fast rules, except that once you adopt a cut-off, it should be the same cut-off used for all the items. Some professors will look at another coefficient – nicknamed alpha – which is the measure of the internal consistency of the scale – and by that I mean an average of the correlation of each item with all the other items. This is the extent to which the scale correlates with itself – the extent to which it can be said to measure the same thing. Researchers want this number to be high because supposedly something cannot correlate with something else to an extent greater than it can correlate with itself – and researchers WANT to devise a scale that correlates with other measures – and by that I mean a scale that is predictive or diagnostic of other things. And what they do is they will recalculate alpha for the scale WITH and WITHOUT each item, and if they notice that alpha would be increased without an item, that item is discarded. Ultimately, they want alpha to be at least .80. Anything below .70 is considered dubious, and some strive for an alpha greater than .90, because such an alpha is said to be admissible in court.”

MOYER: “So what is the problem with item analysis?”

EHRENFELS: “I have seen it abused. And by that I mean that when researchers are expected to apply it universally -- it has a price. I had a scale which I claimed measured x. Now I defined x in such a way that the items themselves were synonymous with the definition of x. So in my opinion, these items measure x as I define it regardless of what the alpha coefficient or individual item analyses tell me. Now imagine that one of the items significantly reduces the alpha.”

MOYER: “Above or below .70?”

EHRENFELS: “Doesn’t matter. Should I throw out the item? Well, no one will publish any research that involves a scale with a lower alpha. So I am advised to drop the item. But if I do, I change the meaning of the scale. The scale no longer measures x but some variation of x. This is all well and good – except I WANT to measure x.”

MOYER: “But according to alpha, you are not measuring x, right?”

EHRENFELS: “Not true. I will contend that x may not be as TIGHT or internally consistent a construct as that we are used to dealing with, but x is still x. You see, in our field, we are used to scales with .80 and even .90 coefficients. Have you ever SEEN one of these scales – that meet these criteria? The questions all look alike. It is a foregone conclusion that people will respond to them similarly because they are all variations of the same question. And THAT is how they are usually created. Someone thinks of a question and then thinks how to re-word it several ways. BORING!”

MOYER: “Not to mention artificial, right? This has been your complaint against field.”

EHRENFELS: “I have also complained that field demands consensus from its members. Well, it would also seem they demand consensus from its subject matter.”

MOYER: “How do you think they would respond to your criticism?”

EHRENFELS: “They would tell me any scientist who does not revise a theory to fit the data would be irresponsible.”

MOYER: “And how would you – ”

EHRENFELS: “Item analysis IS NOT data. If your hypothesis is that people who score high on scale x behave in y way or make z kind of decisions or experience w type of dreams, you have to TEST the hypothesis before you throw it out by changing or discarding scale x. The real data is in y, z, and w, not in x alone. Now I know that x can only correlate with y, z, and w to the extent that correlates with itself, but we demand a lot in the way of self-correlation. You don’t need a .80 or .90 – but this is what you are told you need in a scale if you want your research to compete for publication. This may explain in part why our literature is so lifeless and repetitive. It excludes too much – and what it does include correlates with itself, so to speak. We are also too quick in this field to throw out or revise theories to conform to the first signs of data. Let me tell you something. If we understood our data, we wouldn’t need theories. There is a little piece of every theory that is supposed to transcend the data – that is reserved to help us make sense of something as fickle and variable and contradictory as DATA. So I think we should give our theories the benefit of the doubt and stop attempting as quickly as possible to develop theories that are duplicates – or analog maps – of the data. We are not reassembling engines here. Hell – we use GRE scores as criteria in the admission of graduate students. You want to know how well the GRE scores predict performance in graduate school? The Verbal scale – the most predictive scale – accounts for only 16 percent of the variation in academic performance – 16 percent! And yet we continue to look at it.”

MOYER: “And why do you suppose that is?”

EHRENFELS: “Probably because it tells us a little about what kind of person the applicant is. The applicant may have good comprehension and writing skills, but we all know that doesn’t make the difference between a successful and unsuccessful graduate student. Now if we invented a conformity inventory – with subscales for compliance, sycophancy, acquiescence, obsequiousness, and cadence-and-imitation – then we would really have something. But my point is that sometimes we want to devise scales that measure circumstances we expect to vary or fluctuate. I may not want to measure something stable and internally consistent. I may want a barometer of sorts – and sometimes not even that. I may want a scale on which most people do not score HIGH or LOW most of the time – but when they do – it tells me something about the state they are in. I may want a series of scales that profile a state such that I expect most scale scores to be neither high nor low most of the time. But I expect the topography of the profile -- the variation among the scale scores and the scores that are RELATIVELY higher or lower – to tell me something. But these kinds of statistics are just not standard. No one has devised special rules or formats for them – so they would be overlooked. The field as a whole has fallen into such now routines – and routines are in and of themselves conducive to biases and prejudices. We preclude a range of possibilities concerning what may be learned and how it may be presented. Both beauty and truth suffer – as well as freedom. But like I mentioned earlier, what we do with items that do not fit into the scale is no different from what we do with researchers who do not live ‘in the fold.’”

MOYER: “You mentioned that item analysis is also used in grading exams.”

EHRENFELS: “Professors use item analysis to hunt for multiple choice questions that are answered correctly with as much success by students who score poorly on the exam as a whole as by students who score well on the exam. Such items are said to be ‘negatively discriminating’ – which means they discriminate against good students – and they are discarded. Now I agree that item analysis here has some limited benefits. I would like to use item analysis to make sure I didn’t accidentally key in the wrong answer for a multiple-choice item. And I may even double-check the item to see if it was ambivalent or ambiguous. But if I check the item – and it seems fine to me on the surface – and if the answer is keyed in correctly – I will not discard it regardless of the item coefficient. Poor students or not – they are STILL students. And I will give them what they earned. And let’s face it -- sometimes good students – especially these hyper-memorizing, over-achieving types – ”

Textbooks and Multiple Choice Exams

MOYER: “Careerists?”

EHRENFELS: “Perhaps. Sometimes they study in ways that is conducive not to learning but to performing well on multiple-choice tests. Sometimes a question comes along that discriminates against the bullshit memorization, artificial achievement, and pseudo-understanding. I will not punish the rest of the students for this. But the professors DO this because the technique itself is scientific and exacting, and because it gives them over a time a collection of the best test items. Some of the professors archive this data, thinking they are creating the perfect test or test bank. Some of them do this with an eye to publishing their test one day – that is – if they are not already using a test bank developed for the textbook by the publisher or the author’s graduate assistants. Some of these items can be bad too – despite the research. But hell – it’s easy.”

MOYER: “So some professors don’t even make up their own questions.”

EHRENFELS: “Most don’t. And why should they? They don’t design their own lectures. Those are designed by the textbook and supplemental teacher manuals – and some of the lectures may be delivered by graduate teaching assistants. So why not use the test bank that accompanies these materials.”

MOYER: “I bet you’ll tell me why.”

EHRENFELS: “Textbooks should never be assigned to students. They are survey volumes, like encyclopedias. They should be used only by instructors as reference material. They can be instrumental in developing one's syllabus so that one does not leave out significant content in a course that is supposed to be representative of the field. The widespread practice in modern psychology departments whereby students take multiple choice exams based solely on textbooks (and lectures based solely on textbooks) may be necessary to process classes of 300+ students, but there is very little mind expansion that occurs in (a) studying for such a test, (b) taking such a test, and (c) receiving the results of such a test. Those who acknowledge that the system is tragically but necessarily flawed are honest and make the most of a bad situation. The person who celebrates this science by tauting its 'systematic and psychometric properties,' that's 'Agent Smith.'"

As I mentioned earlier (it is worth repeating), the purpose of the textbook is three-fold:

• Provide a canon that represents that portion of our knowledge base that is relatively consensual (i.e., the common denominator of the field)
  

• Socialize students and new members into the field's academic and professional culture
  

• Provide a nexus for a framework of common expectations that allows everyone to (a) communicate mindlessly, (b) avoid decisions and doubts, and (c) languish in an air of self-contentment and validation, entitlements from the institution's welfare program. Or perhaps 'insurance benefits' for which individualists like myself can tolerate the deductibles (e.g., training) but cannot commit to a lifetime of premiums.

PRE-APPROVAL: Peer Review at the Front End

A more provocative position I've never declared. My adversaries love to exploit this position, lifting a soundbite from this statement to hoist on a flagpole. All to depict me as indifferent to the kind of standards that protect our water from contamination. My adversaries would have you believe that what they do is rocket science or brain surgery, but in actuality, the vast majority of talk therapy is NOT even constructed out of any psychological research (i.e., science), nor is it evaluated against research. Why? Because our science isn't there yet. Some would have you believe it is a matter of time, but I think our science is too crude an instrument to yield knowledge in the units of sensitivity, relevance, and usefulness that can inform therapy or even conversational discourse. Why? So-called "standards" like peer review. Why should I submit to the institutional safeguards and standards for a science of human nature when these arbitrary institutional inventions actually sabotage by enslavement the science of human nature?

Whenever we speak of peer review, and for that matter, committees of any kind, we are talking about a trade off. The benefits of peer review (and other institutional norms) reaped by most fields of scientific endeavor do not pay dividends for a science of human nature. In fact, the liabilities of peer review (and again other institutional norms) that normally hamper fields of scientific endeavor are amplified in the human sciences. Take groupthink for example. It is bad enough you have to make adjustments to appeal to the lowest common denominator of committee members. It's a second cousin to censorship, really. But then there are three forces which prompt or pressure Psychology's communities to refine, and by that I mean narrowly define, their standards. Power and expediency.

Legitimacy. Psychologists apprehend at various levels that psychology, both as a health delivery system and as a science, has not kept pace. Psychology is young. Psychology is also the only science in which the subject and object of science is the same. Within psychology, there is nothing to curb the proliferation of pet theories not only among pscyhologists but among laypersons with access to the subject material (i.e., themselves). By increasing the apparent potency of the standards, psychologists hope to first project a public impression that they belong among the ranks of doctors and scientists and second to restrict this expertise and authority to a certain class of "professionals."

Competition. Whenever a committee presides over a competitive application process, whether its admission to graduate school, appointment to faculty, or publication, the applicants are encouraged not only to appeal to the lowest common denominator of a committee, but to pander to it better than the other applicants. Over time, submissions to journals acquire a superfluous formal aspect and those applicants willing or able to demonstrate the greatest fidelity to the standards, are rewarded with positions of influence in the field. So committees grow accustomed to an escalating standard and the committees become populated by those who satisfied and exceeded those standards (i.e., who wears the epistemology of the field like a fashion runway model). And if you've ever seen some of those fashion shows, you know that most of this high art never reaches the street. Similarly, the contest itself to become a member of the academic and professional communities has conditioned them to lose touch with human nature.

Expediency. Search committees rely heavily on the current system of certification management to decide which applicants to put on the short list for tenure-track faculty positions. Undergraduate admissions offices rely similarly on the Scholastic Aptitude Test (SAT) and other standardized tests for a quick-and-dirty method of separating applicants. The developer of the SAT, Princeton-based Educational Testing Service (ETS) admitted its SAT does not have remarkable utility in predicting success in college, with its most valid scale (the verbal scale) accounting for only 16% of the variance in academic performance at the college level. But up until recently it was the college admissions offices who clamored for the standardized test, for which a difference in 200 points between any two applicants means roughly a difference in 3-4 correct answers. Since the California State School System recently decided it will not rely on the standardized test for admission decisions, ETS announced it will make sweeping changes to its test. The academic communities need to rethink peer review.

Legitimacy, competition, and expediency shape the "standards" in much the same way alcohol increases the potency of a medication.

This is out of place in Psychology precisely because Psychology is a young and human science in desperate need of the kind of spark that produces serendipity and discovery. But we've built a flame-retardant culture in which no spark can flame. By this I mean our standards have become so stingy as to deprive researchers of the degrees of freedom they would be allocated even under other sciences. Our standards have created a culture in which we all seek to think convergently rather than independently. This depives us of another type of fidelity that is actually more critical to our success as psychologists: our faithfulness to our own wits and to the raw phenomena under study. Between our wits and this phenomena there must be a direct relationship, but our Psychology have attempted to broker this relationship and further distance the researcher from the phenomenon, all with institutional norms that we are supposed to mistake for true scientific standards. I will spare you here the list of these norms, but suffice it to say that peer review is one of them. Where we are talking about an exploration of the human condition, we need fluid, flexible, broad exploratory research by divergent thinkers. And peer review spells career death to such individuals.

I once entertained a criticism that if the collective did not rule, that each researcher would be conducting his or her own scientific enterprise such that there would be a 'Bill Science,' a 'James Science,' and a 'Debra Science.' Within the very broad and generous framework that is the scientific method, why should our research decisions not reflect our personal preferences. In other words, as long as the word science is in there somewhere (i.e., as long as 'Bill' and 'James' and 'Debra' are still doing 'science'), then why should they be denied? Why should the personal preferences of one peer reviewer prevail over that of a contributor? Perhaps the system would be less egregious if we were permitted to submit to more than one journal at a time. But holding us to one journal, keeping us waiting 4-9 months for a response, is an additional inducement to conform (to imitate the work of others and to seek out these collective expectations) so as to maximize our chances of success, because failing to publish once a year could spell career death. Outside academia, authors are permitted to submit to as many publishers as they like simultaneously. Perhaps if we changed the system so journals competed with one another for the works of researchers rather than the researchers competing with one another for space in a journal, then our science would move forward.

I suggest a two-tier solution, not unlike the book review piece of Amazon.com. If we are willing to confer a doctorate on a person, then that person is permitted to publish into the database and then individual scientists (and yes, perhaps even committees) can post their reviews of the research. Under this system, only problematic research will be flagged and perhaps even pulled. But everything else that is acceptable (i.e., that meets minimum essential requirements and that does not violate any one of a list of criterial problems) will be available. As it stands right now, there are winners and losers in publishing and this need not be the case. It does not serve science. It only serves logistical constraints and perhaps some gatekeeping or lilly-guilding function. After all, it is convenient in helping selection committees decide among job applicants. If someone has published 12 times, clearly that person has no problem "fitting in," whereas someone else with the same number of years opportunity may have only published 3 times.

I cannot defend the current system. What is published may be acceptable, but there is a lot of research -- good research -- on the outside looking in and there is a lot of potential research that is never conceived or executed because it does not perfectly fit the mold. Now I never said you will not find a diversity of TOPICS literature, only a diversity in METHODS, PRESENTATIONS, and IDEAS. That being said, there are still some subjects that are grossly under-represented due to the fact they do not lend themselves as readily to the institutional norms. Unfortunately, many of these phenomena are what people think of when they think 'psychology.'

I just know I'll be raked over the coals for this one, as my adversaries will seek to exploit my position on peer review, lifting a soundbite from this essay and hoisting it on a flag. But it is difficult to deny that original ideas and less-than-popular research interests make it difficult for an author to appeal to the lowest common denominator of an editorial review committee. Why not create an online database of publications? A PhD is necessary and sufficient for publication. Action against an author/publication is withheld pending complaints about methodological flaws that (a) cannot be construed as liberties with strategic benefits, and/or (b) that cast doubt on the validity of conclusions as written. In even many egregious cases, a work can be salvaged by throwing a disclaimer in the discussion section qualifying or stipulating conclusions. The matter could always be referred to a committee for a hearing.

I mean, what is really the harm? This is not pharmaceutical research or evaluation of space shuttle components. Such research does not belong in Psychology. By pretending there is something at stake, we are denying ourselves a rare opportunity to attack our subject. We have a potential to bring together the best of science and humanities in one discipline. But as long as this false or inflated prestige surrounding publications and publication standards facilitates admission, appointment, and tenure review decisions, Psychology will continue to defy the technology that enables us to make all our research available.

TERRITORY: Cosmetic Fiefdoms

The arbitrary division of Psychology into fiefdoms (e.g., Social, Developmental, Cognitive) is also bad for the health of Psychology. Holistic research cannot survive if applicants to graduate schools cannot earn admission because their interests are too large to be a prototypical leaf on their branch. By replacing such a structure with a division of labor grounded in a natural bifurcation of Exploratory Research, Confirmatory Research, Theory, and Clinical, we capitalize on the strengths of our members. While it would be ideal to require members to demonstrate proficiency in each of these areas, this is not only unrealistic, but it risks replacement of one rigid system with another and would ultimately result in some standardized test. (This is not to say such a test would not be an immense improvement over the current scales of the Graduate Record Examination, but I cannot imagine a means of testing a person's predisposition to think. This is not as much a test of proficiency as of attitudes and aims).

MONEY: Grant Appeal

Let's replace the current policy of encouraging applicants to procure external sources of funding with a policy favoring those armed only with their wits and curiosities. Grants are hardly appropriate (hey, I have found a use for that term) for psychology. We do not need funding for truly psychologistic research. Granted (not pun intended), there is some psychological research of value that required funding, but the field would be much better off if we discouraged rather than rewarded this form of prostitution. Since most fundable research is not psychologistic, those who perform psychologistic research are at a marked disadvantage for tenure. Over time, the face of psychology departments is transformed by granting agencies. We hardly know enough about personality today to notice its disappearance. Meanwhile, the Human Factors branch is emerging in universities across the country at the rate of Starbucks to tell us how to build a better spatula.

Rule-Bound Field

The Matrix character Thomas "Neo" Andersen wanted to show us a world without rules or boundaries. Similarly, I'd like to show you a less proscriptive Psychology. The denizens of the psychological community identify the essential characteristics of science with adherence to 'rules.' This results in a scientific paradigm that constrains independent thinking and that hijacks science in the service of skepticism at the expense of its service to exploration. Is it any wonder industrial-organizational psychologists can defend Gallup's hiring practices by claiming the loss of our freedom and individuality is a small price to pay for validity (see Psychologists Party to Dishonest Hiring Practices at Gallup)? Is it any wonder we sanction all research into anything that smacks of the paranormal (e.g., prospective dreaming) so that we would appear more credible in our efforts to protect the public against people who claim to communicate with the dead? Is it any wonder that most psychological researchers characterize the truth as 'what remains after everything else has been refuted'? (Would we really know when to stop chipping away at that block of marble?)

Well, I deem it my mission to liberate psychological researchers from their self-imposed prisons and thereby rescue the human psyche from the field of Psychology. Take the psychology professor as an instructor fond of using item analysis to Q.A. an exam and build a utopian evaluation tool. Some of the items they would discard are not even wildly discriminating. In other words, they do not just throw out extremely negative coefficients, but also coefficients which are mildly negative, and some even discard items with mildly POSITIVE coefficients in search of that perfect exam and that perfect bell curve. Sometimes I think they were conditioned to see beauty in that normal bell curve shape. Now this practice in and of itself is not that consequential. That is why it has escaped everyone’s notice. But it is symptomatic of some of their more consequential choices – and of a consequential PATTERN of behaviors which – taken collectively –introduces a credentialism that favors careerists and discriminates at more advanced levels of education against the true scholars. They really are creating a race of administrative savants. And what you really end up doing is narrowing the range of skills tapped -- or narrowing the range of tapped skills that are reported – such that you end up with this yardstick that measures JUST ONE THING. And if you are not in the top x percent of this ONE skill or quality or attitude – your odds of making it are very small. We really don’t pay much attention to the fact – and I imagine this flaw dogs every field to a certain extent – that there are people who aspire to be members of the field who are bright and creative – probably brighter and more creative than most in the field – who never really get the chance to make it in. They are weeded out at some point without a fair hearing. I think the half of the public that DOES see this ACCEPTS it as the work of that chance component that is part of life. I am here to put quite a different face on that chance element – to tell you that it really ISN’T chance at all – but the work of something very systematic you may not see. A "prison for your mind."