My three criticisms of DSM-5 have been: 1) risky suggestions; 2) bad writing; and 3) poor planning and disorganization. I have pretty much failed to have any real impact on DSM-5; other than perhaps successfully pressuring APA to delay its publication once before, moving it from May 2012 to May 2013. The one-year extension has been largely wasted, the risky suggestions and bad writing remain, and constant warnings that missed deadlines would lead to a mad and careless race at the end were ignored.
With less than a year remaining before DSM-5 is scheduled to go to print, the signs are clear that it cannot possibly be completed on time unless we are willing to settle for a third rate product. The unmistakable red flag is the recent embarrassing admission that DSM-5 will accept diagnoses that achieve reliabilities as unbelievably low as 0.2-0.4 (barely beating the level of chance agreement two monkeys could achieve throwing darts at a diagnostic board). This dramatic departure from the much higher standards of previous DSM's is a sure tip-off that many DSM-5 proposals must be failing to achieve adequate diagnostic agreement in the repeatedly delayed and yet to be reported field trials. Unable to meet expected standards, the DSM-5 Task Force is drastically and desperately trying to lower our expectations.
After reading the first drafts posted in early February 2010, I warned that DSM-5 was in for severe reliability problems. The criteria sets were in remarkably raw form; clearly, they were no more than the draft product of the work groups deprived of the extensive editing needed to turn vague diagnostic concepts into precise, unambiguous, and consistent diagnostic criteria. It was apparent that reasonable diagnostic agreement would be impossible to achieve with criteria so poorly and confusingly written.
The writing of criteria sets is a highly specialized skill that requires a clinician's experience, a computer scientist's command of algorithmic logic, and a lawyer's vigilance. I have known only a handful of people who have mastered this exotic craft and must admit that I cannot myself write decent criteria, despite years of trying to learn. If anyone working on DSM-5 had this necessary skill, the initial drafts would not have been made public in such a ragged and amateur state and the writing has not improved appreciably since. All of the DSM IV options entered field-testing in final draft form - every word had already been subjected to many iterations and countless reviews. In contrast, DSM-5 went into field-testing with primitive drafts that were painful to read.
The original plan for DSM-5 did have a necessary fail-safe; its field-testing was meant to be conducted in two stages. Those criteria sets that performed poorly in the first phase could then be rewritten by the work groups and retested to prove their mettle in the second stage. But disorganization kept delaying the start and plagued the execution of the field trials and deadlines were consistently missed, so that the reporting of results fell at least eighteen months behind the original schedule. For want of time, the absolutely necessary second phase was cancelled, thus circumventing the rewriting and the retesting needed to improve the poorly written criteria. The decision to take this unfortunate shortcut was done secretively, without any announcement or any discussion of its detrimental impact. So, it now appears that APA plans to publish poorly worded criteria sets as the official DSM-5, despite the fact that they have performed poorly in field-testing. The product will be a confusing DSM-5 that fails to provide the diagnostic agreement that is vital for clinical communication, research, and forensics.
The wise, safe, and responsible thing for APA to do now is to delay publication of DSM-5 until the missing second stage of rewriting and retesting can be completed. The wordings that have done poorly in the first stage of field-testing should be rewritten to finally attain the clarity and consistency necessary in an official manual of psychiatric diagnosis. The newly revised (and hopefully final) versions should then undergo the second stage of field-testing as originally envisaged to ensure that they now work. The extra time will also allow for the independent scientific reviews of controversial DSM-5 proposals called for in a petition that has already been signed by more than 11,000 mental health professionals and is endorsed by 40 professional organizations (including many divisions of the American Psychological Association, the American Counseling Association, and the British Psychological Society).
Will APA do what is needed to protect us from a poor quality DSM-5 and instead guarantee one that is safe and scientifically sound? It seems unlikely. The DSM-5 publishing profits that are essential to APA budget projections require there be a May 2013 debut of the manual in bookstores, come hell or high water. So instead of getting DSM-5 up to minimal standards of quality, DSM-5 is trying to drop the standards to minimal - 0.2-0.4 will have to do.
What about the DSM-5 claim that its field trials are so rigorous that we should entertain only the lowest possible expectations of them? This is nonsense. The DSM-5 field trials were in fact conducted under very privileged circumstances that would guarantee much higher levels of reliability than could ever be achieved in everyday clinical practice: 1) Testing was performed in academic centers with a homogeneous corps of well trained raters interested in psychiatric diagnosis and trying their best because judgments were being observed; 2) Raters had access to the results of a computerized self report instrument, thus reducing information variance; 3) Each site specialized in a limited number of target diagnoses that were known to the raters who would therefore be on the watch for them; 4) The unrealistically high prevalence of target disorders in the sites made agreement much easier than the more needle-in-haystack situation of routine practice; 5) Academic settings attract a selected group of the more severely ill patients who are easier to diagnose reliably; and 6) The time allotted for diagnostic interviews exceeded what is typical in clinical practice.
Despite all these advantages, the DSM-5 Task Force is inviting us to settle for levels of agreement just above chance. If DSM-5 performs so poorly when the deck is heavily stacked in its favor, how will it perform in the rough and tumble of the real world?
Which leads to the question: what can be done now to rescue a failing DSM-5? The APA Trustees are face to face with a chilling but unavoidable moment of truth. The press, the Internet, even the TV is filled with prominent stories highly critical of DSM-5. There is simply no way to hide its recklessness and low quality standards. A May 2013 publication date appears to be completely unrealistic unless we are to settle for a DSM-5 so poorly done that its reliabilities will return us to the dark ages of DSM II. DSM-5 is in a very deep hole with very few remaining options.
My recommendations: 1) Make the publication date flexible and contingent on delivery of a quality product that the field can trust; 2) Subject the current drafts and texts to extensive editing for clarity and consistency; 3) Drop the controversial suggestions that risk harmful unintended consequences or at least subject them to external scientific review; 4) Have the rewritten drafts reviewed word for word by many experts in the clinical, research, and forensic uses of DSM-5; and 5) Field test again to make sure the new versions work adequately.
It will be argued back that my suggestions will take time and cost money. But APA has spent a reported $25 million on preparing DSM-5- much of it clearly wasted in missteps. A little more time and a little more money will be very well spent, if this is the only way to salvage a DSM-5 that can be trusted.
The last point is many critics use the specific failures of DSM-5 as justification to attack the entire enterprise of psychiatry. I strongly disagree. DSM-5 is no more than an unfortunate aberration reflecting the temporary state of weak and misguided APA leadership. The work on DSM-5 went off track because of unrealizable ambitions; a closed and secretive process; and insufficient attention to the day-to-day details of prudent planning, management, and careful writing. Because of its poor performance on DSM-5, APA has probably forfeited its right to sole control of future revisions. But all this represents only the specific failure of DSM-5, not a general reflection on what psychiatry is and what it can accomplish. Done well and within its reasonable limits, psychiatry is an extremely helpful and essential profession. It would be a shame to throw the valuable baby out with the bath water or discourage patients from getting the psychiatric help they need and can benefit from. Admittedly, DSM-5 is an embarrassment and a serious hit to our credibility, but we will recover and our patients should not lose faith.
With less than a year remaining before DSM-5 is scheduled to go to print, the signs are clear that it cannot possibly be completed on time unless we are willing to settle for a third rate product. The unmistakable red flag is the recent embarrassing admission that DSM-5 will accept diagnoses that achieve reliabilities as unbelievably low as 0.2-0.4 (barely beating the level of chance agreement two monkeys could achieve throwing darts at a diagnostic board). This dramatic departure from the much higher standards of previous DSM's is a sure tip-off that many DSM-5 proposals must be failing to achieve adequate diagnostic agreement in the repeatedly delayed and yet to be reported field trials. Unable to meet expected standards, the DSM-5 Task Force is drastically and desperately trying to lower our expectations.
After reading the first drafts posted in early February 2010, I warned that DSM-5 was in for severe reliability problems. The criteria sets were in remarkably raw form; clearly, they were no more than the draft product of the work groups deprived of the extensive editing needed to turn vague diagnostic concepts into precise, unambiguous, and consistent diagnostic criteria. It was apparent that reasonable diagnostic agreement would be impossible to achieve with criteria so poorly and confusingly written.
The writing of criteria sets is a highly specialized skill that requires a clinician's experience, a computer scientist's command of algorithmic logic, and a lawyer's vigilance. I have known only a handful of people who have mastered this exotic craft and must admit that I cannot myself write decent criteria, despite years of trying to learn. If anyone working on DSM-5 had this necessary skill, the initial drafts would not have been made public in such a ragged and amateur state and the writing has not improved appreciably since. All of the DSM IV options entered field-testing in final draft form - every word had already been subjected to many iterations and countless reviews. In contrast, DSM-5 went into field-testing with primitive drafts that were painful to read.
The original plan for DSM-5 did have a necessary fail-safe; its field-testing was meant to be conducted in two stages. Those criteria sets that performed poorly in the first phase could then be rewritten by the work groups and retested to prove their mettle in the second stage. But disorganization kept delaying the start and plagued the execution of the field trials and deadlines were consistently missed, so that the reporting of results fell at least eighteen months behind the original schedule. For want of time, the absolutely necessary second phase was cancelled, thus circumventing the rewriting and the retesting needed to improve the poorly written criteria. The decision to take this unfortunate shortcut was done secretively, without any announcement or any discussion of its detrimental impact. So, it now appears that APA plans to publish poorly worded criteria sets as the official DSM-5, despite the fact that they have performed poorly in field-testing. The product will be a confusing DSM-5 that fails to provide the diagnostic agreement that is vital for clinical communication, research, and forensics.
The wise, safe, and responsible thing for APA to do now is to delay publication of DSM-5 until the missing second stage of rewriting and retesting can be completed. The wordings that have done poorly in the first stage of field-testing should be rewritten to finally attain the clarity and consistency necessary in an official manual of psychiatric diagnosis. The newly revised (and hopefully final) versions should then undergo the second stage of field-testing as originally envisaged to ensure that they now work. The extra time will also allow for the independent scientific reviews of controversial DSM-5 proposals called for in a petition that has already been signed by more than 11,000 mental health professionals and is endorsed by 40 professional organizations (including many divisions of the American Psychological Association, the American Counseling Association, and the British Psychological Society).
Will APA do what is needed to protect us from a poor quality DSM-5 and instead guarantee one that is safe and scientifically sound? It seems unlikely. The DSM-5 publishing profits that are essential to APA budget projections require there be a May 2013 debut of the manual in bookstores, come hell or high water. So instead of getting DSM-5 up to minimal standards of quality, DSM-5 is trying to drop the standards to minimal - 0.2-0.4 will have to do.
What about the DSM-5 claim that its field trials are so rigorous that we should entertain only the lowest possible expectations of them? This is nonsense. The DSM-5 field trials were in fact conducted under very privileged circumstances that would guarantee much higher levels of reliability than could ever be achieved in everyday clinical practice: 1) Testing was performed in academic centers with a homogeneous corps of well trained raters interested in psychiatric diagnosis and trying their best because judgments were being observed; 2) Raters had access to the results of a computerized self report instrument, thus reducing information variance; 3) Each site specialized in a limited number of target diagnoses that were known to the raters who would therefore be on the watch for them; 4) The unrealistically high prevalence of target disorders in the sites made agreement much easier than the more needle-in-haystack situation of routine practice; 5) Academic settings attract a selected group of the more severely ill patients who are easier to diagnose reliably; and 6) The time allotted for diagnostic interviews exceeded what is typical in clinical practice.
Despite all these advantages, the DSM-5 Task Force is inviting us to settle for levels of agreement just above chance. If DSM-5 performs so poorly when the deck is heavily stacked in its favor, how will it perform in the rough and tumble of the real world?
Which leads to the question: what can be done now to rescue a failing DSM-5? The APA Trustees are face to face with a chilling but unavoidable moment of truth. The press, the Internet, even the TV is filled with prominent stories highly critical of DSM-5. There is simply no way to hide its recklessness and low quality standards. A May 2013 publication date appears to be completely unrealistic unless we are to settle for a DSM-5 so poorly done that its reliabilities will return us to the dark ages of DSM II. DSM-5 is in a very deep hole with very few remaining options.
My recommendations: 1) Make the publication date flexible and contingent on delivery of a quality product that the field can trust; 2) Subject the current drafts and texts to extensive editing for clarity and consistency; 3) Drop the controversial suggestions that risk harmful unintended consequences or at least subject them to external scientific review; 4) Have the rewritten drafts reviewed word for word by many experts in the clinical, research, and forensic uses of DSM-5; and 5) Field test again to make sure the new versions work adequately.
It will be argued back that my suggestions will take time and cost money. But APA has spent a reported $25 million on preparing DSM-5- much of it clearly wasted in missteps. A little more time and a little more money will be very well spent, if this is the only way to salvage a DSM-5 that can be trusted.
The last point is many critics use the specific failures of DSM-5 as justification to attack the entire enterprise of psychiatry. I strongly disagree. DSM-5 is no more than an unfortunate aberration reflecting the temporary state of weak and misguided APA leadership. The work on DSM-5 went off track because of unrealizable ambitions; a closed and secretive process; and insufficient attention to the day-to-day details of prudent planning, management, and careful writing. Because of its poor performance on DSM-5, APA has probably forfeited its right to sole control of future revisions. But all this represents only the specific failure of DSM-5, not a general reflection on what psychiatry is and what it can accomplish. Done well and within its reasonable limits, psychiatry is an extremely helpful and essential profession. It would be a shame to throw the valuable baby out with the bath water or discourage patients from getting the psychiatric help they need and can benefit from. Admittedly, DSM-5 is an embarrassment and a serious hit to our credibility, but we will recover and our patients should not lose faith.
Leave a comment