Originally published in Communication News 9(3) May/June 1996
In 1994 the University of Sydney’s Centres for Plain Legal Language and Microeconomic Policy Analysis jointly published a discussion paper on the potential costs and benefits of plain legal language, The Costs of Obscurity (Duckworth & Mills 1994).
This was to be the start of a project undertaken by these two centres—funded by the Law Foundation of New South Wales—into the costs and benefits of plain English. The Centres have published their final report—The Gains from Clarity (Mills & Duckworth 1996).
The project did not turn out as the authors may have expected, judging by their original discussion paper. But it has provided plain English advocates with an opportunity to gain some wisdom which will help them better understand real world problems of communication, and some of the limitations of approaching these problems from a plain English perspective.
In the original discussion paper the authors confidently advanced the proposition:
…that organisations using plain language could potentially gain from efficiency. (Duckworth & Mills 1994, p1)
In the final report there is no discussion of costs, but a far more humble understanding of the difficulties in collecting such data. The authors explain:
… [information about costs] was difficult to gather. Some of the work done on the new documents was done by outside consultants whose costs could be easily quantified. However, a considerable amount of time was taken up in committees and other internal administration. Few organisations seem to take these costs into account when producing new documents. In the outcome, the amount of data we collected on the costs of introducing new documents was very limited. (Mills & Duckworth 1996, p1)
These explanations give a glimpse of the underlying complexity that frequently attends any attempt to measure the costs and benefits of changing communication practices within an organisation, and point to the dangers of assuming that one can attempt to rewrite a document in isolation from the political and administrative environment in which the document functions. Nonetheless, without data on costs and benefits it is impossible to assess whether the reported improvements justified the internal administrative costs or the consultant’s fees. And the absence of this data undermines one of the main purposes of the study, leaving plain English advocates still exposed to the criticism that they offer no economic evidence that their work adds value.
Before and after
One of the other criticisms made repeatedly about plain English advocates is that they assume—often without evidence—that the work they do necessarily results in improvements to documents. The missing data in plain English case studies is often evidence of a document’s usability before and after being ‘plain-Englished’. Mills & Duckworth’s (1996) study attempts to provide that evidence by studying three forms before and after they had been ‘plain-Englished’.
Unfortunately, the attempt is weak. There are four fundamental flaws.
First, there is no detailed account of the exact methods the consultants used. There is considerable variation among plain English consultants in how they define ‘plain English’ and how they go about applying it to improve documents. Calling them all ‘plain English consultants’, as the study does, fails to tell us whether they each applied the same methods. Is the data comparing like with like? Or are some plain English methods more effective than others? This is a serious question for any manager wanting to employ a consultant and eager to get value for money.
Second, in only one case does the study include a copy of the before-and-after documents—the Family Law Court divorce application form. Thus, there is little opportunity to compare the new designs with known good forms design principles. In the one case that is presented, the new form violates some good forms design principles that have been repeatedly proven through testing. It may well be that divorce applicants constitute special types of users who fill out forms differently to the rest of us. However, if this is not the case, as seems likely, the designs could explain some of the contradictory before-and-after data.
Third, without a clear statement of objectives behind the improvement, there is no way of collecting meaningful before and after data. If one doesn’t know what it was that one wanted to improve, one cannot then measure the improvement.
Fourth, some of the before and after data came from questionnaires which collected indirect data: that is, opinions and recollections about completing and processing the form. Indirect data are just that, and are notoriously difficult to evaluate; questionnaires administered retrospectively—collecting peoples’ recollections and opinions—are not necessarily related to their actual behaviour.
Balmford and Kelly, two plain English consultants, recently reported that their plain English documents filled people with confidence—people claimed to find them easy to understand—but, unfortunately, the same people made many errors using the documents. There are many examples in the literature of this worrying over-confidence and misjudgement when faced with documents that look better because of plain English and modern layout. It is therefore surprising that Mills & Duckworth make most of their claims about the success of the new versus the old on the basis of these kind of data, and moreover assert that:
The best way to summarise the experience of self-applicants [for divorce] is to report their own expressed evaluation of the extent of the difficulty…These before-and-after differences are so large as to be very highly significant in formal statistical terms. (pp35–36)
But this is only part of the problem. The questionnaire itself is headed with the title: Making forms simpler This title is then followed by an introduction which says:
The Family Court has changed the divorce applications forms.
This is to make them easier to fill in and quicker to process [bold in original].
With this type of ‘suggestion’ the positive results are hardly surprising; the questionnaire makes it obvious what type of results the researchers are looking for.
There is no doubt that the reported questionnaire data show that people believed the new forms were better than the old. In other similar studies, people have often reported that plain English documents are easier to use than their gobbledegook counterparts. Plain English documents look better. But often, the effect lies in appearances, not necessarily in outcomes. These data fail to satisfy critics who seek a higher standard of evidence before accepting that the documents perform effectively both for the organisation and for the users.
Time is money
Interestingly, the data which throw some light on the persistent question of effectiveness are the most useful data in the study. The researchers collected direct observational data on the time it took administrators to perform various processing tasks using the old and new forms. As we have said many times in Communication News, this type of direct observational data is the most credible when trying to make judgement about how people actually behave. From these direct data of the time it took to perform tasks using the old and new forms, there is clear qualitative and statistically valid evidence that in all cases there was either no significant difference in the time it took to complete tasks with the old and new forms, or—alarmingly—it took longer to complete tasks with the new forms (pp. 85–88).
Given that organisations pay for people to perform these administrative tasks on forms, one might conclude that plain English forms cost more to process than their gobbledegook counterparts! Instead of creating a cost benefit, the solutions seem to offer costs plus. One could argue against this conclusion. Perhaps the extra time spent ensured that the forms were more thoroughly scrutinised and this led to fairer decision-making, or that there was something fundamentally different about the new task. Unfortunately, no detailed evidence of this type is presented.
A revealing conclusion
In the conclusion to this report there is a clue about the underlying problem that continually dogs plain English advocates: In general, the organisations we studied seem to have started by seeing their project as simply one to revise or re-write a document. They appeared not to realise the extent to which the project might involve changes in management or administration, and even revision of the policy behind the document…the document is often seen in isolation, out of its overall context. (pp68–69) But this is in no way surprising. If one offers a plain English service, that is what buyers of the service are likely to expect. They will expect the consultant to improve communication by revising or rewriting their documents. That is what plain English implies.
But Mills & Duckworth protest:
We found that some organisations used the first drafts of the document essentially as a ‘discussion paper’, which revealed problems with the underlying policy or administration. Resolving these problems during the writing process, rather than before, can be a very inefficient approach. Nevertheless, sometimes this may be unavoidable. The existing document may be so obscure that the problem appears only when it is redrafted.
Blaming the victim—the organisation that commissioned the plain English advocate—is unfair. The failure lies with the plain English consultant who underestimated the complexity of the political and administrative environment in which the document has to function, and simply offered a plain English service to solve the problem. There can be no doubt, following Mills & Duckworth’s conclusion, that the authors have learnt a great deal about the realities of document design and communication problems in large organisations. It remains to be seen whether they can turn that learning into practical wisdom, as well as hard evidence.
We recommend to our Corporate Members that if you do contract plain English consultants, that you do not allow the look of a document, its language or graphics, to persuade you that it is intelligible. Demand direct observational evidence that the consultant’s methods actually result in documents that perform better than current documents when given to real users in real situations. Evidence should not be derived from ineffective techniques such as opinion and attitude surveys. Consumers should be as vigilant in scrutinising new-look documents as they are sceptical of six point type and gobbledegook.
Undoubtedly plain English advocates have come a long way in recognising the need for evidence to support their claims. This is to be welcomed. They have additionally learnt that they need to take a much broader view of their activity in order to genuinely improve communication. Such getting of wisdom is also to be welcomed. However, their claims that plain English is cost-effective still remain to be proved.