**************************************************************************************

Personality & Individual Diffferences, Vol. 8, No. 4, pp. 571-573, 1987

MAXIMIZING THE RESPONSE-RATE IN SURVEYS MAY BE A MISTAKE



JOHN J. RAY [1] and LEONIE V. STILL [2]

[1] School of Sociology, University of New South Wales, P.O. Box 1, Kensington, New South Wales, Australia 2033

and

[2] Nepean College of Advanced Education

(Received 24 March 1986)

Summary

A random postal sample of 400 Australians was carried out with the assistance of two conventional response-rate maximization techniques: a preliminary letter and a second mailing of the questionnaire aimed at non-respondents to the first questionnaire. The response-rate enhancement techniques appeared to have doubled the response rate from a usual 25% for a one-wave mailing to 47% on the present occasion. The present data, however, showed severe effects of acquiescent response bias ('saying yes to anything'), thus severely damaging the meaningfulness of the results. It is concluded that the use of response-rate enhancement techniques is not only unnecessary but is also in fact counterproductive.

INTRODUCTION

It is universally accepted that the form of the question can greatly influence the result of a survey. One aspect of question wording that has a highly predictable effect is whether or not a question is worded positively or negatively. People tend to agree rather indiscriminately with many statements so that one survey of a population might for instance find 66% of the respondents agreeing with the statement "The United States should stay in NATO" while another survey of the same population might find 72% of the population agreeing that "The United States should withdraw from NATO". The two results are clearly inconsistent and show that much of the acquiescence recorded in both surveys was essentially meaningless. If such a result was obtained as part of a proper 'split plots' survey design, one might conclude that the true result was the average of the two percentages, i.e. 47% of the population favoured staying in NATO. Split plots designs, however, seem to be more often advocated than practised.

Given the extensive contamination of survey results by acquiescent response bias, therefore, it becomes of some importance to know the magnitude of such effects. Is the effect so small that it can safely be ignored? While claims that such bias is unimportant have been made (Rorer, 1965), the balance of the evidence and argument is to the contrary (Peabody, 1966; Jackson, 1967; Campbell, Siegman and Rees, 1967; Vagt and Wendt, 1978; Ray and Pratt, 1979; Ray. 1983, 1985a). Research designs should therefore take account of this problem.

The two obvious ways to guard against acquiescence contamination are to adopt strategies to discourage its emergence and to take precautions that prevent its presence from distorting our results. Split plots surveys and the use of balanced additive scales are recognized examples of the latter but what can be done to accomplish the former is more controversial. The obvious strategy is to endeavour to make each question highly meaningful but this is easier said than done and it is any case evident that a question that may be perfectly meaningful to one sample may be much less meaningful to another (Ray, 1983). One simply has to adopt all available strategies and hope that at least some will have the desired effect.

One possible strategy that seems to have been rather neglected is to confine oneself to willing respondents. Survey-takers customarily use a whole host of techniques designed to maximize the response rate to surveys. Even techniques as extreme as bribes and threats are on some occasions used. Yet there seems something fundamentally misguided in all such attempts. As the old anti-Nazi song said, "Die Gedanken sind frei". You cannot force people to tell you their real thoughts. Any degree of pressure applied to people to get them to tell you their attitudes may backfire in the form of dishonest or less than frank answers. A very obvious form of such rebellious responding is acquiescent response set. People may simply decide to say 'Yes' to everything. An example of this occurred in Ray (1974) where army conscripts were asked to answer a survey as part of their normal military duties. A nominal 100% response rate was of course achieved as the conscripts were not free to withdraw from the task but the amount of meaningless acquiescence observable in the replies collected was phenomenal -- on occasion overwhelming the available techniques of allowing for its influence (Ray, 1985b). Response-rate enhancement techniques may then simply substitute an acquiescence problem for a non-response problem.

But is this true of all such techniques? Conscription represents extreme pressure. Would milder pressures such as follow-up letters lead to such problems? The research reported below was designed as a first step in answering this.

METHOD

In the case of postal (mail-out) surveys, the sending of follow-up letters to non-respondents may seem the obvious way to increase the response rate but experienced survey researchers know that the technique is in fact generally not very productive. Very few extra replies are gained that way. The most productive of the routine techniques generally seems in fact to be the sending out of a preliminary letter before the questionnaire proper is mailed. Both techniques (preliminary letter and a second wave of questionnaires) were therefore used on the present occasion. As the questionnaire was anonymous, the second wave of questionnaires had to go to all the initial addresses rather than just the non-respondents. The second wave questionnaire was the same as the first wave questionnaire except for the introduction.

A total of 400 questionnaires was sent out by mail [*] to addressees selected at random from the electoral rolls (registered voter lists) of the Australian States of Queensland and New South Wales. Voter registration in Australia is compulsory for all adult citizens and even for many non-citizens. The Australian electoral rolls do thus represent an unusually comprehensive sampling frame and, as such, probably deserve more use by social scientists than they seem generally to have received. Additionally, the present senior author has conducted a large number of one-wave postal surveys (without preliminary letters) using these rolls so a reliable base against which to compare the response rate on the present occasion is available.

The content of the questionnaire was similar to that used in the previous surveys referred to. It comprised the Rathus (1973) Assertiveness scale, the Lorr and More (1980) Assertiveness battery, the Ray (1976) Directiveness scale, the Rosenberg (1965) Self-esteem scale, the Buss-Durkee Hostility inventory (Buss and Durkee, 1957) and the Eysenck (1959) 'N' scale (short form). Such questionnaires have in the previous surveys referred to generally produced response-rates of around 25%. [See Ray and Bozek (1980), Ray and Ray (1982) and Ray (1982).] A response rate substantially above this would then suggest an effect due to the response-rate enhancement techniques used.

RESULTS

A total of 139 replies were received to the first-wave questionnaire plus 47 replies to the second wave. There was thus an overall response rate of 47%, of which the second wave contributed only 12%. Clearly, however, the techniques used on this occasion did overall lead to a marked improvement in the response rate. Adding a preliminary letter and a follow-up questionnaire appears to have roughly doubled the response rate. This is obviously then a context in which the hypothesized acquiescence effects might be found.

The fact that the questionnaire used balanced additive scales rather than unrelated items did enable a variety of tests for the presence of acquiescent bias to be carried out. A balanced scale is one in which the respondent has to give an equal number of 'Noes' and 'Yeses' to get a maximum score. It thus enables control for acquiescent bias without the need to resort to a split plots design. In effect, the varying questions of a split plots design are all given at once.

The scales did in fact show the stigmata of a serious acquiescence problem. As is usual, however, not all scales were equally affected (Ray, 1983). The Rathus Assertiveness scale and the Lorr and More Social Assertiveness scale showed quite good reliabilities (coefficient alpha) of respectively 0.82 and 0.76 even though both scales were balanced. A substantial acquiescent tendency in the data tends to reduce the reliability of a balanced scale and increases the reliability of a one-way worded scale. On the other hand, however, the Ray (1976) Directiveness scale and the Rosenberg Self-esteem scale showed substantial reliability collapse. The Directiveness scale showed a reliability of 0.63 versus a usual reliability of around 0.74 (see Ray and Kiefl, 1984) while the Rosenberg scale showed a reliability of 0.58 compared to a usual figure of more than 0.80 (Ray, 1984a). Even more serious an effect was the fact that the Lorr and More Defence of Rights and Individuality scales showed no internal consistency at all. The mean inter-item correlations of these two balanced scales were 0.013 and -0.032 respectively. Further, the negative items of the only partially balanced Buss-Durkee inventory all showed negative rather than positive correlations with their scale total (after appropriate reverse-scoring). All these results are to be expected where there is a strong tendency to meaningless acquiescence in the data. This is also true of the one-way worded Eysenck `N' scale. Its reliability was unusually high. The alpha of 0.88 recorded on the present occasion is much higher than any previous Australian result. The highest previous such result appears to be the 0.73 reported in Ray (1981b).

There is, however, a way in which the role of acquiescence can be directly measured rather than inferred from its psychometric symptoms. This is to score each balanced scale without reversals to allow for item meanings. Each scale thus becomes a scale of meaningless acquiescence. When this was done several of the scales proved to be very good measures of acquiescence. Alphas of 0.90, 0.80, 0.84 and 0.88 were observed for the Independence, Defence of Rights, Self-esteem and Ray Directiveness scales.

DISCUSSION

Clearly, the improvement in response rate achieved on the present occasion by quite conventional and low-level techniques was accompanied by a much greater amount of meaningless acquiescence in the data. Not only unwilling respondents but even grudging respondents tend to 'say Yes to anything'. A partial solution to the response rate problem seems to have led to the acquisition of a serious new acquiescent bias problem. This shows that in our surveys we are not only utterly dependent on voluntary co-operation but we are even dependent on willing co-operation. Only a survey of willing respondents can aspire to meaningful results in most cases. Much as we would like 100% response rates to our surveys, chasing such a response rate is probably more than futile. In fact, the more we achieve it, the more meaningless our results may become. In doing social science we may just have to settle for interviewing the willing respondents only. A low response rate to a survey may then be not a fault but a necessary precondition for meaningful research. Since conventional one-wave postal surveys do tend to produce a sample with a representative demographic profile in spite of their low response-rate, the loss entailed by a low response-rate may in fact be more apparent than real. At least in Australia, the statistics on age, sex, occupation and education for samples obtained by one-wave postal samples (Ray, 1982; Ray and Ray, 1982; Ray and Bozek, 1980) appear to be virtually indistinguishable from the demographic profile observed in samples gathered by doorstep interviews (e.g. Ray, 1979, 1980, 1981a, 1984b) so the representativeness of such postal samples is at least similar to that of the most widely acceptable alternative sampling method. On balance, then, the use of response-rate maximization techniques for postal samples is probably both unnecessary and detrimental to the quality of the data.

REFERENCES

Buss A. H. and Durkee A. (1957) An inventory for assessing different kinds of hostility. J. consult. Psychol. 21, 343-349.

Campbell D. T., Siegman C. R. and Rees M. B. (1967) Direction of wording effects in the relationship between scales. Psychol. Bull. 68, 293-303.

Eysenck H. J. (1959) Manual of the Maudsley Personality Inventory. University of London Press, London.

Jackson D. N. (1967) Acquiescence response styles: problems of identification and control. In: Response Set in Personality Measurement (Edited by Berg I. A.). Aldine, Chicago.

Lorr M. and More W. W. (1980) Four dimensions of assertiveness. Multivariate Behav. Res. 2, 127-138.

Peabody D. (1966) Authoritarianism scales and response bias. Psychol. Bull. 65, 11-23.

Rathus S. (1973) A 30 item schedule for assessing assertive behaviour. Behav. Ther. 4, 398-406.

Ray, J.J. (1974) Are the workers authoritarian, conservative or both? Ch. 43 in Ray, J.J. (Ed.) Conservatism as heresy Sydney: A.N.Z. Book Co.

Ray, J.J. (1976) Do authoritarians hold authoritarian attitudes? Human Relations, 29, 307-325.

Ray, J.J. (1979) Authoritarianism in Australia, England and Scotland. Journal of Social Psychology 108, 271-272.

Ray, J.J. (1980) Authoritarianism and hostility. Journal of Social Psychology, 112, 307-308.

Ray, J.J. (1981a) The new Australian nationalism. Quadrant, 25(1-2), 60-62.

Ray, J.J. (1981b) Do authoritarian attitudes or authoritarian personality reflect mental illness? S. African J. Psychology 11, 153-157.

Ray, J.J. (1982) The Protestant ethic in Australia. Journal of Social Psychology, 116, 127-138.

Ray, J.J. (1983) Reviving the problem of acquiescent response bias. Journal of Social Psychology 121, 81-96.

Ray, J.J. (1984a) Authoritarian dominance, self-esteem and manifest anxiety. South African Journal of Psychology 14, 144-146.

Ray, J.J. (1984b) Attitude to abortion, attitude to life and conservatism in Australia. Sociology & Social Research 68, 236-246.

Ray, J.J. (1985) Acquiescent response bias as a recurrent psychometric disease: Conservatism in Japan, the U.S.A. and New Zealand. Psychologische Beitraege 27, 113-119.

Ray, J.J. (1985) Authoritarianism of the Left revisited. Personality & Individual Differences 6, 271-272.

Ray, J.J. & Bozek, R.S. (1980) Dissecting the A-B personality type. British Journal of Medical Psychology 53, 181-186.

Ray, J.J. & Kiefl, W. (1984) Authoritarianism and achievement motivation in contemporary West Germany. Journal of Social Psychology, 122, 3-19.

Ray, J.J. & Pratt, G.J. (1979) Is the influence of acquiescence on "catchphrase" type attitude scale items not so mythical after all? Australian Journal of Psychology 31, 73-78.

Ray, J.J. & Ray, J.A.B. (1982) Some apparent advantages of sub-clinical psychopathy. Journal of Social Psychology, 117, 135-142.

Rorer L. G. (1965) The great response style myth. Psychol. Bull. 63, 129-156.

Rosenberg M. (1965) Society and the Adolescent Self-image. Princeton University Press, Princeton, NJ.

Vagt G. and Wendt W. (1978) Akquieszenz und die Validtaet von Fragebogenskalen. Psychologische Beitraege 20, 428-439.

------------------------------

[*] The- assistance of Frances Lovejoy in carrying out this survey is gratefully acknowledged.




Go to Index page for this site

Go to John Ray's "Tongue Tied" blog (Backup here or here)
Go to John Ray's "Dissecting Leftism" blog (Backup here or here)
Go to John Ray's "Australian Politics" blog (Backup here or here)
Go to John Ray's "Gun Watch" blog (Backup here or here)
Go to John Ray's "Education Watch" blog (Backup here or here)
Go to John Ray's "Socialized Medicine" blog (Backup here or here)
Go to John Ray's "Political Correctness Watch" blog (Backup here or here)
Go to John Ray's "Greenie Watch" blog (Backup here or here)
Go to John Ray's "Food & Health Skeptic" blog (Backup here)
Go to John Ray's "Leftists as Elitists" blog (Not now regularly updated -- Backup here)
Go to John Ray's "Marx & Engels in their own words" blog (Not now regularly updated -- Backup here)
Go to John Ray's "A scripture blog" (Not now regularly updated -- Backup here)
Go to John Ray's recipe blog (Not now regularly updated -- Backup here or here)

Go to John Ray's Main academic menu
Go to Menu of recent writings
Go to John Ray's basic home page
Go to John Ray's pictorial Home Page (Backup here)
Go to Selected pictures from John Ray's blogs (Backup here)
Go to Another picture page (Best with broadband)