May 3, 2017

Optimizing Accuracy in Surveys

One of the most challenging tasks in doing good research is asking questions in a way that is clear and that does not "lead" the respondent to a particular conclusion.

One of the first levels of minimizing bias is what we refer to as "positional bias."  That is, when a survey respondent is presented with a number of choices to "select all that apply, "choose one answer," etc., the answer choices are randomized for each participant so that there is not implied bias based on which choices appear first in a list.  Webtorials has employed this method for several years.

Questions in which the respondent is asked to indicate whether they agree or disagree with a particular statement are particularly challenging.  It is natural in many cases to "agree" rather than "disagree."

In our most recent research for the "2017 Workplace Productivity and Communications Technology Report," an "opinion" section of the questionnaire contained statements that were presented n two forms.  The first is a "positive" version of the statement, such as "My organization is investing in tools that make me feel more productive," and respondents were asked to agree or disagree.  The other was an inverse or negative version to the same statement, such as "My organization is not investing in tools that make me feel more productive."  In most cases the negating word(s), such as not, were highlighted and/or in a different color.  Again the respondents were asked to agree or to disagree.


The purpose for asking questions in that manner is to test for whether respondents are agreeing or disagreeing out of habit.  Indeed, in all cases, a higher percentage of respondents agreed than disagreed with the statement, regardless of the form in which it was asked.  Thus, presenting the results from both forms of the question will result in a more accurate reading than just one version.


Whenever possible, essentially equal numbers of respondents saw the statements in each form.  For those responding to "native language" versions of the questionnaire in French or German, the statements were translated.  Additionally, the overall order in which the statements were presented was randomized for each respondent.


In reporting the results of these questions, we will be using a graphical method by which the primary/solid slices of the pie indicate the respondents who agree with the version of the question.  The respondents who disagreed with the statement are shaded, and the color matches the equivalent feeling.  Labels show the combined percentage answering with equivalent answers, providing a balanced view of the two sets of answers.


Additionally, the table below each chart shows the split of answers for each version of the question.

Leave a comment

Search Webtorials

Get E-News and Notices via Email




I accept Webtorials' Terms and Conditions.

Trending Discussions

Featured Sponsor Microsites



Please note: By downloading this information, you acknowledge that the sponsor(s) of this information may contact you, providing that they give you the option of opting out of further communications from them concerning this information.  Also, by your downloading this information, you agree that the information is for your personal use only and that this information may not be retransmitted to others or reposted on another web site.  Continuing past this point indicates your acceptance of our terms of use as specified at Terms of Use.

Webtorial® is a registered servicemark of Distributed Networking Associates. The Webtorial logo is a servicemark of Distributed Networking Associates. Copyright 1999-2018, Distributed Networking Associates, Inc.