Survey design tips & tricks – Part 2

After we had published our “Survey design tips & tricks” recommendations we received a lot of feedback asking if we can continue the list and give some more examples of what one needs to take care of when creating a Limesurvey survey.

So we got some coffee and had a team meeting discussing further issues worth mentioning. Here are 10 more tips & tricks:

11. Be careful when using matrix questions

Matrix – also called array questions – should be used carefully. They are hard to answer for the user if you add too many sub-questions. Recommendation: Repeat headers at larger matrix questions to help users answer such complex questions and keep the sub-questions as short as possible. This setting can be found at the advanced question settings of each array question.

12. Mark keywords in bold

Within lengthy question or answer texts you should mark important keywords in bold. Users are lazy and don’t like reading lengthy texts or explanations. Highlighting the most important parts helps them and improves the completion rate.

13. Force users to agree to terms and conditions

Some surveys may require that the user agrees to certain terms and conditions. There may also be surveys for which you may want them to mark that they have understood important fill-out notes like “I have understood that…X, Y, Z”.

For such cases use multiple choice questions and set the “max answers” question attribute accordingly to force users to check the box of each item (“Please mark all”).

Alternatively, you can add the terms and conditions at the start page and rename the Limesurvey navigation buttons to “I agree / I do NOT agree”.

14. Use conditions/branching

Limesurvey supports complex conditions and you should make use of it. You wouldn’t want to ask a person who doesn’t have children about their preferences for childcare. So how about adding a “Do you have children?” question and only ask such questions the appropriate target group?!

You can even use conditions based on additional token attributes if you are designing a closed survey and information about gender or age is available up front.

Asking users questions they can’t/don’t want to answer is a survey design worst case.

15. Use screen-outs

If you are interested in data of e.g. young people only, don’t waste other peoples’ time. Add a question asking for their age at the very beginning and set a “zero-quota” to screen out people selecting an answer out of range of your preferred age.

16. Use real time data validation

Instead of cleaning up data later make use of Limesurvey’s real time data validation features:

  1. Email addresses can be checked for valid syntax using a regular expression.
  2. Date ranges can be limited by setting a min/max date. Such date ranges can even be set dynamically. Using the Expression Manager you can e.g. set the max date to today’s date.
  3. For numeric input questions you can also set reasonable min/max limits e.g. when asking for height/weight.
  4. When asking for percentage values, set a max value of 100 and make sure the sum is not greater than 100.
  5. You can reference previous answers to set the max of the current question to a value that was entered at a previous question

17. Always use similar scales

You should always use similar scales within a survey. Users will be confused if question 2 uses a 1-7 scale, question 5 uses a 1-5 scale and question 10 uses a scale ranging from 1 to 10. It also makes comparing data far more difficult. So when designing your survey think of scales which you can use for all questions.

In addition to that you should also make sure to never change the direction of scales. Only for certain psychological experiments that may make sense but users may not even notice that so when doing so show them according hints.

18. Use an even number of items for scales

Scales should always use an even number of answers to force users to give a tendency. At a 1-5 scale people tend to marking the “3” answer in the middle which doesn’t really tell you much.

Instead, use an even number of items at your scale like:

“very good” – “good” – “bad” – “very bad”

19. Offer “save and resume later” option

Especially for lengthy surveys you should enable the “save and return later” feature at “Survey Settings” -> “Notification & Data Management”. Users may be interrupted, may have to leave suddenly or may receive a phone call. Offering an option to save their answers prevents them from just closing their browser or suffering from a session time out.

20. Give feedback about the user’s progress

The user should always be aware of his/her progress when answering a (lengthy) survey. Thus, you should always show a progress bar and also offer additional information like “Just some final questions about XYZ and then we are nearly done.” or “This is the last page of questions, we are nearly there.”

Survey design tips & tricks

Designing your own survey is pretty easy when using Limesurvey. Unfortunately, creating a really bad survey is also pretty easy. That’s why we have set up a list of survey design tips & tricks. Based on our experience with more then 100 survey projects per year the following recommendations hopefully help you designing better surveys.

Motivate users to raise the completion rate

A user being presented with a simple “Please help me by filling out the survey, it will take 10-15 minutes.” will see no benefit in wasting his/her time with this. In such cases an incentive like “All users which fill out the survey completely will be allowed to take part in a lottery and can win one of 5 Amazon vouchers worth 20,00€.” can raise participation and completion rates significantly.

Start with simple questions

A survey should never start with complex questions people have problems answering at all. Instead, place very simple questions like demographic questions at the beginning. Everyone can answer those and they present an easy introduction. Once people have started filling in simple data they tend to finish their work and not closing a survey early.

Questions should be clear

This may sound pretty obvious but for every question make sure it is clear and precise, collectively allowing for detailed, unambiguous and meaningful answers.

Add screening questions to the start

Nothing is more annoying than having completed several questions asking for complex information and being told later that for certain reasons like your age or gender you were suddenly screened out and are not allowed to complete the survey and maybe receive a certain incentive.

If users need to agree to certain terms and conditions or have to be of a certain age or gender you should always place such screen out questions at the beginning.

Add problematic questions to the end

Similar to placing easy questions at the beginning you should place problematic questions (e.g. about illnesses) at the end. People having already completed 80% of your survey will most likely answer such questions as well.

The fewer questions the better

People tend to ask this and that instead of focusing on the really important questions only. Always make sure a certain question is really important. The fewer questions the higher the completion rate!

Use free text questions to ask for additional information

Open questions may reveal unexpected and very worthy additional information. You should add optional text questions at all important parts of the survey, asking users “Is there anything else you want to tell us about XYZ?”. This may lead to important user feedback you would not have received when using closed questions only.

Make sure your questions are non-suggestive

Asking users “What is your opinion about XYZ?” uses neutral speech while a question like “Don’t you agree that LimeSurvey is a really great tool?” is a very suggestive question which will cause users to give more positive feedback.

Only make the most important questions mandatory

Having to complete a survey at which each question is mandatory can be really painful, especially when it comes to complex matrix questions.

When designing a survey you always need to ask yourself “Is this question important enough to make it mandatory?”. Only the questions related to data you really need should be mandatory.

Keep questions and answers short and easily readable

Nothing to add to this…

Note: There is a second blog post “Survey design tips & tricks – Part 2“.

 

BTW, if you are not very familiar with designing surveys but need to create an important one, we can help with that. Our experience can help you at creating a great survey with a high completion rate.
Just drop us a note if we should review your survey or set it up for you.

 

How to re-order the “other” option at a Limesurvey survey

[Tested with Limesurvey v2.06LTS]
Several Limesurvey question types allow adding an option to enter a free text answer. This option can be enabled when editing a question (“Option ‘other’ -> Yes”).
By default Limesurvey always places that option at the very end of the list of answers and there is no feature to change the order of answer options.

To place the “other” option at the very beginning of the list of answers or at any other position one needs to use some custom JavaScript. We have developed such a snippet for you which is easy to adjust and allows you to easily add the code to your survey and adjust it as necessary.

Assuming your answer options are using answer codes “1”, “2”, … you can use the following code snippet to place the “other” option before the 6th answer option (“none of these”):

<script type='text/javascript'>
$(document).ready(function() {
/* Adjust these settings */
 var move_element="other";
 var place_before="6";
/* Please do NOT change the lines below */
var place_before="javatbd{SGQ}"+place_before;
var move_element="javatbd{SGQ}"+move_element;
$("#"+move_element).insertBefore("#"+place_before);
});
</script>

The best approach is to insert the code in source code mode of the integrated editor:
Source code mode of integrated Limesurvey editor

Using the survey preview you should now see something like this:

You can simply place the “other” option on top of the list by adjusting the code above as follows:

/* Adjust these settings */
 var move_element="other";
 var place_before="1";

You can find the used JavaScript (you need to switch to source code mode to see it) at the help texts of this demo survey: Limesurvey-re-order_other_option_demo_survey (use right-click -> “save link” to save LSS file!).

If you want to place a certain item of a randomized list of answer options to the end of the list, have a look at How to place a certain option at the end of a list with randomized answers at Limesurvey survey.

Adding “I agree” button at the beginning of your Limesurvey survey

Sometimes you need participants to agree to certain terms and conditions before starting to take a survey. There is no build-in feature for this at Limesurvey, but we can easily work around this with some lines of JavaScript. Here we go…

Turn of XSS filter at Limesurvey

To be able to add custom JavaScript at a certain survey question or text, we first need to disable the internal Limesurvey XSS filter. So when being logged in at Limesurvey go to Global Settings -> Security -> “Filter HTML for XSS” and set it no “No”.

Limesurvey XSS Filter

Adding your terms and conditions

Once you have turned off the filter you can add your terms and conditions to your survey’s welcome text. Click the edit icon and select “Edit text elements”. Add your text at the “Welcome message” text box.

Adding the “I agree” / “I do NOT agree” buttons

When a user calls your survey, he will at first be presented the welcome text holding the terms and conditions. There will also be a “next” button to start the survey and on option to “Exit and clear survey”. So instead of re-inventing the wheel, we simply rename these buttons for the welcome page only.

For this you need to go to show the welcome message in source code mode. You’ll see the terms and conditions you entered previously. Go to the end of that text and add the following code there to rename the buttons:

<script>
 $(document).ready(function()
 {
    $('#movenextbtn').attr('value','I agree!');
    $('.clearall').attr('value','I do NOT agree!');
 });
</script>

Of course, you can change the “I agree!” / “I do NOT agree!” texts if you wish.

The result:
Custom Limesurvey button labels

 

How to validate email addresses at Limesurvey

Update (12/2018): Since these days domain names can not only hold special characters like “ü” but also use more than 3 characters as domain name like .tech or .company, we have updated our email regex to only check if “@” and “.” exists within the entered email address.


At many surveys you want to ask the user for his/her email address, be it for a lottery or to add the email address to your address data. If you later want to send an email to such an email address, a simple typo or missing @-sign will cause problems and you might not know how to correct these invalid email addresses.
So let’s be careful right from the start and do the email validation right when asking for the data. We will show you different approaches below.

Limesurvey email validation using a regular expression

The easiest way to validate an email address is to use a regular expression which checks for certain syntax details like the @-sign, a dot and a valid domain. You can find such regular expressions (and many more) at the Limesurvey manual section about Regular Expressions.

For this example we use a simple text question and the following regular expression which has to be added at the “validation” field when editing the question at Limesurvey:

/^.+?@.+?\..+$/

Note that when using regular expression at Limesurvey, these always have to start and end with a slash (“/”).

Unfortunately, this solution can’t auto detect typos, so let’s create an advanced example at which the user has to verify the email address he has typed in.

Verify email addresses at Limesurvey

We can extend the above example by adding a second text question which also uses a regular expression for validating the second email address. To give some feedback about whether the email addresses the user has entered really match, we will also add two questions of type “text display” with texts like “Sorry, these email addresses don’t match. Please correct.” (warning message) and “Email addresses match, well done!” (success message).

Of course, we only want to show this feedback after the user has entered some texts at both fields. To achieve this we add the following relevance equation to both text display questions, assuming the email questions use question codes “email1” and “email2”:

((! is_empty(email1.NAOK) and ! is_empty(email2.NAOK)))

This equation should now be extended so that the warning only shows up if both text inputs don’t match and the success message is only shown if both strings are equal:

Warning message:
((email1.NAOK != email2.NAOK and ! is_empty(email1.NAOK) and ! is_empty(email2.NAOK)))

Success message:
((email1.NAOK == email2.NAOK and ! is_empty(email1.NAOK) and ! is_empty(email2.NAOK)))

This is not yet a perfect solution because the user can still proceed, even if the warning message is shown. So let’s improve this a little at the next example.

Interrupt a Limesurvey survey if email addresses don’t match

To stop the user from continuing a survey even if the email and email verification address don’t match we have to create our own validation function. Since Limesurvey 1.92 this can be done by using the Expression Manager and the “Whole question validation equation“. We can even create our own error messages.

The difference at the survey design is that we are now using a multiple short text question type to have two text fields within the same question for later question validation.

The validation equation we want to set up should check

  • if text was inputted at field 1 (email address) -> field may not be empty
  • if text was inputted at field 2 (email verification) -> field may not be empty
  • if the inputted text strings match the regular expression for valid email addresses
  • if both text fields contain the same strings -> emails_1 == emails_2

Assuming that the question code of our question is “emails” and the sub-question codes are “1” and “2”, we can use the following string at the “Question validation equation” field which can be found at the advanced question settings:

(!is_empty(emails_1.NAOK)
AND
regexMatch('/^.+?@.+?\..+$/',emails_1.NAOK)
AND
(!is_empty(emails_2.NAOK)
or
regexMatch('/^.+?@.+?\..+$/',emails_2.NAOK))
AND
(emails_1.NAOK == emails_2.NAOK))

To show different error messages according to the current state of the two input fields, we can put the following equations into the “Question validation tip” below:

{if(is_empty(emails_1.NAOK)
or
regexMatch('/^.+?@.+?\..+$/',emails_1.NAOK),
'','Please enter a valid email address.<br />')}
{if(is_empty(emails_2.NAOK) or
regexMatch('/^.+?@.+?\..+$/',emails_2.NAOK),
'','Please verify the email address.<br />')}
{if(!is_empty(emails_1.NAOK)
and
!is_empty(emails_2.NAOK)
and
emails_1.NAOK == emails_2.NAOK,
'Email addresses match!',
'Email address and email verification do NOT match!')}

That’s it, feel free to pick the solution which matches your requirements.

Limesurvey – Conditions based on token attributes

Sometimes you want to show certain questions of a survey to a specific user group only. With Limesurvey this is no problem. If your survey is set to be non-anonymous and uses tokens, you can use user defined attributes for conditions. Here we describe the necessary steps.

1. Adding additional attributes

Set the survey to be non-anonymous:
Limesurvey survey settings

Enable tokens:
Enable tokens at Limesurvey

Add a certain number of user defined attribute fields:
Add token attributes at Limesurvey

Assign attribute names after having created the new fields:
Label additional attributes

Add some dummy tokens with user defined values

Add dummy tokens

Set conditions for a certain question

Go to the question and call the condition designer:
Limesurvey condition editor

Select the token field on which the condition should be based on (e. g. “Department”) and define the constant which the condition should match (e. g. “Management”):
Condition based on token attribute

Check condition

You can use the group preview for a first check if the condition was stored successfully,
Limesurvey group preview

because in this case the question will not be shown at the group preview (because the conditions isn’t met). The arrow indicates where the question was previously shown before setting the condition:

Test survey at runtime

Once tokens were assigned when adding users or by using the “Generate tokens” feature, the “Display tokens” screen contains an icon to test each token entry:
Limesurvey token overview

Of the above users, only “Jane” belongs to the department “Management”. Let’s check the different survey behavior to check if the condition works as expected:

Survbey at runtime Survey at runtime

More information

 

How to randomly show 1 of X questions at a Limesurvey survey

Limesurvey offers various randomization features. One common randomization use case is showing different versions of a question at a Limesurvey survey. To do so, you need to randomly show one of the X question versions to the user so the first step would be to generate a random number between 1-X. For this we can use the powerful Limesurvey Expression Manager which is available since release 1.92.

The new question type “equation” can be used to store the result of the random function. So create an “equation” question with question code “randnumber” and the following content at the question text field, which will generate a random number between 1 and 2:
{if(is_empty(randnumber.NAOK),rand(1,2),randnumber.NAOK)}

The whole code basically means that the system should first check if the rand variable was already populated. If it is still empty, a random number between 1 and 2 is created. Else, the already created number is kept and no new random number gets created.

If you want to hide the question from the user, set “Always hide this question” to “yes” at the advanced question settings.
Now we have a question that will be filled with either “1” or “2”, we can use conditions at following questions to show version A or version B of a question. For this you can use the Limesurvey condition manager and set a condition based on a constant value of 1 or 2 for the equation question. Alternatively you can use the “relevance” field to define when a question should be shown. The appropriate relevance reads:
((randnumber.NAOK==”1″))

…and for the second question:
((randnumber.NAOK==”2″))

Don’t forget to check if you set up everything correctly by having a look at the survey logic file.

…and if this still sounds too tricky, we can set up the survey for you.

How to prevent Limesurvey email invitations or reminders being treated as spam

Spam really is a problem these days and without spam filters, email would hardly be useable. But this also causes problems when sending email invitations or reminders at Limesurvey. Often users complain not having received an invitation/reminder email because the user’s spam filter treated the email as spam.

Spam filters

If an email is treated as spam mostly depends on the spam filter being used and its settings. A spam filter usually checks the sender address and the email subject and content, so there are ways and means to influence how they treat your Limesurvey invitation and reminder emails.

Tips & Tricks

(The following list will be updated every now and then)

Sender address

  1. Use a valid sender address (can be set at the Limesurvey survey settings -> General -> Admin email).
  2. Do not use a sender address from a one time email provider. These are often used for spam and therefore blacklisted by most spam filters. Email from email providers like Gmail, Hotmail, GMX and the like might also be considered as spam quite often because everybody can get an account there.
  3. If possible use a sender address which has been in use for years and was never related to any spamming, e. g. your company’s email address.

Sending emails in batches

If you have to send several hundred or even several thousand emails this can really be painful. When sending them all at once using a huge email batch size (see details below at “Mailserver”) the script will most probably time out. Furthermore, there is a high risk of being treated as spammer so your IP address or domain could be put on a blacklist. And if you send them manually you will need a lot of coffee to not fall asleep when clicking the send button 50, 70 or 100 times.

The best approach for sending a huge amount of emails is using cronjob scripts. For this we have coded a script to automtically send email invitations. There is also another script for automatically sending email reminders.
You just upload the script to your server and set which survey(s) the script should deal with. Afterwards, you can set up a cronjob to call the script every X minutes. So if you need to send 3000 emails, just set the batch size to 50, call the script every 5 minutes and within 5 hours (50*12*5=3000) you are done without any further user interaction required. Of course these scripts only send an invitation once so users do not get spammed.

Sending emails this way significantly reduces the risk of your emails being treated as spam or being blocked. If your provider sets a certain limit of max X emails every Y minutes you can easily work around that by adjusting the cronjob details accordingly to not hit such limits.

Email content

  1. The content of your email is probably the most important part. Do never use any words like “free”, “buy”, “sex”, “shop”, “viagra” or the like at your email invitation or email reminder texts. You can change the default texts at the “email templates” link of the survey options menu.
  2. Same for the email subject: Use a meaningful text that maybe doesn’t use the word “invitation” because it can be found at spam emails quite often.
  3. If you send the exact email text several times, this might also look suspicous to spam filters. The solution is to use the {FIRSTNAME}/{LASTNAME} email placeholders at the Limesurvey email templates to create some unique strings.
  4. Often it’s worth a try to switch from HTML emails (Limesurvey default setting) to raw text emails. At Limesurvey this can be changed at Survey settings -> Tokens -> “Use HTML format for token emails” = “no”.
  5. Spammers often use images for fake emails, so better not use any images at your emails.
  6. Of course you want to add a link to the survey at your invitation and reminder emails. But generally links should be reduced to a minimum and opt-out links are often used for fishing, so if you are facing serius spam problems, not adding any links at all might help.
  7. When using HTML emails and editing the email messages, check that the HTML being used has a valid HTML syntax, e. g. all tags should be closed properly. There are several HTML validators available, e. g. the official W3C validator where you can copy paste the HTML code. You can view the HTML source code text by switching the editor to source code mode:
    Source code mode of integrated Limesurvey editor

Mailserver

  1. Emails can either be sent by the webserver using PHP emailing which is the Limesurvey default setting or by connecting to an SMTP server. Since PHP emailing is mostly used for spam, we recommend to switch to SMTP if you are facing spam problems. At Limesurvey this can be configured at Global settings -> Email settings -> “Email method” = “SMTP”
  2. Especially when using “PHP” email setting, some webservers will block emails if a certain amount of emails is send in a minute/an hour/a day. So make sure that when manually sending the emails the batch size is not too big. This can either be set when sending email at the token screen or by editing Limesurvey’s default setting (50) at Global settings -> Email settings -> “Email batch size” = “XYZ”.
  3. If possible use a webserver which hasn’t been connected to spamming anyway. Spam filters also analyse the IP range a server belongs to and if there have been spam issues with this range before. So using your company’s server or the SMTP server of known providers such as Gmail should help.
  4. A wrong mailserver configuration also influences spam filters. There are several different settings that could be wrong. We are currently collecting some of them and will update this section later.
    The experts out there can have a look at the email source code to check some of the sender server configuration settings like email header, spam score (if available), …

Additional information and tools:

Cronjob scripts for automatic email sending

Testing

Limesurvey forums / Limesurvey manual

How to use JavaScript at a Limesurvey template or survey

Limesurvey offers a lot of on board features but for certain issues you can add JavaScript to your survey to extend its abilities. Particularly when it comes to on-screen data validation, data filtering, setting default values or any kind of randomization, using JavaScript is the way to go and we have collected several workarounds in the Limesurvey wiki to deal with different problems.
Since some users have problems embedding JavaScript in their surveys, we want to give some examples on the different approaches.

To avoid re-inventing the wheel, Limesurvey makes use of the JavaScript library jQuery. This library is documented very well and we recommend having a look at some tutorials and the official documentation.

If you want to embed any of the jQuery based workarounds, then there should be a

$(document).ready(function())

function in the template.js file of your template where you can add all of this code so that it is only executed after the page is fully loaded. So option 1 to add JavaScript to your survey is to simply place it in the template.js file of your template.

Pros:

  • You don’t need to disable the internal Limesurvey XSS filter if you add the code directly at the template.js file.
  • You can add your code ot the existing $(document).ready(function()).
  • It is very easy to update/modify.

 

Cons:

  • The file is loaded at every page call, so it might cause some overhead if you only need the script for a certain survey question(s).
  • The code will be loaded for all surveys that use the template, so don’t add survey-specific JavaScript to any template which may be used for other surveys, instead create a copy of the template.

 

The second option is to add the JavaScript/jQuery code directly to the affected question or group. This can easily be done by embedding it in the question/group text for which you have to switch to source code view of the integrated editor:
Source code mode of integrated Limesurvey editorFurthermore, you have to disable the Limesurvey XSS filter at
Global settings -> Security -> “Filter HTML for XSS” = “No”.

Once you have done this, you can use the following code snippets. Copy them to the affected question/group text in source code mode and test if the alert message appears:

1. Embedding Javascript

<script type="text/javascript" charset="utf-8">
    alert("Test!");
</script>

2. Adding jQuery code

<script type="text/javascript" charset="utf-8">

	$(document).ready(function() {

		alert("Test!");
	});
</script>

Pros:

  • The code will only be loaded once (for the affected group/question).
  • You can easily have several similar code snippets and just change some IDs in the code without having any overhead.
  • Limesurvey placeholders and Expression Manager tags are parsed.

Cons:

  • You have to disable the internal XSS filter for which requires admin access to the Limesurvey installation.
  • The code will be split over several questions and gets harder to maintain if you add many different script snippets.