Kupferman.com

Qualtrics Review (Online Survey Software)

Qualtrics survey software from Utah-based Qualtrics is an excellent web-based survey software package that offers a fantastic array of question types, a well-designed survey development interface, good fielding/survey promotion capabilities and a powerful reporting engine. It has both panal management features and multi-users capabilities and should definitely be a contender if you’re a corporate research department or academic organization looking for a survey system. 

Questionnaire Features

Qualtrics offers a lot of different questions types. More, perhaps, than I remember seeing in any other program. A lot of these are variations on old favorites (there are 21 different multiple-choice question types, 13 matrix type questions, 11 constant-sum type questions, 12 rank order type questions and 5 different text entry questions — and that doesn’t even include the “Pick, Group and Rank” questions, the “Side by Side” questions or the “Drill Down” questions. Without listing all of them in all of their variations, I think it is safe to say that Qualtrics has all of the standard question types covered.

[Please note: this review covers Qualtrics’ new survey development engine, which is still in “preview” mode. There may or may not be more features to come that aren’t described here]

Question types are extensive. In terms of special question types, Qualtrics has a variety to choose from. There are a variety of easy to build matrix questions/table questions that allow you to easily create bi-polar questions, Likert scales, text entry matrices, and even scales with different descriptions above every box in the matrix. There are constant sum questions that instead of asking you to enter numbers, it instead has you draw bars of different lengths. It is very well done, and goes a long way to ensure that your respondents have a good survey taking experience. Another great example of a well designed question type is the ranking question, which lets you drag the items in list into whatever order you want (much more fun than entering numbers or selecting values from drop down lists).

Qualtrics Question Types
Qualtrics offers over 70 differeny question types

Qualtrics isn’t the first to offer a sliding scale question type, but they are the first I’ve seen to let you control little gauges with your slider. Or a stop light. Or a smiley face (you can control several different variations of smiles that work their way into a frown).

Qualtrics has a wonderful “Pick, Group and Rank” question type that lets you present the user with a number of items in the column on the left and then lets them drag each item into a number of different bins you create (you can create as many options or as many bins as you like). One variation of this question type even lets the respondent define each bin themself.

These types of interactive question types are wonderful not just because they make your survey more fun to take, but because they let you be much more creative and efficient in how you collect your data. I can’t tell you how many times I’ve struggled to figure out how to mimic a methodology that works fine in real life but is difficult to execute using just drop down boxes or radio buttons. In my opinion, the more question types you have available, the better.

The Qualtrics survey system has a few more question types that are worth noting — the “drill-down” question type does exactly what it says — your survey might ask for a country, and after they do the dropdown right below shows them cities within just that country to choose from. “Gap Analysis” questions let respondents answer a question and then automatically asks them to explain their answer if they rate it positively or negative (depending on which you choose). Qualtrics also has a “timer” question type that will keep track of how long they spend on a page, and a “Meta” question type that will automatically insert all of the respondent’s browser related information in the response data (browser type, browser version, operating system, screen resolution, etc).

Qualtrics offers a “Conjoint” type question which asks you to first specify all of your attributes and then the levels for each. It is fairly easy to configure, although inserting it in the survey is a little awkward (it doesn’t appear into your list of questions — you have to insert it using the “Survey Flow” page). It appears to use a form of adaptive conjoint methodology (which includes a little max-diff) which can be great for evaluating product features but not so accurate for pricing research (in adaptive conjoint pricing tends to be understated in importance compared to choice-based conjoint). You should also be “warned” that the conjoint question exports only the raw data and does not calculate part-worth utilities for you — which means that you will need to have some other methods of doing the analysis (if you don’t have such a method then the conjoint question will be fairly useless to you). My impression is also that there isn’t enough documentation for the conjoint question to make it really useful to many researchers.

Qualtrics conjoint capabilities
Example of Qualtrics’ Conjoint Question

Qualtrics also offers a “Conjoint” question which actually seems pretty robust. Instructions for using it weren’t available and I would have to play with it a bit more to understand its methodology (if you’re looking for conjoint capabilities, you might want to play with it and ask questions).

Although Qualtrics makes it is exceptionally easy to create a basic survey without delving into any of the advanced options, there are a lot of advanced options for every question type. Question text and responses can all handle embedded HTML code, which offers a lot of flexibility. In multiple choice questions, for example, you can edit and easily rearrange answer choices. You can randomize some of the choices or all of the choices, and you can insert choices that respondents selected in other questions. You can even assign “recode” values to all of your questions so you can show the respondent a text value but actually export a number you assign to your data file. This is exceptionally useful if you’re trying to match your values to another table. You can even give each of the answer choices a name, and every question can have a unique ID that you assign.

A variety of different validation types are available in Qualtrics. Multiple choice questions, for example can require that a certain number of responses are chosen or that no more than a certain number are chosen. Text fields can be validated to ensure that the contents are a valid e-mail address, or a valid postal code, or even a valid state code or date. One (minor) limitation here is that you can’t validate against regular expressions — but frankly, I’m not sure how many people would use that feature if it were available.

Qualtrics features [tag]skipping[/tag] and [tag]branching[/tag] which appears to be both friendly and functional. For example, if I click the button to add skip logic, it automatically inserts not just a “Skip Logic” item, but also a page break (skip questions in almost all software packages only work when next to a page break, and most make you insert them manually). Then, using drop down boxes, it has me describe what I want to do: If Blue is Not Selected then skip to What Color Are My Shoes? [Done] … where the bold represents drop down boxes that tie to questions in my survey. Very easy, doesn’t require a lot of thought.

Qualtrics offers more advanced survey flow options, such as [tag]advanced branching[/tag], [tag]looping[/tag] and [tag]section randomization[/tag] is accomplished through the use of the “Survey Flow” page, which while functional struck me as somewhat less intuitively designed as the rest of the survey editor. It allows you to set up any number of question blocks which can be individually randomized. It also allows you to insert advanced survey elements such as the conjoint question mentioned earlier.

The ability to create multiple criteria for a branching decision (for example, if you only want to branch if their favorite color is red AND they like to eat pizza) is available, but it is not nearly as easy to implement as the basic branching.

Other advanced survey elements in Qualtrics include the capability to set up survey [tag]quotas[/tag] based on individual responses. That is, you can have certain questions displayed only if certain other questions have been answered a certain way a specified number of times; or, alternately, you can display certain groups of questions only if a specified question has not been answered an indicated number of times.  This is a neat feature — my only concern here is that the criteria definition capabilities are limited — you can’t, for example, combine multiple conditions.

Qualtrics doesn’t offer any scripting or advanced programming capabilities. While most potential users probably won’t consider this an issue, it will definitely be a concern to high end advanced users who want absolute control over survey flow (for example, sometimes I like to bounce back and forth through a survey). Nor does Qualtrics offer an API that lets you work directly with the flow and the survey data mid survey. But these are fairly advanced features, and most people won’t know that they’re missing (if you’re the type of person who would notice that they are missing then you probably want to look at something more advanced).

Perhaps the best way of summing up the questionnaire related capabilities and features of Qualtrics is to say that it is a great survey package for someone who wants to write great looking surveys that are fun/easy to take where you have a lot of options in your questionnaire development. Or it is good for someone who doesn’t care about advanced question types but who wants good looking surveys that are easy to write.

Qualtrics Survey Development Features

Qualtrics offers a very friendly environment for basic survey development. In addition to a very clean interface, it uses a lot of [tag]AJAX[/tag] in its interface, which makes the experience of developing your survey very pleasant and visual. The annoyances I experienced were mostly related to the integration of the advanced features — if you’re just doing basic survey development you should be very pleased with the experienced offered by Qualtrics.

I should point out that this survey development environment is new. The old Qualtrics survey development environment — especially the “Survey Wizard” was very slow and very frustrating. This new version is completely different and much better.

Creating and editing questions is absolutely painless in Qualtrics. There are very few postbacks to slow you down. When you press the “Create a New Question” button the new question pops onto the screen without having to refresh the page. Adding choices to multiple-choice questions or editing the basic question options for each question type have an instant impact on the question as it appears in your preview mode. This is especially helpful for more complex question types such as matrixes which most programs force you to guess at what the finished product is going to look like.

Qualtrics Development Environment
The Qualtrics development environment

Setting the question type in Qualtrics is also fun because as you hover over each question type it shows you samples of what the question will look like which are pretty close to the same thing. This will make your survey development process go much faster.

Copying and moving questions within the survey is extremely easy and the software offers several different ways to do it. Deleting questions is easy too, and Qualtrics gets extra points for saving questions you delete in the “Trash/Unused Questions” bin which makes it very easy to restore a question you thought you didn’t need any more back to the survey.

Qualtrics does not allow you to import questions from Microsoft Word or a text editor which, depending on your personal survey development process, could be somewhat frustrating.

Qualtrics offers a very well executed [tag]library[/tag] that includes dozens of survey templates, question templates, graphics, and messages. You can use questions and templates created by Qualtrics or you can create your own. Adding questions to a survey that come from the library is very easy as well and is integrated into the new question process.

As I mentioned earlier, Qualtrics offers you advanced options for most question types. These appear in an embedded pop-up window so it doesn’t disturb your survey development environment (although it makes it pretty much impossible to see your survey while you’re looking at the advanced options — I don’t know how frustrating this would be in real life).

I got a little bit frustrated with how some of the advanced survey flow features were integrated into the Qualtrics questionnaire editor. While all of the basics are very well integrated into the survey design (including basic skipping and branching) items such as question randomization, survey blocks, and looping are handled on the “Survey Flow” page and were much more confusing to work with than the main survey features. For example, when I added a conjoint question set to my survey I got no indication of its existence on the main question development page. And while Qualtrics gives you the ability to add survey triggers that will send you an e-mail message if respondents choose a particular option in the survey, there is no indication of where and when these items occur within the survey itself.

In term of the look and feel of the surveys, Qualtrics doesn’t provide a lot of direct customization options. When creating a survey you can choose from a limited library of design schemes or you can use a custom scheme that the company develops for you. Depending on the package that you buy, a certain number of these custom design templates will come with the product, so unless you like to change your look a lot you will probably be fine with what they set up for you. Qualtrics does not let you add your own CSS files or page templates. Once again, these are only going to be issues for you if you are an advanced user who, perhaps, have a number of different clients who require their own personal look and feel.

Qualtrics lets you change the text of your “Next” and “Previous” buttons, add or remove a progress bar, and add survey-wide header and footer text to your survey. This is the extent of your control over the overall look and feel of the survey.

On especially wonderful option that I really appreciate is the ability to be able to re-edit a survey once it is published. For example, if my question is “What is your favorite color” and after publishing the survey I realize that I forgot to include “Yellow” I can easily go and add that choice. Or if I misspelled “green” I can go back and edit the existing choices. Many survey software solutions don’t let you edit the choice lists or the questions after a survey i fielded because (a) it could mess up how you interpret the results (if, for example, you changed “green” to “red” after collecting 300 responses) and (b) it messes with the way that they store data. It is a dangerous capability — but come on — we’re not children. If I want to mess up my survey, let me make that decision! I very much applaud Qualtrics for making it so easy for me to go back and make changes once a survey is in the field.

There is one thing about the Qualtrics interface that is somewhat frustrating: the “Back” button on your browser doesn’t work. Not only that, but if you push it, you get a “Webpage has expired” error message and the only solution is the edit the address bar (pressing the back button or the forward button once you get this error message just gives you more error messages).

Overall, the Qualtrics survey development environment is among the best, most friendly, most easy to use that I’ve seen and is perfect for the beginner and intermediate user. Advanced users — that is, advanced survey programmers who like to tweak anything and everything — may feel somewhat stifled. Those who like to regularly edit the overall look and feel of their surveys may not be happy with the inability to manipulate the basic page code or the cascading style sheets that control the look/feel/design of the page. Also, you can’t directly important surveys from Microsoft Word or a text editor — each question has to be built manually. But generally speaking, the survey development process was a pleasant one.

Qualtrics Data Collection Features

Qualtrics supports a variety of different ways to promote your survey including e-mail blasts, survey links that you post on your web site, popup surveys, embedded surveys and something called a “web site feedback link.” All of these are very easy to implement and provide you with several options for doing so.

I am personally always a little leery of using e-mail invitation systems built into survey systems. First there is the issue of allowing them to manage your list and ensuring that they properly follow the e-mail related laws. Then there is the question of whether or not they are doing everything in their power to ensure that your survey invitations don’t end up in the spam folder or get stopped by the ISP before they even make it to the spam folder. Which is why I generally tend to use either my own e-mail server (where I can control my own deliverability) or use a professional e-mail list company to make sure all of my messages get through.

So while the Qualtrics e-mail setup function is fairly straightforward, I had mixed results when it came to deliverability. One test message I sent went through just fine, the other got stopped by Outlook’s spam filters. Your experience will vary and there is nothing to indicate that Qualtrics’ e-mail server is any less reliable than other e-mail systems connected to other survey software programs. I can verify from looking at the header information that other than SPF header verification, Qualtric’s isn’t using any special deliverability services such as Sender Score or Habeas. Meanwhile, Qualtrics isn’t on any blacklists and according to SenderScore has a 90% e-mail acceptance rate with only a medium risk of non-deliverability which is better than SurveyMonkey(88% deliverability, high risk of non-deliverability) and much better than SurveyGizmo(33% deliverability, high risk of non-deliverability).

Qualtrics E-mail Deliverability Statistics
Qualtrics E-mail Deliverability Statistics

You can’t just upload an e-mail list and send to it — you need to first important your e-mail list to a Qualtrics panel (or you can type in the e-mail addresses one at a time, but I wouldn’t recommend it!). My demo account didn’t give me access to the panel management features of the system, so I couldn’t upload an e-mail list to test it out. I can say that it does at least have panel management capabilities and sample management capabilities and if/when you demo the software you will want to get more details about how these work.

Incidentally, every individual you send your e-mail to gets their own unique link which makes it fairly easy to track who has taken the survey and who has not. It is also possible to set the system to send out reminder e-mail messages.

Qualtrics also supports popup email invitations, and it is worth noting that the Qualtrics style of javascript popup is very good and is unlikely to be stopped by any popup software that your users may be using. I didn’t give it a full test, but in my experience this particular style of javascript popup is very hard to stop. And you can control how frequently it appears (does it show up for every user or for every 10th user?), whether or not it is animated and how big it is.

Qualtrics also lets you embed surveys in a web page on your site through the use of an IFrame (which is a little like embedding a mini web browser in the middle of a page). This type of execution can be prone to user issues because actions to the page in which the survey is embedded in can have a negative impact on the Iframe and the contents of the Iframe. However, whether or not this style of embedding is to some degree up to the taste of the web site developer.

Qualtrics gives you some control over the survey experience. You can, for example, control whether or not the respondent gets a “Previous Page” button, and you can allow respondents to save their surveys and return later. You can even control how long the partial survey results are saved. Surveys can be protected with a password (so only certain respondents can take it) or open to the public. Surveys can be set to automatically open at a certain time and then automatically close at another time. Responses can even be “bucketed” into multiple response sets. At the end of the survey, a default message can be displayed or you can redirect the users to another web site, or display a message page you store in your library.

Incidentally, Qualtrics also has the capability to do one question polls that you can embed on your web site. Exploring the details of this feature goes beyond this already-too-long review, but suffice it to say that the feature exists.

Qualtrics offers all of the basic data collection features one would expect from an online survey software package. It offers several different ways to promote your survey, and all of them seem to be executed well. I have some concerns that the e-mail invitation function does not appeared to be structured to ensure deliverability, but at the same time it is no worse than most other online survey software packages and better than many. For most users, Qualtrics is as good as anything they are likely to find in other packages.

Qualtrics Reporting/Data Management Capabilities

Qualtrics has put some time into the reporting capabilities of the software package, and overall it appears to be as friendly as the survey development tools. Qualtrics data management and export features are fairly basic, but will probably get the job done for most users.

Qualtrics makes it fairly easy to review the individual records in your dataset. The software makes it easy to see how long each respondent took to answer the survey, which is a quick way of telling if some respondents are clicking through and not reading the questions. Respondents you don’t like can be easily deleted. Qualtrics gets extra points for making it fairly easy to importsurvey results into the system (so that you can presumably use the Qualtrics reporting engine to create reports from data collected outside of Qualtrics, or so that you can merge outside data with the data you’ve collected in Qualtrics).

Qualtrics Reporting Environment
The Qualtrics reporting system

You can look at the responses collected from individuals, although I have to say it is executed in kind of a funny way — when I pull up a report on respondent #R_6h4QhkGskGxeU4Y, for example, it shows me each of their responses in the form of multiple graphs (one per question) which looks OK, but seems a little wasteful (what is the point of seeing a graph of one person’s responses?). Nothing wrong with the way that this is implimented — it’s just kind of odd.

Data can be exported from Qualtrics in a variety of different formats including CSV, XML, HTML, and SPSS. Warning about SPSS data export: it doesn’t create .SAV files — instead, it exports a CSV file and a syntax file which creates all of the variable labels and the value labels. While this isn’t necessarily a bad way of doing things it does add a couple of extra steps and there are, frankly, lower level packages out there that actually export .SAV files. Given Qualtrics’ focus on the academic community, it is a little surprising that they haven’t taken the extra step to allow for true SPSS export.

I would describe the reporting capabilities of Qualtrics as “friendly, mid-level, but not super advanced” You can, for example, create graphs, charts and tables. You can embed them in any number of custom reports which can be saved and rerun at will. You can even make reports that you create public, which is a great feature if you either want to be able to monitor your data collection without logging into the software or if you have clients whom you want to be able to share your results with without letting them log in.

You can control which questions appear in each report. You can make a variety of different types of charts ranging from bar charts to pie charts to even spider charts. In tables, you can control which choices appear in each table and the order that the choices appear in. You can even create cross-tab tables that are regularly generated inside of your custom report. You can even add statistic tables for each question which include the mean, variance, standard deviation and total responses for each response choice. Reports can be exported to Microsoft Word, Microsoft Powerpoint, and Microsoft Excel. Qualtrics even lets you drill down in your responses, which is a pretty neat feature.

All of these are great reporting features and for those of you who tend to do your reporting directly from the survey software — who want something quick, easy and fairly powerful right out of the box — Qualtrics fits the bill.

If you’re looking for advanced analysis capabilities, such as those offered by a few other expensive packages, you aren’t going to find them here. But then, there really aren’t that many survey software packages that offer advanced analytics. And if you’re an advanced market research kind of a person, chances are you are going to use an external software package to analyze your data anyways.

Overall, I was very impressed by Qualtrics reporting and data management capabilities. I was a little disappointed by the fact that it doesn’t offer “true” SPSS export, but what it does offer is enough to make getting the data into SPSS manageable. The respondent editing capabilites are fine, and the report development system is excellent.

Qualtrics Security, Data Protection, Company Viability

Qualtrics is a privately held company located in Provo, Utah, just south of Salt Lake City. The company started a littled over ten years ago and has a very wide range of corporate clients. In my opinion, you shouldn’t have any concerns about working with this company.

Survey links are secure — there is no reasonably way for other people to guess the link to your survey based on links generated for other surveys. Furthermore, every invitation sent through the system has its own unique complex URL, which means that respondents can’t guess other respondent links and use it to get access to personal information.

Qualtrics allows each individual user to have their own account, and there are a number of different security options that can be assigned to each user. Surveys can be shared or not shared. Data can be shared or not shared.

Qualtrics maintains its data behind a firewall and all data can only be accessed only by the owners of the survey. I do not not whether survey data is encrypted on their server.

Pricing/Options

When you sign up for a Qualtrics account, you are generally allowed to have an unlimited number of users and can create an unlimited number of responses. A certain amount of service is included, and a certain number of custom templates will be included depending on the package that you purchase. Your account will generally be priced based on the number of responses you want to collect over the course of the year.

Qualtrics isn’t cheap — accounts tend to start at around $10,000 and go up depending on the number of responses you want to collect. Qualtrics will give you a quote based on what you’re looking to do, and my impression is that the company is pretty flexible in terms of being able to come up with a package that meets your needs at a price you can afford.

Qualtrics prides itself in offering expertise in addition to survey software — they have a lot of well educated research experts on their team and the organization has a very academic bent to it. If you’re looking for a survey system that will be able to provide in-depth consulting capabilities to your project, this might be the right survey software for you.

Conclusion

There is a lot to like about the Qualtrics survey software system. The questionnaire features and questionnaire development environment is among the best that I have seen in a web-based application. The reporting capabilities should please anyone who is into that sort of thing, and the system provides for both panel management and multi-user account capabilities.

For my own purposes, I struggle with the fact that Qualtrics doesn’t offers some of the advanced capabilities I like to have in a survey package. I miss the ability to edit the survey page structure or the CSS files. I wish I could write my surveys in Microsoft Word and import them directly into Qualtrics. I wish it offered some version of scripting/advanced programming capabilities. I’d also like it if the software allowed me to export data in true SPSS format. And for my own purposes, I wish that there was a way to install it on my own server instead of having to use it on the Qualtrics server.

But my personal approach tends to be much closer to that of a research firm than a corporate research team or an academic organization. And while I probably wouldn’t recommend this survey software for a major research firm, I would definitely recommend it for an academic organization or a corporate research department.

Overall, Qualtrics offers an extremely rich feature set and performs extremely well compared to other survey software systems in its price range.

9 comments for “Qualtrics Review (Online Survey Software)

  1. Greg Brassfield
    August 6, 2010 at 2:04 pm

    Here is my disclaimer: I work for Qualtrics Labs, Inc. That being said I wanted to make a couple of corrections. In fairness some of these may have been developed after the article was written.

    1. Qualtrics does have an API that is used for extremely advanced user design. Additionally, Qualtrics has integrated the ability to design advanced web services. Among users of the API and web services are academics, government agencies, and Fortune 50 companies.
    2. Qualtrics allows you to do your own CSS to design the look and feel of your surveys.
    3.Qualtrics is encrypted in transit and at rest and adheres to HIPAA compliance regulations.
    4. Qualtrics does can create a .sav file for SPSS export

    Thanks for the article and feel free to contact me with any questions you may have.
    [email redacted to save Greg from getting lots of spam]

    • August 18, 2010 at 12:05 am

      Thanks for the comments, Greg! As you mention, my post was made a couple of years ago and there have been many updates since then. I probably need to do a new post. I’m a big fan of Qualtrics and use it just about every day, including the API and web services. As I think I’ve posted elsewhere, it is one of my favorite online survey tools.

  2. October 20, 2010 at 3:45 pm

    Thank you for such an excellent review article. My research group has been using surveymonkey for a couple of years and are looking to upgrade for a couple of grants we’ve received. This article answered a couple of unanswered questions as we begin our search for a better survey design company and was very informative overall.

    Thanks!

    • Craig Daly
      May 12, 2011 at 4:08 pm

      Great article Mark. I actually work at Qualtrics so I wanted to get a feel for what people are saying. I was blown away when I first started so I thought it would be advantageous to check out the buzz and see if others thought likewise.

      And Sean, email me if you need a better trial account. I can set a more “stacked” version up for you. Craigd@qualtrics.com

  3. Ryan
    May 23, 2011 at 10:15 am

    Just as an update, Qualtrics can now validate against regular expressions. I realize this article is a bit old, but much of the “can’t do”s listed here have now been added.

  4. cyber
    August 12, 2011 at 2:58 am

    A great article. I would love to see what new features are available since this blog was originally posted. It may even warrant a new review…

  5. Anonymous
    November 14, 2011 at 6:35 pm

    @Craig @Greg

    Do Qualtrics Labs staff support installing Qualtrics to a local instance, run by the end-user IT staff?

    For example: a government institution wanting the ability to store confidential data and policy restrictions forbid storing data on non-government equipment.

    Thank you.

  6. Eve
    July 26, 2012 at 4:46 am

    Hi Mark,

    I just thought I would let you know, when I click on the link for Qualtrics question types, it directs me to a link http://www.expressivecounseling.com/?!?! I assume this is an error?

    Thanks for your your review.

    Cheers,

    Eve

  7. dave
    October 25, 2013 at 1:51 pm

    Can anyone tell me the price of qualtrics to use there software (I know they have a trial package).. I asked them for one but they want to demo it, etc and I really just want a ball park cost , and can’t get a straight answer.

Leave a Reply