Kupferman.com

SurveyMethods.com Review (Survey Software)

SurveyMethods (at SurveyMethods.com) is a basic online survey software package that does OK on its own but doesn’t really offer the features of many of the other survey software packages it competes with. And while it is not a software package I’d be likely to recommend (there are even free survey software packages that provide more than SurveyMethods does), it is free to try — so you can make your own decision!

SurveyMethods Questionnaire Features

SurveyMethods offers the traditional selection of question types, including single choice closed-ended questions (radio buttons or drop down boxes); multiple response questions (checkboxes); open-ended questions (essay, single line, multiple “form” questions); ranking questions and constant sum questions. It also offers the option of “points questions” which means simply that you can assign values to each response (red = 5, green =2, blue = 1, etc). The system also allows for basic matrix (table) questions.

Implementation of most of these question types in SurveyMethods isn’t detailed. Although in their marketing they say that they offer 20 different question types it seemed to me that many of these were just different variations on the question types listed above. The most advanced question type that is offered is a conjoint question, which sounds pretty impressive until you try to use it — besides being confusing, all it really does it let you rank various combinations of attributes (it doesn’t create an orthogonal design and it doesn’t generate utilities). If you’re serious about doing conjoint you’re much better off looking at the conjoint question option offered by SurveyZ or if you’re really serious, check out Sawtooth Software’s solution). Other “advanced” question types are limited compared to other survey systems as well – the ranking question, for example, makes you enter numbers into each box rather than letting you click or drag each option in order.

SurveyMethods Conjoint Example
Example of SurveyMethod’s conjoint question.

The advanced options available in SurveyMethods for each question are limited. For example, I can choose to make a closed-ended single response question mandatory and I can randomize the order of the responses.  but I can’t add a reporting value for each option, I can’t assign “other” text boxes to each option (I can add one text box to the end as an “other” response), and I can’t lock items in place. If I have a lot of items, I can break them into sections. I can’t put the responses in multiple columns. I can’t select a default value or assign a unique CSS class to the question. You may not need to do any or all of these things, but if you do any detailed work in survey development you may miss them.

SurveyMethods does allow for some basic branch logic, that is very easy to understand but potentially frustrating if you have to do a lot of branching or if you want to create complex scenarios (you can’t, for example, have it branch based on the responses to two different questions). And while your survey can appear on multiple pages (which is a good thing) you can’t name the pages, which can make setting up the branch conditions somewhat annoying (when you branch you don’t tell it what question to branch to — you tell it instead what page you want to branch to, and your options are “Page 1”, “Page 2,” etc. SurveyMethods does not offer dynamic branching (which will hide questions on the page based on the responses to earlier questions on the page).

SurveyMethods does not offer the ability to “skip over” questions based on specific criteria other than through branching (although they say they offer skip logic, what they’re really offering is branching logic — skip logic is assigned to a particular question and says “only show me if certain criteria are met), which branch logic says “look at the answer to this question and depending on what it says take me to this other question.” Subtle differences, but there are times when it helps to have both types of logic available.

SurveyMethods does not allow you to randomize questions, groups of questions or pages of questions, although these aren’t common features offered by low end packages anyways.

Although SurveyMethods allows you to enter text blocks and images, you cannot enter blocks of HTML directly, nor can you include any special javascript headings or other code. There is no scripting language. You can’t include Flash or video.

Incidentally, SurveyMethods comes in three different flavors — Basic (free), Advanced, and Professional. The free version does not allow you to make questions mandatory, randomize answer choices or utilize branch logic.

Survey Designer Features in SurveyMethods

The survey designer is fairly basic — no AJAX here. The list of questions in your survey are displayed roughly as they will appear in real life, and one neat feature is the ability to change the “presentation style” of the survey on the fly through a drop-down box and instantly see the impact on your list of questions. But other than that, the look/feel of the survey designer isn’t particularly friendly (although it isn’t unfriendly either).

There are two methods for rearranging questions in SurveyMethods. One option is to click the little blue up and down arrows next to each question. The other option is to go to the move quesiton screen which lets you select from a drop down box which question you’d like to stick the question you’re moving in front of or behind.

SurveyMethods does not allow you to edit the response set of a closed-ended question once the survey has responses. This is a fairly common safety measure, but can be frustrating/annoying if you start the survey, collect fifty responses and realize you want to add an extra response. Your only option is to either delete all of the response data. Note that you also are not allowed to delete or move questions around once any response data is collected.

SuveyMethods.com Editor
The SurveyMethods Survey Editor.

You are able to choose from a variety of different color schemes and the fonts, and if you like you can go ahead and create your own although with only a limited set of options to choose from (you can’t, for example, create your own CSS code). You don’t have a tremendous amount of control over the look/feel of your surveys — they all have pretty much the same structure/format. You can also upload a logo to appear at the top of your survey.

SurveyMethods offers online help, although I didn’t really like the way it was structured. Basically, you click on the “Help Center” icon and it takes you away from your survey (rather than opening the help in separate window) and presents all of the help options in the form of frequently ask questions which do not appear in any context sensitive manner.  If you’re able to find the subject you’re looking for (which can take a frustratingly long time) the posted answer is extremely short and not particularly helpful. And then when you’re done you have to hit your back button about four times to get back to the question that you’re working on.

Spell check is available, but not in the free version. The ability to add logos is also limited to the Advanced and Professional packages. Professional users can allso”collaborate” with others on survey development. It does not have a special test/debug mode for surveys.

SurveyMethods Response Collection Features

SurveyMethods offers an average set of features for collecting responses.  Users are limited in the number of responses they can collect each month based on their account level. Promotional methods are limited to e-mail invites and URL links, and only Professional level accounts can be configured to show a custom completion page or redirect to another site at the end of the survey — everyone else sees an add for SurveyMethods. The respondent experience (how fun/interesting it is to take a survey) is fairly limited.

SurveyMethods allows each account to have an unlimited number of surveys, although it limits the number of total responses an account can collect each month. A free account can collect 500 responses a month; an Advanced account can collect up to 1000, and a SurveyMethods Professional account can collect 5,000 responses. If you exceed the maximum number of responses that you’re entitled to on an Advanced or Professional account SurveyMethods will automatically add another 2,000 responses to your account and charge your credit card $5.00.

Sample view of a survey in SurveyMethods
A sample SurveyMethods Survey in real life

I wasn’t completely impressed with my experience as a respondent. To ensure higher completion rates on voluntary surveys it is often helpful to make surveys interesting or entertaining — SurveyMethod surveys are somewhat plain to look at and if you had a lot of questions it is likely that the respondent would become bored faster than they would using a more interesting interface. Of course, this is just my subjective perception of the experience. To its credit, SurveyMethods does give you the option of providing several different types of progress indicators that may make the experence more pleasant. Respondents cannot save or print their responses, although if allowed by the designer they can go back to previous responses. It appears that respondents who quit the survey can re-enter their survey later to finish.

SurveyMethod surveys can be promoted in the usual ways, including web URL and e-mail invitations. One neat feature is the way in which a survey is “lauched” — it takes you through an easy to follow set of six steps that let you choose your method of publicizing your survey, setting up start pages, setting up completion options (if they’re available at your account level) and then launching of the survey. It does not allow you to embed surveys into your web site, nor does it allow you to embed the first page of your survey into an e-mail message. The e-mail invitation system is very basic, but it is also easy to use. E-mail addresses can be imported from Excel or typed in manually. If you have the Advanced or Professional version the system can also be configured to automatically send out reminders. The Advanced and Professional version of SurveyMethods can also be configured to stop collecting responses after a certain date or after a specific quota has been met.

SurveyMethods makes it possible to create a custom greeting page for your survey, but frankly even if you couldn’t you could accomplish the same effect by adding a text question to the beginning of the survey. At the end of the survey it appears that SurveyMethods will automatically take you to a built-in thank you message, presumably one that promotes SurveyMethods — except in the Professional version, where you have the additional options of creating a custom thank-you page (no HTML allowed), showing the respondent the results or redirecting respondents to a web page of your choice.

SurveyMethods Response Management / Reporting

SurveyMethods offers very limited reporting and analysis capabilities for free accounts — almost to the point of what I would consider worthless. Free accounts cannot export results, cannot generate reports, and cannot create charts and graphs. Even Advanced accounts are only allowed to generate reports and export data — they still can’t create charts and graphs or cross-tabs. All of these features are reserved for the $39 a month Professional account.

You are allowed (presumably at all levels) to view and delete individual respondents. You can also create basic summary reports of all respondents which do include horizontal bar graphs. It is a nice feature that you can choose to display results only from completed or not-completed surveys, and you can choose to have it display only certain questions or surveys that match certain criteria.

Sample of a SurveyMethods report
Sample SurveyMethods Summarized Data Report

One neat features of SurveyMethods is the ability to create reports featuring tables and charts. This charting options are fairly basic, but could be useful if you don’t want to take the time to create your own Excel charts (they look about as good as the basic Excel charts). You do not appear to have control over the order that the charts and tables appear in, although it appears that you can remove charts that you don’t want to have included. SurveyMethods allows you to export reports to Microsoft Powerpoint or PDF files, and you can manually e-mail reports to address you specifiy. Data can be exported into Excel, but oddly cannot be saved into a CSV file or SPSS format.

Survey response data is retained up to a year for the free accounts and for an unlimited period of time for the Advanced and Professional accounts.

Data Security, Company Stability, Etc.

The company does not indicate where/how the store their survey data and whether or not the data is encrypted. Nor, frankly, do they even provide any of the names of their owners on the web site or even which company they reside in. Even the SurveyMethods.com domain is registered privately so it is impossible to see who precisely owns the domain. The company’s phone number is, according to WhitePages.com, an unpublished landline located in Dallas, Texas (although with virtual phone numbers these days it is impossible to know whether the company is based in Dallas, London, or somewhere in India). Even the Terms of Use and privacy policy of the site offer no indication of who runs the company and where they are located.

SurveyMethods does not offer individual respondent authentication, but it does offer secure survey URLs (which means that there is no way to guess other survey URLs based on the URLS used to identify your survey). However, SurveyMethods doesn’t block any of their links from Google,which means it fairly easy to see a variety of studies that have been conducted on the site just by doing a domain search within Google.

SurveyMethods Summary Evaluation / Pricing

SurveyMethods comes in three subscription flavors: Basic (free), Advanced ($9 a month) and Professional ($39 a month) which dictate the number of responses you are allowed each month as well as which features you have access to. SurveyMethods offers a seven day trial account which gives you access to all of the features for free. In my opinion, the free account is fairly useless for most survey applications because it won’t allow you to make question mandatory, export data or redirect respondents to another web site after the survey is over. The Advanced option isn’t bad, although I’ve seen other survey systems that offer more at about the same price. SurveyMethods offers free Professional accounts to education and non-profit institutions — as long as they’re willing to post a link to SurveyMethods on thier web site and give credit research reports. If security matters to you, I might also be a little be leary of using

All in all, SurveyMethods is an OK system and is perfectly usable for many purposes. If this were a few years ago I probably wouldn’t be as critical of it as I am. However, the fact is that right now it does not offer nearly as many features as many of the other online survey systems that are out there. Off the top of my head I can think of at least three other systems that offer many more features as SurveyMethods at about the same price.

1 comment for “SurveyMethods.com Review (Survey Software)

  1. Olivier
    October 28, 2010 at 2:24 pm

    Thanks for this review. Please if you mention other survey systems better than the one you are reviewing, why don’t you just give their name?? that is frustrating about your article. But overall, thanks a lot.

Leave a Reply