IMRO Frequently Asked Questions

My client has permission to e-mail their customers. They wish to get unbiased reactions, but as the 2nd party research firm, do I need to identify my client when sending e-mails?

Answer:The identification of the sponsor of the research is not specifically required by CAN SPAM, but it is inferred, and should be considered a best practice element. Without identifying the sponsor, the contactee is unable to determine whether the contact is legitimate or not. Therefore, even if one or both of the CAN SPAM pre-cursor elements are in place (e.g. either an EBR or an opt-in condition) the recipient cannot be expected to recognize the condition of these elements without the identification of the sender.

Points for Consideration: Some controversy has been generated as to whether companies can give their client lists to some other entity to contact potential panelists and respondents without revealing the source of the information. IMRO has traditionally recommended against this procedure (others forbid it), because without establishing an EBR or opt-in condition, the invitation will appear to be spam to the recipient. While some argue that this is not technically in conflict with CAN-SPAM, the potential for damaging the image of research is clear: by appearing to be spam and casting doubt on all future invitations.

Back to Top

Does IMRO have a set of profressional standards we can employ as guidelines?

Answer: Yes, in 2006 IMRO published its guidelines for Best Practices in Online Sample and Panel Management. Those guidelines represent the best thinking as to the questions that individual practitioners and panel providers must be able to address to their client's satisfaction, along with recommended policy points, where appropriate. They are available for download here.

Back to Top

Where does IMRO stand on the idea of aesthetics versus functionality?

Answer: The concern you need to keep in mind is that graphical content and enhanced aesthetics clog up bandwidth. You need to keep a balance and not sacrifice ease of responding for the sake of cool looking surveys. At least that is the concern at this point in time. Although broadband usage continues to grow, there is still a sizable third (32%) of the US population still relying on painfully slow phone connections. Too often we test online surveys from the comfort of our office without also trying them on a 56k connection. And that's where intensive graphics can really slow things to a crawl. So I would agree with the questionnaire that keeping everything "clear, concise, uncluttered, unbiased" is the main concern. The goal then is to have the survey as visually appealing as possible without sacrificing speed and ease. Because if those cool looking graphics cause a long delay in displaying a page and/or moving on to the next page, you've got a terminated interview in the making.

The definitive work on this topic is Don Dillman's Mail and Internet Surveys: The Tailored Design Method. Make sure you get the latest edition, as its seen revision. You can use it as a reference to validate your offering and then refer clients to it who are criticizing the methodological effects of your interface. This may not answer their criticisms about flash-bangs from a client appeal perspective, but it should change the tone of the conversation.

Back to Top

Is there any current info available about completion rates for pop-up surveys?

Answer: Insight Express did a survey back in 2004 that provided the following cooperation rates:

    Banner: .5 to 5%
    Popup: 1 to 10%
    Email: 2 to 50%

Of course this depends on how "directed" the ads are depending on type of website for banner/popup and demographics/interests for email. For pop-ups, if you are on MSNBC, your cooperation rate will be low because people are coming there to get "news" -- not to do a survey. If you are doing a survey on soccer on the FIFA website, you will have much higher interest and maybe approach the 10% level.

This is not to be confused with completion rates which are much lower. Once a person agrees to "cooperate" or clicks on an ad or banner, completion rates are sometimes only 20%.

The rapid acceptance of pop-up blockers must also be considered. The analogy I would make to pop-up blockers is the impact Caller ID has had on telephone completes. Our telephone friends don't like to consider incidence figures using the total number of attempted calls as the denominator. They prefer to look at completes based on folks who pick up the call, which is a rather convenient way of keeping incidence figures steady.

I would think your chances of launching a successful pop-up survey have decreased significantly in the past two years. The following article (March 2005) from the fine magazine WIRED states:

While pop-ups and pop-unders may seem to be getting more troublesome to certain users, broad internet traffic data indicates that their numbers are actually declining. According to internet measurement firm Nielsen/NetRatings, advertisers are using fewer pop-up and pop-under ads, largely because so many people have installed software to block them. In January, NetRatings found that advertisers recorded 918 million impressions of pop-up ads, down from 2.2 billion in the same period last year.

At the same time, other forms of online advertising often criticized as intrusive -- screeching animations and floating ads that superimpose themselves over a page -- are on the rise.

Back to Top

Where can I find out about new trends in Online?

Answer: You can start by checking out The Interactive Marketing Research News on the IMRO Site. Another excellent means for staying abreast of industry trends is IMRO's Journal of Online Research. In the most recent issue, there is an excellent Q&A with renowned market research industry expert, Jack Honomichl.

Back to Top

I'm new to online. Where can I get a good basic understanding of when and where to use online research?

Basics of Conducting Online Research
For someone new to online research, conducting a survey over the Internet may appear somewhat intimidating. However, with the right tools as well as some basic computer knowledge, and a target audience, anyone can conduct their own surveys. There are even several online software companies out there that let you try out their software for free during a specific period of time, so you can learn how user-friendly the software really is. Nonetheless, it is often much easier, and in some instances, more cost effective to contract a research vendor rather than conducting the research yourself. When deciding which way to go (i.e., do it yourself vs. selecting the vendor), take the following into consideration*:

Do it yourself Vendor
Time investment: Your time to select a software, learn how to use it, program the questionnaire, manage recruiting, resolve any fielding issues or questions, and manage the data output Your time to select a vendor
Cost: Cost outlay for software, respondent honoraria (if applicable) Costs can include programming, and usually include fielding, all data management, and respondent honoraria
Audience: You are limited to a sample audience for which you have access to, and your research may yield a smaller number of completes since audience may not recognize you as a trustworthy source Using a vendor panel may yield a larger number of completes while being able to closely target a specific type of audience, and more easily control quotas for sub-groups
Data output: Do you have the resources and know-how to manage your data output? Vendors should be able to give you your data in a pre-selected format.
Additionally, some vendors can also tabulate, and analyze data (for an additional cost)
*Note: assumes US research only.

Whether you do-it-yourself or contract a research vendor, both qualitative and quantitative research may be conducted online. There are several approaches used to conduct online qualitative research, including posting questions on message or bulletin boards and conducting online "chat" groups. For quantitative research (where a large number of respondents is often necessary), or for longer or more complex surveys, a traditional online survey approach is more suited.

Back to Top

What are the primary advantages and disadvantages of using online?

Answer: Main advantages to conducting online research include:

  • Fast turnaround
  • Can be less expensive than telephone or in-person interviewing (but not necessarily)
  • Respondent-friendly
    • Convenient: respondent can complete the survey at a place/time convenient for them (24 x 7)
    • Complex questionnaires are easier to handle
    • No interviewer bias: chance for interviewer error is eliminated (for quantitative studies only)
    • Preserves respondent anonymity – particularly good for highly sensitive topics
  • Can include pictures, videos/sound (and links to these materials so they can be referred to again later in survey)
  • Easier recruiting method: most panels include respondents who have opted-in and are willing to receive e-mail invites to participate in online surveys
  • Geographical coverage

Disadvantages of conducting online research are:

  • Cannot probe and clarify open- ended responses (for quantitative studies only); Responses could be very brief and are limited to the respondent's ability to type
    • What can you do?
      • Add more instruction about exactly what kind of information you are looking for, add a note to "please be specific"
      • Add specific probing questions
  • Cannot collect true unaided responses unless using an open-ended format
    • This is more problematic for quantitative studies where response coding is necessary to tabulate responses
  • Interviewer fatigue with long surveys (no more than 45 minutes is recommended for online quantitative research, and no more than 90 minutes for online qualitative research)
  • No real way to ensure that the person who takes the survey is the person invited to take the survey
  • During quantitative online research, the respondent cannot ask questions during the survey without waiting for an email response (typically, there's a 24-hour wait to receive a response)
  • For qualitative research, the moderator or client can not see body language, facial expressions, or hear tone of voice
    • However, it has been argued that interpretation of these non-verbal cues may be biased based on those observing the interviews
There are many resources available online that provide information for either purchasing software for conducting online research yourself, or for selecting a vendor. Using any search engine yields thousands of results, therefore here are some popular sources:
  • The Marketing Research Association's Blue Book (www.bluebook.org): A Research Services Directory

  • GreenBook (www.greenbook.org): A directory of market research companies, focus group facilities, moderators, and online research companies.
When considering whether to conduct online research or a more traditional method (such as in-person or telephone), keep in mind that the most important part of the research process is the survey design itself. The old Garbage In, Garbage Out (GIGO) philosophy is very important when considering that the research output may be deemed unusable or unreliable if careful survey design is not undertaken.

Back to Top

Can IMRO direct me to parallel studies - online quant vs. phone?

Answer:
Here is what we found online. The most informative is a piece put together by Harris Interactive, which is ultimately a sales piece, but not blatantly so. And I think it answers the question most directly.

Reviews common problems that can occur in online research if not aware, with a brief section comparing online to telephone research.

Discusses difficulties of telephone interviewing at the advent of the "Do not call" registry; brief comparison to online interviewing.

Back to Top

What is mobile marketing and what role will mobile research play in this industry?

Answer: Mobile marketing is direct marketing to mobile phones and PDAs. Mobile marketing is comprised of the following delivery systems:

    Text Messaging (also known as SMS for short message service) is the ability to send text-based messages person to person, from person to application (such as a voting application) or application to person (such as an outbound marketing alert or information.) Text Messaging is now built into over 95% of mobile phones.
    MMS (multimedia messaging service) is adding multimedia to text messaging. For example, taking pictures or videos with your phone and sending them to a friend's phone or email account. MMS is going mainstream and Citrus has done consumer television advertising to promote this activity for our wireless client Edge Wireless (a subsidiary of AT&T).
    WAP (wireless application protocol) is simply the wireless Web. When you access an Internet session on your mobile device, you access via a WAP session.
When you consider the ubiquity of mobile phones, their presence in the daily lives of users, advances in the wireless networks and improved handset technology, it is easy to see why marketers are becoming hooked on the idea of mobile marketing. Mobile marketing is unique in its ability to be always-present and always-on making mobile marketing a strong candidate for integrated programs where active participation and the power of "brand in the hand" is desired.

The largest and most immediate opportunity in mobile marketing is text messaging (95% of all mobile phones are text message ready while approximately 30% of phones are mobile Internet ready). Text messaging allows you to connect to your prospects and customers with alerts, information, contests, surveys, voting and couponing (more on this later.)

A significant benefit of text message marketing is that it's easy to measure from a research perspective and you can immediately build a CRM database of mobile subscribers for repeat marketing. In fact, building that database today before your competitors will be critical to your ongoing success. You may have been working on building an email database for years. Now you can build a database that allows you to communicate directly with consumers on a 24/7 basis.

Most mobile research today is conducted via text messaging. Although web based surveys can be delivered to handsets with wireless web, the percentage of respondents with mobile web capabilities is low (30%).

Mobile surveys can be conducted using a pre-existing database of consumers (i.e. client supplied list of cell phone numbers) or via a mobile panel (many of the large online sample providers now offer mobile panelists). There are also firms that provide programming and hosting services for text based mobile surveys- a simple web search will provide you with a list of these companies.

Pointers for mobile surveys:

  • Keep it short and simple. Recommend no more than five questions.
  • Limit question type to closed end. Open ended questions are difficult for respondents to manage via text.
  • Keep timing top of mind (no one likes getting beeped with a text message at 2am…)
What's next for mobile research…
  • Incentive cash will be delivered direct to respondents mobile phones
  • Mobile barcoding will make it possible to more accurately measure couponing and other "call to action" advertising
  • GPS tracking will allow marketers to more accurately target and measure consumer reaction to mobile campaigns (send a consumer a survey about The Gap while they're exiting The Gap)

Back to Top

What are the advantages and disadvantages of online bulletin boards (BBFGs) and online focus groups (OLFGs) versus traditional qualitative research methods?

Answer: Online market research offers many advantages over traditional qualitative research methods. These include the ability to gather respondents who are Geographically dispersed or difficult to reach, and the ability to discuss sensitive topics in a private and anonymous environment. OLFGs happen in a real time chat discussion running around 90 min.and are ideal for top of mind directional feedback BBFGs happen in an asynchronous environment over a period of days which develops into a rich discussion thread which can be used to gather much more depth from the discussion.

Advantages

  • Easy Access to Geographically-Dispersed Respondents – Geography and time zones are no longer an obstacle. Because there are no set times to Participate as with other qualitative methods, it's possible to include multiple time zones in a global study.
  • Higher Participation Rates - For busy individuals such as physicians, lawyers, working moms or corporate ex executives, participation rates can be higher because respondents do not have to travel to a facility and can login wherever and whenever it is convenient for them.
  • Anonymity - Participants are more likely to respond openly and honestly about sensitive issues because of the anonymity and privacy afforded in an online environment.
  • Access to Low Incidence Respondents – Low incidence is less of a barrier in online market research as participants can be targeted across a wider geographic area with no need to shoulder the expense of transporting them to one central facility.
  • Time and Cost Effective – There are no travel or hospitality costs when conducting online market research allowing you to invest financial resources into areas with a greater return on investment. Because of the virtual nature of online qualitative solution, national or global projects can be conducted concurrently from one location - saving both time and money.
  • Rich, In-Depth Data - The content of one bulletin board often contains as much or more information than three or four traditional focus groups. The BBFG allows for a greater evolution of concepts or reactions to revisions of stimuli over time. Because BBFGs are not a time-constrained environment, responses tend to be richer and more developed than traditional methods.
  • Larger Groups – standard BBFG groups range from 10 – 30 respondents. Utilizing an 'Interview" mode, it's possible to include hundreds of respondents who can be interviewed by just one moderator.
  • Utilization of Different Methodologies - The bulletin board offers a convenient and flexible data collection method that can be used for a variety of applications that include individual interviews, group interactions, product placements, homework assignments and/or uninfluenced discussions across multiple days ranging from three days to several weeks.
  • "Egalitarian - There tends to be fewer incidences of individuals dominating a discussion with online methods since everyone has an equal chance "to be heard".
  • Speed – Transcripts are available in real-time and available to numerous stakeholders concurrently through any internet-accessed computer.
  • Less Bias - There is less bias due to factors such as appearance, age, sex, or other visual characteristics that might affect a session's outcome.

Disadvantages

  • Consistent Monitoring – The BBFG must be monitored by the moderator several times daily to maximize probing opportunities and ensure potential areas of exploration are not missed.
  • Lack of Facial or Vocal Clues - Online qualitative does not yet allow for facial expressions or vocal clues. Therefore, we do not recommend it for projects in which participants have to touch, feel or taste objects or products, or where visual or vocal clues are paramount to the research.
  • Technology Requirements – Although software is built to function at 28K modem speed, the capabilities of the software are best leveraged in a high speed environment. Participants must have dependable access to the internet.
  • Significant Content - The bulletin board generates a tremendous amount of content which requires a greater investment of time during the analysis stage of the research process as compared to traditional focus groups. Text analyzers can help to ease this burden.

Back to Top

Is online qualitative market research a replacement for traditional market research or just an alternative?

Answer: The key to successful research is to select the best possible methodology given the objectives, parameters and constraints of the project. There are research projects which would be difficult if traditional methodology was used but which are well-suited for an online environment. If projects involve sensitive topics, geographically dispersed respondents, low incidence rates or technology-related issues, then there is a strong case for utilizing online market research. As demonstrated above, there are advantages and disadvantages to online market research. A good moderator must assess the advantages and disadvantages of any given methodology with respect to the project under consideration.

Back to Top

What are unique characteristics of online qualitative research as compared to traditional qualitative research methods?

Answer: One of the key differentiators is the ability to simultaneously conduct long-term studies with larger groups of geographically-dispersed participants without the need for travel.

The immediacy of online research is another differentiator. The transcripts are available immediately and in real-time. There are no transcription errors or additional transcription costs.

Less bias, greater individual participation and less dominance of group leaders are some of the other unique features of online qualitative market research.

Back to Top

Do online bulletin boards provide an insight that you can’t receive from traditional qualitative research?

Answer: Yes. A skilled moderator will be able to leverage the anonymity characteristic of online research to extract more in-depth and honest responses from individuals. Additional insights are gained through the ability to measure changes in attitudes over the length of the group and the ability to revisit comments and get additional clarification over a period of days or weeks.

Back to Top

What qualities should you look for in an online moderator?

Answer: First and foremost, you need to have an experienced qualitative researcher who is skilled in traditional research methods. Familiarity with the Internet community and a considerable degree of confidence with technology are very important qualities for a successful online moderator. An understanding of technology culture provides insight into the way people communicate on the internet and which moderator skills transfer well to this new medium. This familiarity can be gained through participation in chat rooms, bulletin boards, email and list-serves. A moderator who is conducting synchronous (Online Focus Groups - OLFGs) must be a skilled multi-tasker with proficient keyboarding skills. They must be able to keep track of client’s comments and questions, manage multiple participant discussion threads and probe and clarify responses from participants – all while maintaining the focus of the group. A reliable, high-speed internet connection and a computer with sufficient RAM are essentials to effectively conducting and exploiting online market research. An awareness of online recruiting techniques and companies that provide quantitative and qualitative research participants would also be a useful knowledge set for online moderators. There are a number of moderators with extensive online experience. Inexperienced moderators may have supplemented their online experience through educational courses offered through RIVA, PMRS, AQR, Burke, various professional workshops or seminars. If you are seeking to equip yourself or your market research personnel with online skills, we have found that having inexperienced online moderators co-moderate with more experience online professionals shortens the learning cycle and allows for an effective transfer of skills. Strong written communication skills are also important characteristics of a good online moderator. Moderators must be able to question and probe effectively and succinctly, “read between the lines” and ensure that the discussion remains focused and on track. Moderators should have developed their own online personality that promotes interaction and contribution while effectively managing the group dynamics.

Back to Top

What types of qualitative research can and can’t be performed in an online bulletin board methodology?

Answer: Studies involving non-tactile stimuli can easily be administered online. Concept testing, website usability, idea generation, opinion tracking – particularly longterm studies, sensitive topics and product development are types of research that can successfully performed with a bulletin board. Many providers can incorporate graphic files, streaming video and streaming audio, this tool is also useful for testing commercials, ads, movie trailers, package design or other creative. Qualitative research in which auditory or visual clues are important inputs into the research are not well-suited for an online environment. Research projects which target socio-economic disadvantaged groups may not be appropriate for online methodology as internet penetration of this group is not yet reflective of the population at large.

Back to Top

What is the decision process that should be followed to determine whether to use online bulletin boards online focus groups or traditional qualitative methods?

Answer: There are three key areas to consider when selecting online bulletin boards; Target Audience, Subject Matter and Timelines Target Audience:

Geography / Incidence – Would it difficult or expensive to gather respondents together for traditional sessions?

Subject Matter:
Sensitive Issues – Is there a need or preference for anonymity to ensure open, honest responses?

In-Depth Discussion – Are clients looking for “top-of-mind” reactions to concepts and stimuli (OLFGs) or are they looking for more “in-depth” analysis of relatively complex issues (BBFGs)?

Time Lines:
Quick Turnaround – Do clients need results within a short period of time (BBFGsOLFGs) or have they allowed time for traditional research (travel, transcript production, etc.)

Back to Top

What research, studies or articles discussing online qualitative research are available?

Answer: Quirk’s Market Research Review has a number of good articles that will provide an excellent overview of online research methodology. These articles were published prior to 2002.

Back to Top

Tips for Selecting a Real-Time Virtual Focus Group Facility

Online Focus Group FAQs

Successful Online Qualitative Market Research

Online Qualitative Research Task Force: Report of Findings

Focus Groups without Walls - or Borders

Anatomy of an Online Focus Group

Online Focus Groups: Mainstream in the Millennium?

Online Brainstorming

The articles listed below are from various sources and are more recent. Focus Groups Go Online to Measure the Appeal of Web Sites

Focus Online

The Ins and Outs of Shifting Your Focus (Groups) to the Net

Back to Top

Do you need to warn/alert respondents about the Patriot Act in your privacy policy for web surveys? How secure is my data?

Answer: 1) It is true that electronic transmissions may be monitored by the government under the Patriot Act in the US. Other countries, specifically Germany, other European countries, and most Asian countries have similar laws.

2) The nature of market research surveys when using a web survey would make it extremely difficult for any "listening" post or hacker to make any sense of the transmissions. Data are sent to the central server in small packets of information. The routing of the internet means that data do not always follow the same path to arrive at the server. Also, interviews tend to be over in a matter of days or weeks so an interviewing is “live” and then terminated in a very short period of time.

3) If your survey is collecting sensitive personally identifiable information such as medical records, health care numbers, government id numbers, credit card numbers, account numbers, passwords, etc, then a secure server should be used (https:) with 128 bit encoding. Then a "listening" post or someone trying to "hack" the system would not be able to decipher the transmissions. A secure server is much more expensive to operate and degrades response times. For most marketing research applications; including health care questionnaires, a secure server is not necessary.

4) A reliable web interviewing software provider should be able to report the results of security testing (government agencies, security testing firms, corporate it departments) to make sure the respondent data are secure behind firewalls.

5) The current "Best practices for web interviewing" , require a "privacy policy" (or a link to a privacy policy) on all web interviews which approximately says "We are collecting this information for marketing research purposes only. Your data will be pooled with other respondent's data and it will not be personally identifiable. We will protect/secure your data and insure your privacy."

There has never been a suggestion brought to IMRO by others or its members that a privacy statement should include a government's right to subpoena data for national security or being suspect of criminal activity. I think this goes beyond what is reasonable and necessary.

Sincerely,

Chris King

Back to Top
Interactive Marketing Research Organization
110 National Drive, 2nd Floor, Glastonbury, CT 06033
Phone: 860-682-1000 ~ Fax: 860-682-1010
imro@mra-net.org