In our (limited) experience blogging, we’ve come across relatively few comprehensive aggregations of information surrounding what we’d call “usability research” (i.e, info that relates to the business of testing, analyzing, validating and evolving interface/application design). Thus bereft, and bold folks that we are, we’re going to try and build just such a collection . So we thought it might be appropriate to begin with some thoughts on the topic of multivariate testing. Below you’ll find a quick discussion of the multivariate approach in a usability context, as well as some links to additional resources.

Introduction/Definition
Multivariate testing in the context of usability research might, in layman’s terms, be described as automating aspects of the design/testing process. Let’s say you have a homepage where it is vital that you get people to either click on a tout or sign up for something. Let’s also suppose you have a robust flow of visitors to the site.
Nick calls this a “pivot point.” The design of the pivot point itself might appear simple (some text, a button, an email field). But, on consideration, that design could vary nearly infinitely – especially when you are trying to figure out where to put it on a page that has other content. Is the radio button red? How much copy? How big is the tout? Is it in a pop-up? What you’re trying to do is determine the optimal combination of these elements for a wide swath of users.

One way of testing what works best is to build prototypes of the page/pivot that you then expose people to in a testing-lab environment. From this, you get qualitative feedback as well as behavioral indications regarding the effectiveness of a given arrangement of elements. But fundamentally, you are limiting your options here – because it’s not effective to show a given user more than about two versions of a given interface or element. This binary choice is essentially what’s known as “A/B testing.” Another approach is to simply keep tweaking your website and analyzing conversion trends based on the changes you make. This strategy also limits you, however, to what can be imagined, built, fielded and analyzed by a development team.

Such limits make it nearly impossible to cycle through ALL a pivot point’s possible variations – and therefore to know for certain you’ve achieved the optimum balance of elements. This is where multivariate usability testing comes in. Simply put, practicioners use software and robust traffic flow to continuously tweak and record the impact of thousands or millions of permutations on an interface. So, a company need not choose between shortening the marketing copy, shrinking the tout, moving it over, changing its color, lengthening the copy and lowering the price. It can test all of the above, in real time, and get reliable data that’s been pre-parsed.

Other Discussions of Multivariate Usability Testing
Paid Content.
Recap of Forrester analyst’s speech
Marketing Vox

A Few Companies Specializing in Such Testing
Vertster
Optimost
Offermatica
SiteSpect

When to Consider Multivariate Usability Testing
- If your interface or site relies heavily on defined paths that try to convert a user at a specific place (or pivot point)
- If marginal improvements to this pivot point will bring you a substantial increase in revenues
- (Corollary to above) if your site is fairly heavily trafficked

Bear in mind that you won’t get individual qualitative data from this process – nor direct insights into online behavior. Nor will you be able to design an entire site using this method. But you will be able to be more certain that a given pivot point has been evolved to its ideal form. As professionals (no cracks, please) who primarily do design and qualitative testing, we’ve been impressed with this technology for a while. So we’ll be looking at it from time to time.

JF

Filed under E-commerce, Usability Research. You can leave a response, or trackback from your own site.