Machinima for Dummies: Focus grouping fer cheap

Focus grouping fer cheap

Hugh "Nomad" Hancock — Tue, 06 Nov 2007 12:15:49 +0000

We recommend focus-grouping your work a bunch of times in Machinima for Dummies - where by “focus-grouping”, we mean “getting people who aren’t you to watch your film before it’s released”.

It’s one of the most valuable processes you can go through. Ideally, it’s most valuable when you can cross-question your audience, but any eyeballs that aren’t yours or your team’s on your work will help pick up the problems that you’re too close to what you’re doing to be aware of.

The problem, of course, is finding people to focus-group on - particularly people you don’t know! By definition, you can’t just call them. Or at least, you couldn’t.

Amazon.com’s new ”Mechanical Turk” technology looks to be perfect for doing low-cost “mall tests” of your film. Essentially, it allows you to hire as many people as you want for a very small task - like, say, watching a short movie and commenting on it. In terms of costs, it’s very cheap indeed - this Slashdot poster paid $1.45 for people to come up with rebuttals to an argument he was making online, for example.

So it would be trivial to set aside a budget of $30 or so - well within reach if you’ve spent a month or two on a film - and use Mechanical Turk to find out what Real People think of your film before release, allowing you to correct any mistakes, make clearer anything that isn’t, polish the bits that clunk, and generally get you closer to that 4,500,000 hit YouTube smash!

"Focus grouping fer cheap" by Kate

Thu, 08 Nov 2007 12:06:38 +0000

This sounds similar to humanbot forum spamming techniques..humans are employed to bypass the ‘are you a human being and not a spambot’ type tests needed to join / post on certain forums.

"Focus grouping fer cheap" by Anthony Bailey

Wed, 07 Nov 2007 20:28:30 +0000

Hmmm, interesting. This isn’t really the kind of task that MTurk was originally designed for.

The most obvious HIT is to ahow the video and ask for free-form text comments - but that’s open to gaming, so you’d have to create a suitable qualification (this is MTurks technology for screening those who are allowed to take your HITs.) You would also need to do that if you’re going to require any kind of non-disclosure agreement from your testers.

The kind of HITs that work well without that are ones where you make the set of possible answers more constrained and then assume that the correct answers are the majority ones. But it takes time to generate the right set of questions, and in matters of opinion where truly correct answers don’t exist, it’s not a fair way to judge the validity of answer - you’d end up penalizing people with valid and interesting minority opinions.

So I think any one production company might find that making good use of MTurk as a system for micropaying random screen testers requires quite a bit of investment. It’s probably easier to ask friends and forum-dwellers.

Having said that, if StrangeCo or similar are interested in trying this kind of approach out, particular along the lines of giving back what they find to the community (e.g. come up with a screen-testing qualification that other groups could make use of) then let me know. Even though MTurk is a different part of Amazon to where I work, we have experimented a bit with using it, so I at least sit next to people who have used it in anger.