Comments for cond = false http://blog.cond.org Thu, 11 Nov 2010 23:18:08 +0000 hourly 1 http://wordpress.org/?v=3.1.2 Comment on Why I Hate Mechanical Turk Research by iamreddave http://blog.cond.org/?p=28#comment-6 iamreddave Thu, 11 Nov 2010 23:18:08 +0000 http://blog.cond.org/?p=28#comment-6 Really interesting post. As a Turk simpleton two things strike me 1. It needs a decent reputation system 2. But more importantly I feel the digital sweat shop nature of the mturk (partly due to the reputation problems) is due to a misunderstanding of what computers and humans should be for. The Human Use of Human Beings by Norbert Wiener and also Computer Power and Human Reason by Joseph Weizenbaum dealt a long time ago with how we should divide tasks between humans and machines. If Norbert Wiener was willing to worry about these problems that seems to say that it is important the mturk be used effectively for experiments on what humans and computers should do. Really interesting post. As a Turk simpleton two things strike me
1. It needs a decent reputation system
2. But more importantly I feel the digital sweat shop nature of the mturk (partly due to the reputation problems) is due to a misunderstanding of what computers and humans should be for. The Human Use of Human Beings by Norbert Wiener and also Computer Power and Human Reason by Joseph Weizenbaum dealt a long time ago with how we should divide tasks between humans and machines.

If Norbert Wiener was willing to worry about these problems that seems to say that it is important the mturk be used effectively for experiments on what humans and computers should do.

]]> Comment on Why I Hate Mechanical Turk Research by Edward Vielmetti http://blog.cond.org/?p=28#comment-5 Edward Vielmetti Thu, 11 Nov 2010 22:08:26 +0000 http://blog.cond.org/?p=28#comment-5 The clever counterpoint would be to design a system for evaluating papers about Mechanical Turk, and to parcel out the work so that the Turkers themselves are the reviewers. (Peer review and all that.) The clever counterpoint would be to design a system for evaluating papers about Mechanical Turk, and to parcel out the work so that the Turkers themselves are the reviewers. (Peer review and all that.)

]]> Comment on Why I Hate Mechanical Turk Research by Stephen Purpura http://blog.cond.org/?p=28#comment-4 Stephen Purpura Thu, 11 Nov 2010 22:01:36 +0000 http://blog.cond.org/?p=28#comment-4 I work on the "building signal from noise" aspect of using Turk, so I'm glad that you value that. While I haven't reviewed a Turk paper yet, I have to admit that I have a low bar for Turk papers. As long as they show me something about the nature of tasks that *need* to be done by 'trained people' and the tasks can be done by 'Turkers', I'm usually happy. (Special note: I consider the training of an annotator to be inducement of bias, which is a more complicated discussion.) For many conferences, this isn't enough of a contribution, but if someone where churning out blog posts about the difference between Turkers and 'trained' annotators (that are commonly used in social science research), I would think it was an interesting contribution. Having worked on many, many annotation projects, I am deeply skeptical of the annotation process, especially as it is applied both in academia and in business. If I can't use more sophisticated science than is typically applied to the annotations, I have grave concerns about the work. So, a movement to large N annotators (possible with Turk), can shed a lot of light on how 'regular' people respond under a variety of conditions instead of measuring the responses of WEIRD subjects (where WEIRD is an acronym from recent research that, if you don't know it, you need to read). ;) I work on the “building signal from noise” aspect of using Turk, so I’m glad that you value that. While I haven’t reviewed a Turk paper yet, I have to admit that I have a low bar for Turk papers. As long as they show me something about the nature of tasks that *need* to be done by ‘trained people’ and the tasks can be done by ‘Turkers’, I’m usually happy. (Special note: I consider the training of an annotator to be inducement of bias, which is a more complicated discussion.)

For many conferences, this isn’t enough of a contribution, but if someone where churning out blog posts about the difference between Turkers and ‘trained’ annotators (that are commonly used in social science research), I would think it was an interesting contribution.

Having worked on many, many annotation projects, I am deeply skeptical of the annotation process, especially as it is applied both in academia and in business. If I can’t use more sophisticated science than is typically applied to the annotations, I have grave concerns about the work. So, a movement to large N annotators (possible with Turk), can shed a lot of light on how ‘regular’ people respond under a variety of conditions instead of measuring the responses of WEIRD subjects (where WEIRD is an acronym from recent research that, if you don’t know it, you need to read). ;)

]]> Comment on Why I Hate Mechanical Turk Research by admin http://blog.cond.org/?p=28#comment-3 admin Thu, 11 Nov 2010 21:57:34 +0000 http://blog.cond.org/?p=28#comment-3 No question... s/turk/buzzword/ in the above post (that's "replace" for those of you who don't speak regexp). I should do that every year when I do CHI reviews. No question… s/turk/buzzword/ in the above post (that’s “replace” for those of you who don’t speak regexp). I should do that every year when I do CHI reviews.

]]> Comment on Why I Hate Mechanical Turk Research by Michael Bernstein http://blog.cond.org/?p=28#comment-2 Michael Bernstein Thu, 11 Nov 2010 21:03:07 +0000 http://blog.cond.org/?p=28#comment-2 The interesting challenge for me is to build extremely non-trivial crowd-powered systems. What can we do with crowd platforms that you wouldn't think would be otherwise possible? The design space used to look like a tradeoff between user and AI. The user could step in when an AI couldn't solve a problem, and guide it in the right direction. Now, we have a triangular (simplex) design space: user, AI and crowd. What do we rely on the user to do? What can an AI do? What can a crowd do? How can they play off each others' strengths to reach new heights? We've chatted about this, but a similar critique could be levied against most paradigm shifts in science and engineering. Researchers rush in, and it produces bad work as well as good work. This happens in mechanized systems as well. "Why I hate incremental LDA research", "Yet Another Slight Tweak on SVMs", "Why I Review Too many Twitter Papers that Don't Have A Good Grip On Twitter", etc. The interesting challenge for me is to build extremely non-trivial crowd-powered systems. What can we do with crowd platforms that you wouldn’t think would be otherwise possible?

The design space used to look like a tradeoff between user and AI. The user could step in when an AI couldn’t solve a problem, and guide it in the right direction. Now, we have a triangular (simplex) design space: user, AI and crowd. What do we rely on the user to do? What can an AI do? What can a crowd do? How can they play off each others’ strengths to reach new heights?

We’ve chatted about this, but a similar critique could be levied against most paradigm shifts in science and engineering. Researchers rush in, and it produces bad work as well as good work. This happens in mechanized systems as well. “Why I hate incremental LDA research”, “Yet Another Slight Tweak on SVMs”, “Why I Review Too many Twitter Papers that Don’t Have A Good Grip On Twitter”, etc.

]]>