Home AI AI Takes a Bite Out of Mechanical Turk: Workers Use AI to...

AI Takes a Bite Out of Mechanical Turk: Workers Use AI to Automate Their Tas

AI Takes a Bite Out of Mechanical Turk: Workers Use AI to Automate Their Tas

It is tempting to rely on crowdsourcing to validate LLM outputs or to create human gold-standard data for comparison. But what if crowd workers themselves are using LLMs, e.g., in order to increase their productivity, and thus their income, on crowdsourcing platforms?

To get a general sense of the problem, they assigned an “abstract summarization” task to be completed by turkers. By various analyses described in the paper (still not published or peer-reviewed) they “estimate that 33%-46% of crowd workers used LLMs when completing the task.”

To some, this will come as no surprise. Some level of automation has likely existed in turking ever since the platform started. Speed and reliability are incentivized, and if you could write a script that handled certain requests with 90% accuracy, you stood to make a fair amount of money. With so little oversight of individual contributors’ processes, it was inevitable that some of these tasks would not actually be performed by humans, as advertised.

Integrity has never been Amazon’s strong suit so there was no sense relying on them.

The researchers who conducted this study, Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West, caution that this task is particularly suited to surreptitious automation.

This means that it is easy for LLMs to generate summaries that appear to be human-written, even though they are not. The state of the art in LLMs is steadily advancing, and multimodal models that can process text, images, and video are becoming increasingly popular. As a result, it is becoming more difficult to distinguish between human-written and AI-generated data.

The threat of AI “eating itself” has been theorized for many years, and it became a reality almost instantly upon the widespread deployment of large language models (LLMs).

For example, Bing’s pet ChatGPT quoted its own misinformation as support for new misinformation about a COVID conspiracy. This suggests that we should be wary of any information that is generated by LLMs, as it is possible that the model has simply recycled its own false data.

Previous articleNorth Korean Hackers Strike Again, Steal $100 Million in Crypto
Abhinav is an experienced software engineer who has transitioned into the world of blogging. With over 13 years of expertise in the software industry, he brings a deep understanding of technical concepts and trends to his writing. TechyNewsNow provides valuable insights and practical advice for fellow professionals, combining his technical knowledge with a passion for effective communication.


Please enter your comment!
Please enter your name here