Toloka

A toloka used to be a form of mutual assistance among villagers of Russia, Ukraine, Belarus, Estonia, Latvia, and Lithuania. It was organized in villages to perform urgent work requiring a large number of workers, such as harvesting, logging, building houses, etc. Sometimes a toloka was used for community works (building churches, schools, roads, etc.). The idea of joint useful work for a common result is consistent with the principle of crowdsourcing.[6]

Data labeling helps to improve search quality and effectively tune result ranking algorithms. For example, in August 2017, Yandex introduced a new search algorithm called "Korolyov". It's a neural network in creation of which, to train neural networks, the estimates of «tolokers» were used. In more than two years of service, the tolokers have put about two billion ratings, allowed developers to significantly improve the algorithm.[7]

To "teach" a machine to solve a particular problem, it is necessary to give it a huge number of examples, both positive and negative.[8] Among other tasks, users of the Toloka service can receive ones to describe objects defined by a computer in an image. For example, the photo shows objects that artificial intelligence was able to recognize, but in order to use this data in work, precise information is needed about what kind of objects they are. Artificial intelligence can be trained on the mass of such recognitions provided by humans, who, in turn, will receive encouragement for their work.[2] In tasks of another type, a context of the dialogue is given and a scale is proposed by which it is necessary to assess whether a chatbot's answer in this context is appropriate, interesting, and so on.[9] Another group of tasks in Toloka is checking automatically received drafts of dictionary entries for Yandex.Translate.[10]

Checking the quality of the online store, delivery service, writing reviews about products and services. For example, in 2017, Toloka users took part in checking the quality of the product return service on Yandex.Market. Such experiments allow to control the quality of the service and identify weaknesses, over which work will be carried out in the future to improve and eliminate the identified problems.[11] In another project, users were asked to write a review for a product from popular categories and provide it with their own photos.[12][13]

Service users also answer questions related to the choice of the most successful name and design for a new product or service,[14] personal or consumer preferences of users. In one of these polls, tolokers chose a name for Yandex's voice assistant.[15] The purpose of the assignment was to determine the character traits of a girl with one name or another. For the survey, they took both the necessary characteristics and completely inappropriate ones, and began to ask: if a girl has such a name, is she rather kind or aggressive? The name "Alice" won the poll by a wide margin.[16]

Completion of tasks in Toloka are paid for - the corresponding amount is indicated next to each task. The amount they can earn is indicated next to the task.[8] It is possible to withdraw funds using several payment systems: YooMoney, PayPal, Skrill, Papara. Also, money can be withdrawn to the mobile phone account of Russian operators MTS, MegaFon, Beeline, Tele2. At the stage of debugging processes on the part of Toloka, withdrawal of money could take up to 30 days, at present - from several minutes to several days.[6]