Crowdsourced testing

Crowdsource testing companies provide the platform for the testing cycles, generally free of charge to the companies whose products are being tested. They then crowdsource the product to a community of testers, who register for testing the software voluntarily. These testers are generally paid either by time, per action, or per bug, depending upon the type of bug and its market price. The crowdsourced testing team can be in addition to the in-house quality assurance team, or a replacement.

Depending on the platform and provider, the method of access varies. The most common are:

• Self-service, where the customer directly interact with testers
• Managed, where the customer interacts with a dedicated person, rather than directly with testers
• API, where the customer requests tests to run via an API, and does not directly interact with testers
• Mixed, where it is a combination of the above.

There are several advantages to crowdsourced testing:

• It's possible, and generally easier than hiring, to get 24x7 coverage and be able to test at anytime.
• The internal testing team may not have the needed setup or hardware to test the software in different environments and in different situations. e.g. different Internet bandwidths, devices, etc.
• If paid per bug, it is can be cost effective, as the product company pays only for the valid bugs reported.
• Testers performing the work are unbiased from knowing much about the software under test. This can help them notice things that may otherwise be missed, but does require more detailed test cases.
• Crowdsourcing can find testers with diverse languages as well as locales. This helps in testing applications which require localization testing.
• Most platforms let you scale from a team of zero to the number you need on demand. This is a benefit to the customer as the more testers testing software simultaneously, the quicker testing can be done.
• Testers performing tests of different applications will identify common anti-patterns.[1]

There are several disadvantages to crowdsource testing:[2]

• Confidentiality must be managed closely as the number of non-internal individuals looking at the system under test increases. This can increase the risk of innovation and new features being leaked to competitors. Some companies require testers to sign a Non-disclosure_agreement.
• Immediate and prompt communication with a group of crowdsource testers can be difficult, or impossible depending on the method of management.
• Crowdsourced testers who are compensated by the number of bugs detected may find a larger number of less impactful bugs, creating noise for the customer, whilst skipping over more harder to replicate bugs. This is due to the principal-agent problem.
• If directly managing a crowd, crowdsourced testing will result in increased need for management oversight due to differences in testers' time zones and locations, languages, and cultures. This can be mitigated by managed, or API based services, as the customer does not directly manage testers.
• Ensuring test coverage in crowdsource testing can be difficult, as some platforms do not support good reporting. This means testing is not planned or tracked the same way as traditional waterfall or Agile test efforts.

Crowdsourced testing may be considered to be a sub-type of software testing outsourcing.

While for some projects it may be possible to get away with only using one approach or the other, a more thorough approach would use a more diverse software testing method, [3] which uses both a dedicated testing team in addition to the crowd. Crowdsource testing is best for things like beta and compatibility testing, which are necessary final steps for testing; however, most software is far too complex for late-stage testing like this to cover all of the possible issues. A dedicated outsourced or in-house testing team will give a better idea of the software's possible defects, but will not give anywhere near the scope of crowdtesting. Therefore, a good solution is to integrate multiple test teams into any development project (and also to develop with the principles of testability[4] in mind from the very beginning.)

Crowdsourcing alone may not give the best feedback on applications. A diverse testing approach that pools both crowdsource testing and a dedicated testing team may be favorable. "Having this diversity of staffing allows you to scale your resources up and down in a fluid manner, meeting tight deadlines during peak periods of development and testing, while controlling costs during slow periods."[5]