Rating scale

All rating scales can be classified into one of these types:

1. Numeric Rating Scale (NRS)
2. Verbal Rating Scale (VRS)
3. Visual Analogue Scale (VAS)
4. Likert
5. Graphic rating scale
6. Descriptive graphic rating scale

Some data are measured at the ordinal level. Numbers indicate the relative position of items, but not the magnitude of difference. Attitude and opinion scales are usually ordinal; one example is a Likert response scale:

Statement

e.g. "I could not live without my computer".

Response options

1. Strongly disagree
2. Disagree
3. Neutral
4. Agree
5. Strongly agree

Some data are measured at the interval level. Numbers indicate the magnitude of difference between items, but there is no absolute zero point. A good example is a Fahrenheit/Celsius temperature scale where the differences between numbers matter, but placement of zero does not.

Some data are measured at the ratio level. Numbers indicate magnitude of difference and there is a fixed zero point. Ratios can be calculated. Examples include age, income, price, costs, sales revenue, sales volume and market share.

More than one rating scale question is required to measure an attitude or perception due to the requirement for statistical comparisons between the categories in the polytomous Rasch model for ordered categories.[1] In terms of Classical test theory, more than one question is required to obtain an index of internal reliability such as Cronbach's alpha,[2] which is a basic criterion for assessing the effectiveness of a rating scale and, more generally, a psychometric instrument.

Validity refers to how well a tool measures what it intends to measure. With each user rating a product only once, for example in a category from 1 to 10, there is no means for evaluating internal reliability using an index such as Cronbach's alpha. It is therefore impossible to evaluate the validity of the ratings as measures of viewer perceptions. Establishing validity would require establishing both reliability and accuracy (i.e. that the ratings represent what they are supposed to represent). The degree of validity of an instrument is determined through the application of logic/or statistical procedures. "A measurement procedure is valid to the degree that if measures what it proposes to measure."

Another fundamental issue is that online ratings usually involve convenience sampling much like television polls, i.e. they represent only the opinions of those inclined to submit ratings.

Validity is concerned with different aspects of the measurement process. Each of these types uses logic, statistical verification or both to determine the degree of validity and has special value under certain conditions. Types of validity include content validity, predictive validity, and construct validity.

Sampling errors can lead to results which have a specific bias, or are only relevant to a specific subgroup. Consider this example: suppose that a film only appeals to a specialist audience—90% of them are devotees of this genre, and only 10% are people with a general interest in movies. Assume the film is very popular among the audience that views it, and that only those who feel most strongly about the film are inclined to rate the film online; hence the raters are all drawn from the devotees. This combination may lead to very high ratings of the film, which do not generalize beyond the people who actually see the film (or possibly even beyond those who actually rate it).

Qualitative description of categories improve the usefulness of a rating scale. For example, if only the points 1-10 are given without description, some people may select 10 rarely, whereas others may select the category often. If, instead, "10" is described as "near flawless", the category is more likely to mean the same thing to different people. This applies to all categories, not just the extreme points.

The above issues are compounded, when aggregated statistics such as averages are used for lists and rankings of products. User ratings are at best ordinal categorizations. While it is not uncommon to calculate averages or means for such data, doing so cannot be justified because in calculating averages, equal intervals are required to represent the same difference between levels of perceived quality. The key issues with aggregate data based on the kinds of rating scales commonly used online are as follow:

• Averages should not be calculated for data of the kind collected.
• It is usually impossible to evaluate the reliability or validity of user ratings.
• Products are not compared with respect to explicit, let alone common, criteria.
• Only users inclined to submit a rating for a product do so.
• Data are not usually published in a form that permits evaluation of the product ratings.

More developed methodologies include Choice Modelling or Maximum Difference methods, the latter being related to the Rasch model due to the connection between Thurstone's law of comparative judgement and the Rasch model.