Verification and validation of computer simulation models

A model that has face validity appears to be a reasonable imitation of a real-world system to people who are knowledgeable of the real world system.[5] Face validity is tested by having users and people knowledgeable with the system examine model output for reasonableness and in the process identify deficiencies.[1] An added advantage of having the users involved in validation is that the model's credibility to the users and the user's confidence in the model increases.[1][5] Sensitivity to model inputs can also be used to judge face validity.[1] For example, if a simulation of a fast food restaurant drive through was run twice with customer arrival rates of 20 per hour and 40 per hour then model outputs such as average wait time or maximum number of customers waiting would be expected to increase with the arrival rate.

Assumptions made about a model generally fall into two categories: structural assumptions about how system works and data assumptions. Also we can consider the simplification assumptions that are those that we use to simplify the reality.[7]

Assumptions made about how the system operates and how it is physically arranged are structural assumptions. For example, the number of servers in a fast food drive through lane and if there is more than one how are they utilized? Do the servers work in parallel where a customer completes a transaction by visiting a single server or does one server take orders and handle payment while the other prepares and serves the order. Many structural problems in the model come from poor or incorrect assumptions.[5] If possible the workings of the actual system should be closely observed to understand how it operates.[5] The systems structure and operation should also be verified with users of the actual system.[1]

There must be a sufficient amount of appropriate data available to build a conceptual model and validate a model. Lack of appropriate data is often the reason attempts to validate a model fail.[3] Data should be verified to come from a reliable source. A typical error is assuming an inappropriate statistical distribution for the data.[1] The assumed statistical model should be tested using goodness of fit tests and other techniques.[1][3] Examples of goodness of fit tests are the Kolmogorov–Smirnov test and the chi-square test. Any outliers in the data should be checked.[3]

Are those assumptions that we know that are not true, but are needed to simplify the problem we want to solve.[7] The use of this assumptions must be restricted to assure that the model is correct enough to serve as an answer for the problem we want to solve.

The model is viewed as an input-output transformation for these tests. The validation test consists of comparing outputs from the system under consideration to model outputs for the same set of input conditions. Data recorded while observing the system must be available in order to perform this test.[3] The model output that is of primary interest should be used as the measure of performance.[1] For example, if system under consideration is a fast food drive through where input to model is customer arrival time and the output measure of performance is average customer time in line, then the actual arrival time and time spent in line for customers at the drive through would be recorded. The model would be run with the actual arrival times and the model average time in line would be compared with the actual average time spent in line using one or more tests.

Statistical hypothesis testing using the t-test can be used as a basis to accept the model as valid or reject it as invalid.

The hypothesis to be tested is

H₀ the model measure of performance = the system measure of performance

versus

H₁ the model measure of performance ≠ the system measure of performance.

The test is conducted for a given sample size and level of significance or α. To perform the test a number n statistically independent runs of the model are conducted and an average or expected value, E(Y), for the variable of interest is produced. Then the test statistic, t₀ is computed for the given α, n, E(Y) and the observed value for the system μ⁰

${\displaystyle t_{0}={(E(Y)-u_{0})}/{(S/{\sqrt {n}})}}$ and the critical value for α and n-1 the degrees of freedom

${\displaystyle t_{a/2,n-1}}$ is calculated.

If

${\displaystyle \left\vert t_{0}\right\vert >t_{a/2,n-1}}$

reject H₀, the model needs adjustment.

There are two types of error that can occur using hypothesis testing, rejecting a valid model called type I error or "model builders risk" and accepting an invalid model called Type II error, β, or "model user's risk".[3] The level of significance or α is equal the probability of type I error.[3] If α is small then rejecting the null hypothesis is a strong conclusion.[1] For example, if α = 0.05 and the null hypothesis is rejected there is only a 0.05 probability of rejecting a model that is valid. Decreasing the probability of a type II error is very important.[1][3] The probability of correctly detecting an invalid model is 1 - β. The probability of a type II error is dependent of the sample size and the actual difference between the sample value and the observed value. Increasing the sample size decreases the risk of a type II error.

A statistical technique where the amount of model accuracy is specified as a range has recently been developed. The technique uses hypothesis testing to accept a model if the difference between a model's variable of interest and a system's variable of interest is within a specified range of accuracy.[8] A requirement is that both the system data and model data be approximately Normally Independent and Identically Distributed (NIID). The t-test statistic is used in this technique. If the mean of the model is μ^m and the mean of system is μ^s then the difference between the model and the system is D = μ^m - μ^s. The hypothesis to be tested is if D is within the acceptable range of accuracy. Let L = the lower limit for accuracy and U = upper limit for accuracy. Then

H₀ L ≤ D ≤ U

versus

H₁ D < L or D > U

is to be tested.

The operating characteristic (OC) curve is the probability that the null hypothesis is accepted when it is true. The OC curve characterizes the probabilities of both type I and II errors. Risk curves for model builder's risk and model user's can be developed from the OC curves. Comparing curves with fixed sample size tradeoffs between model builder's risk and model user's risk can be seen easily in the risk curves.[8] If model builder's risk, model user's risk, and the upper and lower limits for the range of accuracy are all specified then the sample size needed can be calculated.[8]

Confidence intervals can be used to evaluate if a model is "close enough"[1] to a system for some variable of interest. The difference between the known model value, μ₀, and the system value, μ, is checked to see if it is less than a value small enough that the model is valid with respect that variable of interest. The value is denoted by the symbol ε. To perform the test a number, n, statistically independent runs of the model are conducted and a mean or expected value, E(Y) or μ for simulation output variable of interest Y, with a standard deviation S is produced. A confidence level is selected, 100(1-α). An interval, [a,b], is constructed by

${\displaystyle a=E(Y)-t_{a/2,n-1}S/{\sqrt {n}}\qquad and\qquad b=E(Y)+t_{a/2,n-1}S/{\sqrt {n}}}$,

where

${\displaystyle t_{a/2,n-1}}$

is the critical value from the t-distribution for the given level of significance and n-1 degrees of freedom.

If |a-μ₀| > ε and |b-μ₀| > ε then the model needs to be calibrated since in both cases the difference is larger than acceptable.

If |a-μ₀| < ε and |b-μ₀| < ε then the model is acceptable as in both cases the error is close enough.

If |a-μ₀| < ε and |b-μ₀| > ε or vice versa then additional runs of the model are needed to shrink the interval.

If statistical assumptions cannot be satisfied or there is insufficient data for the system a graphical comparisons of model outputs to system outputs can be used to make a subjective decisions, however other objective tests are preferable.[3]