Fundamentals of Statistics contains material of various lectures and courses of H. Lohninger on statistics, data analysis and chemometrics......click here for more.

Decimal Places and Precision

Beginners in the field often ignore some simple facts concerning the number of significant digits of a result. The use of pocket calculators usually results in values with too many decimal places, regardless of whether this is meaningful. In general, one should use only as many decimal places as are compatible with the precision of your experiments. It may therefore be necessary to determine the precision of a measurement by repeating it several times and calculating the standard deviation.

Example: Suppose the outcomes of several repetitions of an experiment have been recorded as follows
   12.3075
   12.3351
   11.9949
   12.2722
   12.3117
   12.0766
The four decimal places used to denote the results are meaningless, since the repeated measurements indicate that at most the first decimal place is valid (a closer analysis shows that the average of the result is 12.21, with a standard deviation of 0.14).

In addition to the signal variation the accuracy of measurements may become even worse for quantities which are determined by the calculation of differences or ratios.

Example: The calculation of the electric resistance according to Ohm's law: R = U/I. In order to determine the resistance we measure the voltages U1 and U2 on both ends of the resistor. Let's assume that U1 is 8.97+/-0.01 V, U2 is 8.71+/-0.01 V, and the current I through the resistor is 0.015+/-0.001 A. From Ohm's law the resistance calculates as follows:

R = (8.97-8.71)/0.015 = 17.33 Ohm.

If we consider the measuring uncertainty, the calculated resistor may take the following extreme values:

Rmax = (8.98-8.70)/0.014 = 20 Ohm
Rmin = (8.96-8.72)/0.016 = 15 Ohm

Thus the resistance is known with low precision - the accuray is around +/- 15%. Thus the resistance should be rounded to the nearest integer number, decimal places are meaningless and superfluous.

Last Update: 2012-10-08