Decimal Places and Precision
Beginners in the field often ignore some simple facts concerning the
number of significant digits of a result. The use of pocket calculators
usually results in values with too many decimal places, regardless of whether
this is meaningful. In general, one should use only as many decimal places
as are compatible with the precision of your experiments. It may therefore
be necessary to determine the precision of a measurement by repeating it
several times and calculating the standard deviation.
||Suppose the outcomes of several repetitions of an experiment have been recorded as follows
The four decimal places used to denote the results
are meaningless, since the repeated measurements indicate that at most
the first decimal place is valid (a closer analysis shows that the
of the result is 12.21, with a standard deviation
In addition to the signal variation the accuracy of measurements may become even worse for quantities which are determined by the calculation of differences or ratios.
||The calculation of the electric resistance according to Ohm's law: R = U/I. In order to determine the resistance we measure the voltages U1 and U2 on both ends of the resistor. Let's assume that U1 is 8.97+/-0.01 V, U2 is 8.71+/-0.01 V, and the current I through the resistor is 0.015+/-0.001 A. From Ohm's law the resistance calculates as follows:
R = (8.97-8.71)/0.015 = 17.33 Ohm.
If we consider the measuring uncertainty, the calculated resistor may take the following extreme values:
Rmax = (8.98-8.70)/0.014 = 20 Ohm
Rmin = (8.96-8.72)/0.016 = 15 Ohm
Thus the resistance is known with low precision - the accuray is around +/- 15%. Thus the resistance should be rounded to the nearest integer number, decimal places are meaningless and superfluous.