We've updated our
Privacy Policy effective December 15. Please read our updated Privacy Policy and tap

Study Guides > Mathematics for the Liberal Arts

C1.04: Precision

Section 3. Precision of a number  In elementary school we learned that doing arithmetic with the number 18 and the number 18.0 gives the same results. But when we are thinking of approximate numbers, those two ways of reporting a number do not imply the same thing. In particular, they imply a different rounding precision for the number, so they imply a different set of actual values that could have led to this rounded number.   These are important distinctions when we work with measured numbers. Review. Names of the places in a number. Example 1.   Consider the number 38,145.   The places, from the right, are the ones, tens, hundreds, thousands, ten thousands. So this number is

[latex]\begin{array}{l}\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,5\text{ ones}\\\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,4\text{ tens}\\\,\,\,\,\,\,\,\,\,\,\,\,\,\,1\text{ hundred}\\\,\,\,\,\,\,\,\,8\text{ thousands}\\\underline{+3\text{ ten-thousands}}\\\,\,\,\,\,38,145\end{array}[/latex]

  Example 2. Consider the number 1.2479. The left-most place, before the decimal point, is the ones place. Immediately after the decimal is the tenths place, then the hundredths place, then the thousandths place, then the ten-thousandths place.

[latex]\begin{array}{l}1\text{ one}\\\,\,\,\,2\text{ tenths}\\\,\,\,\,\,\,\,\,4\text{ hundredths}\\\,\,\,\,\,\,\,\,\,\,\,7\text{ thousandths}\\\underline{+\,\,\,\,\,\,\,\,\,9\text{ ten-thousandths}}\\1.2479\end{array}[/latex]

Precision:   We can define the precision of a number in three different ways.
  1. State it in words.
  2. State it with a number.
  3. Imply it by how the number is written.
Example 3:   The amount $5200 is measured to the nearest hundred dollars. That can be stated as
  1. The amount $5200 is measured to the nearest hundred dollars.
  2. The amount $5200 is measured to the nearest 100 dollars.
  3. The value $5200 has the obvious implied precision.
Example 4: The number 73.123 is rounded to the nearest one-thousandth. That can be stated as
  1. The number 73.123 is rounded to the nearest one-thousandth.
  2. The number 73.123 is rounded to the nearest 0.001.
  3. The number 73.123 has the obvious implied precision.
Example 5: Round 18.038 to the nearest tenth. Solution: Since we need to cut off everything past the tenths place, we must cut off the marked-out part here 18.038. Since the 3 at the beginning of the 38 is less than 5, we round down and the answer is 18.0. We could communicate that as “The answer is 18, rounded to the nearest tenth.” But no one would do that because it is confusing. It is much less confusing if we always include the tenths digit when we report the result of rounding to the nearest tenth. So here we could say “the answer is 18.0, rounded to the nearest tenth.”   In fact, technical people would not give the words here because everyone would understand that the number 18.0 implies that it is rounded to the nearest tenth. Example 6. Round 2.1397354 to the nearest 0.001. Solution. Since we need to cut off everything past the thousandths place, we cut off the marked-out part here 2.1397354 and notice that the first digit of the 7354 that we need to cut off is greater than half, so we must round up. But the previous digit is 9, so to round up we must go to 10. That means the answer is 2.140 since the 3 in front of the 9 must go up 1. The answer is 2.140. Example 7: We measured the length of pipe as 6.30 meters. What is the implied precision of that number? Solution: Since this number is given in hundredths of a meter, and there is no reason to write the zero on the end except to give the hundredths place, this implies that the number is measured to the nearest hundredth, that is, the nearest 0.01. Example 8: In this summary of a financial report of a small business, the amount spent on utilities last year is given as $13,000.   What is the implied rounding precision of that number? Solution: The implied rounding precision is that the number is rounded to the nearest thousand dollars. Caution: (Extending the previous example.) If the actual value was $12,973.37 and we wanted to round to the nearest hundred dollars, the answer would also be $13,000.   In our previous examples, where we were doing all the rounding somewhere after the decimal point, we could easily communicate the rounding precision by extra zeros as needed, so the rounding precision could always be given unambiguously. But when we have large numbers, where the extra zeros are not after the decimal, then the rounding precision cannot always be given unambiguously by just writing the number. In those cases, if we want to be completely clear in reporting a rounded number, we must state the rounding precision in words or numbers and not rely on our readers just using the implied rounding precision. When we are reading rounded numbers reported by others, we should look at other information given besides the actual number to see whether the implied rounding precision is consistent with that. When scientists and others working in technical fields report approximate numbers, they sometimes use the concept of identifying significant digits. The significant digits in a number are those which give an actual value that was measured or recorded, as opposed to the digits in a number which merely to indicate the size of the number. (Think of the usual meaning of the word “significant.” These digits are significant because they are actually measured.) Identifying the significant digits is simply another way of identifying the implied precision of the number.
Number Implied precision in words Implied precision in numbers Significant digits underlined
17.3 Tenths 0.1 17.3
18 Ones 1 18
18.0 Tenths 0.1 18.0
100.6 Tenths 0.1 100.6
83.20 hundredths 0.01 83.20
97.1080 Ten-thousandths 0.001 97.1080
13000 Thousands 1000 13000
20800 Hundreds 100 20800
In identifying significant digits, we have focused here mainly on identifying the right-most significant digit to be consistent with the implied rounding precision. However, when we change the units of a measurement after the measurement has been made, we don’t want that to change the number of significant digits.   Specifically notice the second number in our previous table.
Measurement   and its precision Completely clear correct report in centimeters Fairly clear correct report Not a clear report Completely clear correct report in meters # of sig digits
12 cm, 1 12 cm 0.12 m 2
2.7 cm, 0.1 2.7 cm 0.027 2
18.2 cm, 0.1 18.2 cm 0.182 m 3
33 cm, 0.01 33.0 cm 0.330 m 3
30 cm, 1 30. cm 0.30 m 2
Estimate of 20 cm, 10 20 cm 0.2 m 1
Estimate of 180 cm, 10 180 cm 1.8 m 2
Estimate of 300 cm, 10 300 cm 3.0 m 2
Estimate of 300 cm, 100 300 cm 3 m 1
Some books give “rules for identifying significant digits.” These include
  1. leading zeros in a decimal number less than 1 are not significant
  2. trailing zeros in a number greater than 1 are not significant
The reason behind these rules is that, unless you are specifically told otherwise, these zeros are in the number just to indicate the size of the number and not to indicate an actual measured value. The second and last rows in the table are examples of these situations.  Going Deeper: In scientific and technical work, the measurement units will usually be chosen so that the rounding precision is either to the nearest whole number or is in the decimal portion so that it can be given unambiguously. Sometimes they will use scientific notation to convey the rounding precision unambiguously. See the course web pages for supplemental information for this Topic. This includes a discussion of how scientists and other technical workers usually choose measurement units so that the implied precision is almost always at the level of 1, 0.1, or 0.01, so that the numbers’ implied precision is easy to read and unambiguous. Some discussion of the use of scientific notation is also included.  

Licenses & Attributions

CC licensed content, Shared previously

  • Mathematics for Modeling. Authored by: Mary Parker and Hunter Ellinger. License: CC BY: Attribution.