In computer science, the **precision** of a numerical quantity is a measure of the detail in which the quantity is expressed. This is usually measured in bits, but sometimes in decimal digits. It is related to precision in mathematics, which describes the number of digits that are used to express a value.

Some of the standardized precision formats are

- Half-precision floating-point format
- Single-precision floating-point format
- Double-precision floating-point format
- Quadruple-precision floating-point format
- Octuple-precision floating-point format

Of these, octuple-precision format is rarely used. The single- and double-precision formats are most widely used and supported on nearly all platforms. The use of half-precision format has been increasing especially in the field of machine learning since many machine learning algorithms are inherently error-tolerant.^{[1]}

Precision is often the source of rounding errors in computation. The number of bits used to store a number will often cause some loss of accuracy. An example would be to store "sin(0.1)" in IEEE single precision floating point standard. The error is then often magnified as subsequent computations are made using the data (although it can also be reduced).

- Arbitrary-precision arithmetic
- Extended precision
- IEEE754 (IEEE floating point standard)
- Integer (computer science)
- Significant figures
- Truncation
- Approximate computing

**^**Mittal, Sparsh (May 2016). "A Survey of Techniques for Approximate Computing".*ACM Comput. Surv*. ACM.**48**(4): 62:1–62:33. doi:10.1145/2893356.

Experimental data in science and engineering is data produced by a measurement, test method, experimental design or quasi-experimental design. In clinical research any data produced are the result of a clinical trial. Experimental data may be qualitative or quantitative, each being appropriate for different investigations.

Generally speaking, qualitative data are considered more descriptive and can be subjective in comparison to having a continuous measurement scale that produces numbers. Whereas quantitative data are gathered in a manner that is normally experimentally repeatable, qualitative information is usually more closely related to phenomenal meaning and is, therefore, subject to interpretation by individual observers.

Experimental data can be reproduced by a variety of different investigators and mathematical analysis may be performed on these data.

Floating-point arithmeticIn computing, **floating-point arithmetic** (**FP**) is arithmetic using formulaic representation of real numbers as an approximation so as to support a trade-off between range and precision. For this reason, floating-point computation is often found in systems which include very small and very large real numbers, which require fast processing times. A number is, in general, represented approximately to a fixed number of significant digits (the significand) and scaled using an exponent in some fixed base; the base for the scaling is normally two, ten, or sixteen. A number that can be represented exactly is of the following form:

where significand is an integer, base is an integer greater than or equal to two, and exponent is also an integer. For example:

The term *floating point* refers to the fact that a number's radix point (*decimal point*, or, more commonly in computers, *binary point*) can "float"; that is, it can be placed anywhere relative to the significant digits of the number. This position is indicated as the exponent component, and thus the floating-point representation can be thought of as a kind of scientific notation.

A floating-point system can be used to represent, with a fixed number of digits, numbers of different orders of magnitude: e.g. the distance between galaxies or the diameter of an atomic nucleus can be expressed with the same unit of length. The result of this dynamic range is that the numbers that can be represented are not uniformly spaced; the difference between two consecutive representable numbers grows with the chosen scale.

Over the years, a variety of floating-point representations have been used in computers. In 1985, the IEEE 754 Standard for Floating-Point Arithmetic was established, and since the 1990s, the most commonly encountered representations are those defined by the IEEE.

The speed of floating-point operations, commonly measured in terms of FLOPS, is an important characteristic of a computer system, especially for applications that involve intensive mathematical calculations.

A floating-point unit (FPU, colloquially a math coprocessor) is a part of a computer system specially designed to carry out operations on floating-point numbers.

Significant figuresThe significant figures (also known as the significant digits) of a number are digits that carry meaning contributing to its measurement resolution. This includes all digits except:

All leading zeros;

Trailing zeros when they are merely placeholders to indicate the scale of the number (exact rules are explained at identifying significant figures); and

Spurious digits introduced, for example, by calculations carried out to greater precision than that of the original data, or measurements reported to a greater precision than the equipment supports.Significance arithmetic are approximate rules for roughly maintaining significance throughout a computation. The more sophisticated scientific rules are known as propagation of uncertainty.

Numbers are often rounded to avoid reporting insignificant figures. For example, it would create false precision to express a measurement as 12.34525 kg (which has seven significant figures) if the scales only measured to the nearest gram and gave a reading of 12.345 kg (which has five significant figures). Numbers can also be rounded merely for simplicity rather than to indicate a given precision of measurement, for example, to make them faster to pronounce in news broadcasts.

TruncationIn mathematics and computer science, truncation is limiting the number of digits right of the decimal point.

This page is based on a Wikipedia article written by authors
(here).

Text is available under the CC BY-SA 3.0 license; additional terms may apply.

Images, videos and audio are available under their respective licenses.