Programming with Python › Python Setup and First Programs › Day 46
Day 46: Numbers, Math, and Precision
After this lesson you will be able to choose the right Python number type for any task, explain why 0.1 + 0.2 is not exactly 0.3, compare floats safely, and use the decimal module for exact money math.
Hands-on lab for this lesson
Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/programming-with-python/day-046-numbers-math-and-precision
- Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git cd ai-roadmap-365.github.io - Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path — section / subsection / week / day:
cd labs/sections/programming-with-python/day-046-numbers-math-and-precision - Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work — read it before running anything.
- Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
bash tests/run_tests.sh # or the test command named in the lab README
You can also open the lab as a local page (works offline, shows the file tree and expected output).
Learning objectives
By the end of this lesson you will be able to:
- Distinguish Python's numeric types — int (exact, unlimited), float (fast, approximate), Decimal (exact base-10), and Fraction (exact ratio) — and pick the right one for a task
- Use the arithmetic operators + - * / // % ** correctly, and explain why / always returns a float while // stays an exact integer operation
- Explain, in terms of binary and a fixed mantissa, why floating-point cannot represent 0.1 exactly, and connect it to the base-10 impossibility of writing one third
- Compare floats safely with math.isclose or rounding instead of ==, and know when each is appropriate
- Compute an exact currency total with the decimal module, building Decimals from strings rather than floats
- Use floor division and modulo to split whole quantities into units and remainders without any rounding error
- Connect float precision to machine-learning practice: why float32/float16/bfloat16 choices trade memory, speed, and accuracy, and what quantization gives up
Prerequisites
- Day 45 — Variables, Types, and Assignment (you can create variables and read basic Python)
- Day 43 — Python installed and runnable as python3
- Day 4 — binary and data representation (numbers are bit patterns)
Why this matters
-
0.1 + 0.2 != 0.3is not a Python quirk. It is how every language on every machine stores fractions, and today is where you meet it. -
Every model weight is a float, and every training run accumulates the rounding this lesson describes.
-
nanin your loss is almost always an overflow, a division by zero, or a log of zero — all of them number problems. -
Money must never be a float, and knowing why saves a class of bug that is expensive in the literal sense.
-
Comparing floats with
==is a bug, and the correct alternative takes one line.
The idea in plain language
-
Integers in Python are exact and unbounded. They grow until memory runs out.
-
Floats are approximations. They trade exactness for range and speed.
-
Most decimal fractions cannot be stored exactly in binary, the way one third cannot be written exactly in decimal.
-
So arithmetic accumulates tiny errors, and comparing two floats for equality asks the wrong question.
-
DecimalandFractionexist for when exactness matters more than speed.
Historical background
-
Early machines each had their own float format, so the same program gave different answers on different hardware.
-
IEEE 754, published in 1985, standardised the layout and the rounding rules, and essentially every processor has implemented it since.
-
That is why the same rounding error appears in every language. It is the hardware’s arithmetic, not Python’s.
-
The standard also defined
infandnanas values rather than errors, so a computation could continue past an overflow. -
Python’s integers became arbitrary-precision in Python 3, unifying the old
intandlong— which is why factorials do not overflow here. -
Decimalarrived to serve money and accounting, where a fraction of a cent is a legal problem rather than a rounding one.
What it is — and what it is not
A float is:
-
A fixed-width approximation: sign, exponent and mantissa in 64 bits.
-
Exact for integers up to 2^53, and for fractions whose denominator is a power of two.
-
Fast, because the processor implements it directly.
It is not:
-
Not a decimal. It stores binary fractions, and most decimal fractions are not among them.
-
Not exact.
0.1is stored as the nearest representable value, which is not0.1. -
Not associative.
(a + b) + ccan differ froma + (b + c), which matters when summing many numbers. -
Not suitable for money. Any system that adds fractions of a currency unit wants
Decimal.
Why it was created and what problems it solves
-
Range. A float covers roughly 10^-308 to 10^308 in 64 bits, which no fixed-point format of that size can.
-
Speed. The hardware does it directly, which is why every numerical library is built on it.
-
A single portable standard, so the same program gives the same answer everywhere.
-
Graceful overflow.
infandnanlet a computation continue and be inspected rather than aborting at the first problem. -
The trade is exactness, and Python provides
DecimalandFractionfor when that trade is the wrong one.
How it works
How a float is stored
-
Three fields in 64 bits: one sign bit, eleven exponent bits, fifty-two mantissa bits.
-
The value is the mantissa scaled by two to the exponent — a binary fraction, not a decimal one.
-
A fraction is exact only if its denominator is a power of two. One half, one quarter and three eighths are fine.
-
One tenth is not. In binary it repeats forever, exactly as one third repeats in decimal, so it is stored as the nearest representable value.
-
0.1 + 0.2gives0.30000000000000004because two approximations were added and the result was rounded again. -
Integers are exact up to 2^53. Beyond that, consecutive integers stop being representable, which is Day 24’s big-identifier problem.
Comparing floats
0.1 + 0.2 == 0.3 # False
math.isclose(0.1 + 0.2, 0.3) # True
-
==asks whether two approximations landed on the same value, which is rarely the question you mean. -
math.iscloseasks whether they are near enough, with both a relative and an absolute tolerance. -
Use an absolute tolerance near zero. Relative tolerance is meaningless when the expected value is
0.
Integer division, modulo, and rounding
-
/always gives a float, even for6 / 3.//gives the floor. -
Floor division rounds towards negative infinity, so
-7 // 2is-4, not-3. -
%follows from that, so-7 % 2is1— the sign follows the divisor. -
round()uses banker’s rounding: halves go to the nearest even number, soround(0.5)is0andround(1.5)is2. -
That is deliberate. Always rounding halves up biases a long series of sums upwards; going to even cancels out.
When exactness matters
| Type | Exact? | Speed | Use it for |
|---|---|---|---|
int | Yes, unbounded | Fast | Counting, indexing, identifiers |
float | No | Fastest | Measurement, science, anything with a tolerance |
Decimal | Yes, decimal | Slow | Money, and anything audited |
Fraction | Yes, rational | Slowest | Exact ratios, symbolic work |
-
Decimal("0.1") + Decimal("0.2")is exactlyDecimal("0.3"). -
Construct
Decimalfrom a string, not a float.Decimal(0.1)inherits the float’s error beforeDecimalever sees it. -
Fraction(1, 3)is exactly one third, and stays exact through arithmetic.
Special values
-
infresults from overflow and from dividing a non-zero float by zero in some libraries. -
nanmeans “not a number” — the result of0/0,inf - inf, or the log of a negative. -
nanis not equal to anything, including itself. That is by design, and it is whyx != xis a valid test for it. -
Use
math.isnanrather than that trick in code anyone else will read. -
nanpropagates. One in a dataset poisons every sum and mean computed from it, silently.
An everyday analogy
A ruler with finite markings.
| Measuring | Floats |
|---|---|
| The finest marking on the ruler | The mantissa’s precision |
| Reading to the nearest marking | Rounding to the nearest representable value |
| A length that falls between markings | A fraction with no exact representation |
| Adding many measured lengths | Accumulated error |
| ”Within a millimetre” | math.isclose |
| A tape measure that reaches further, less finely | The exponent trading precision for range |
-
The ruler is not wrong. It has finite markings, and every reading is to the nearest one.
-
Adding twenty readings adds twenty roundings, which is why a long sum drifts.
-
Asking whether two measurements are exactly equal is asking the wrong question about a measured thing.
Examples in practice
-
A test asserting
total == 0.3that fails on a machine where it passed before. -
A running total that drifts after summing a million values, fixed by
math.fsum. -
A price computed as
0.1 * 3and billed as0.30000000000000004. -
A loss that becomes
nanat step 4,000 and staysnanforever after, because it propagates. -
round(2.5)giving2, which is banker’s rounding working as designed.
Implications: security, privacy, performance, scalability, and cost
-
Security: float comparison in a permission check is a bug. Thresholds computed differently on two paths can disagree.
-
Integer overflow does not exist in Python, which removes a whole class of vulnerability present in C — at the cost of speed on very large values.
-
Performance: floats are what hardware accelerates.
Decimalis roughly two orders of magnitude slower and is the right answer anyway when money is involved. -
Accumulated error grows with the number of operations, which is why summing a large array in a naive loop is less accurate than a pairwise or compensated sum.
-
Cost:
nanis expensive because it is silent. A poisoned value propagates through an entire training run before anyone notices the metric. -
Cost: a currency bug is a legal and financial problem, not a rounding one.
Alternatives: free, open source, and commercial
| Tool | Type | Where it fits |
|---|---|---|
math.isclose | Free, standard library | Comparing floats, always |
decimal.Decimal | Free, standard library | Money, invoices, anything audited |
fractions.Fraction | Free, standard library | Exact ratios |
math.fsum | Free, standard library | An accurate sum of many floats |
| NumPy | Free, open source | Arrays, and explicit control of precision |
float32 / float16 | Free, in NumPy and frameworks | Half the memory, half the precision |
-
math.fsumis worth knowing today. It sums without accumulating the usual error, and it costs one import. -
Lower precision is a deliberate AI trade-off, not a mistake:
float16halves memory and bandwidth, at real risk of underflow.
Comparison with related concepts
| Term | What it actually is |
|---|---|
| Mantissa | The significant digits |
| Exponent | The scale they are multiplied by |
| Precision | How many digits are kept |
| Range | How large and small a value can be |
inf | Overflow, as a value rather than an error |
nan | Not a number; unequal to everything, including itself |
| Banker’s rounding | Halves go to the nearest even |
When to use it — and when not to
-
Use
intwhenever the quantity is countable. Counts, indices and identifiers are never floats. -
Use
floatfor measurement, where a tolerance is meaningful anyway. -
Use
Decimalfor money, constructed from strings. -
Use
math.isclosefor every float comparison, with an absolute tolerance near zero. -
Do not compare floats with
==. -
Do not construct
Decimalfrom a float. The error is already baked in. -
Do not accumulate a large sum naively when accuracy matters.
math.fsumexists.
The AI connection
-
Every model weight is a float, and every training run accumulates the rounding error this lesson describes.
-
float32versusfloat16is the same trade at a smaller mantissa: half the memory, half the precision, and a real risk of underflow. -
Loss going to
nanis usually this — an overflow, a division by zero, or a log of zero, propagating through every subsequent operation. -
Never compare floats with
==in a training loop or a test. Use a tolerance, which is whatmath.iscloseandtorch.allcloseare for. -
Money is not a float. Anything you bill for wants
Decimal, and that rule survives into every system you will build.
Knowledge check
quiz:
question: "A test asserts that 0.1 + 0.2 equals 0.3 and fails. What is the correct fix?"
options:
- "Compare with a tolerance, using math.isclose, because binary floats cannot represent either operand exactly"
- "Round both sides to one decimal place before comparing them"
- "Use Decimal(0.1) + Decimal(0.2) so the arithmetic is exact"
- "Increase the float precision by using a 128-bit type"
answer_index: 0
explanation: "Neither 0.1 nor 0.2 has an exact binary representation, so their sum is the nearest representable value to 0.3 rather than 0.3 itself. Rounding hides the issue rather than addressing it, and Decimal(0.1) inherits the float's error before Decimal sees it — the string form would be needed. A tolerance asks the question that actually makes sense about approximations."
Hands-on exercise
See the error, measure it, then eliminate it where it matters. Worked through fully in the Day 46 lab directory.
| What you are doing | Command |
|---|---|
| See the classic result | 0.1 + 0.2 |
| See what is really stored | format(0.1, '.20f') |
| Compare properly | math.isclose(0.1 + 0.2, 0.3) |
| Find the integer limit | 2**53 == 2**53 + 1 |
| Use exact decimals | Decimal("0.1") + Decimal("0.2") |
| See the float trap | Decimal(0.1) |
| Sum accurately | sum([0.1]*10) against math.fsum([0.1]*10) |
Expected output
0.30000000000000004
0.10000000000000000555
True
True
0.3
0.1000000000000000055511151231257827021181583404541015625
0.9999999999999999 1.0
-
2**53 + 1equals2**53as a float — the integer is no longer representable. -
Decimal(0.1)shows the float’s actual value, which is why the string form matters. -
fsumgot exactly1.0where the naive sum did not.
Validate your work
- You saw the stored value of
0.1to twenty decimal places - You replaced a
==comparison withmath.isclose - You found where consecutive integers stop being representable as floats
- You demonstrated that
Decimal(0.1)andDecimal("0.1")differ - You compared
sumwithmath.fsumon a case where they disagree -
bash tests/run_tests.shreports 0 failures
Troubleshooting
-
isclosestill returnsFalsenear zero. Relative tolerance is meaningless there. Passabs_tol. -
Decimalarithmetic raises. You mixed it with a float. Convert explicitly. -
round()surprises you. Banker’s rounding.round(0.5)is0. -
Your
nancheck with==never fires.nanis not equal to itself. Usemath.isnan.
Common mistakes
- Comparing floats with
==. Decimal(0.1)instead ofDecimal("0.1").- Using a float for money.
- Assuming
nanwill raise. It propagates silently instead.
Practice assignment
-
Find three decimal fractions that are exact in binary and three that are not, and explain the pattern.
-
Write a currency calculation twice, with
floatand withDecimal, and find an input where they disagree by a cent. -
Accumulate a million small floats and compare
sum,math.fsumand a runningDecimalfor accuracy and speed. -
Introduce a
naninto a list and trace how far it propagates through a mean, a max and a sort.
Extension challenge
-
Reproduce non-associativity: find three floats where
(a+b)+cdiffers froma+(b+c), and explain which rounding caused it. -
Find the smallest float greater than 1.0 using
math.nextafter, and relate it to the 52-bit mantissa. -
Compare
float32andfloat64on the same computation in NumPy, and find an input where the difference is visible. -
Implement compensated summation yourself and compare it with
math.fsum.
Quiz
Q1. In Python, what does the expression 0.1 + 0.2 == 0.3 evaluate to, and why?
- True, because Python rounds all decimals to two places automatically
- False — 0.1, 0.2 and 0.3 are not exact in binary, so it is off
- True, because floating-point arithmetic is always exact
- It raises an error, because you cannot compare floats
Show answer
Answer: B. False — 0.1, 0.2 and 0.3 are not exact in binary, so it is off
Each of 0.1, 0.2, and 0.3 is stored as the nearest available binary fraction, and the rounding errors do not cancel: 0.1 + 0.2 is stored as 0.30000000000000004, which is not the stored value of 0.3, so the comparison is False.
Q2. Which numeric type should you use to total a shopping cart so the amount reconciles exactly to the cent?
- float, because it is the fastest
- int, because money is always whole numbers
- Decimal, because it does exact base-10 arithmetic
- complex, because it has two parts
Show answer
Answer: C. Decimal, because it does exact base-10 arithmetic
Decimal performs exact base-10 arithmetic, so decimal fractions like 0.10 and 0.20 add up exactly. float uses base-2 and introduces rounding error; Decimal (or integer cents) is the correct choice for money.
Q3. What is the difference between 7 / 2 and 7 // 2 in Python 3?
- They are identical; both give 3.5
- 7 / 2 gives 3 (an int) and 7 // 2 gives 3.5 (a float)
- 7 // 2 raises an error because you cannot floor-divide odd numbers
- 7 / 2 gives 3.5 (float); 7 // 2 gives 3 (floor)
Show answer
Answer: D. 7 / 2 gives 3.5 (float); 7 // 2 gives 3 (floor)
In Python 3, / is true division and always returns a float (7 / 2 is 3.5), while // is floor division that divides and rounds down to a whole number (7 // 2 is 3).
Q4. You need to check whether two computed floats are effectively equal. What is the recommended approach?
- Use math.isclose(a, b), which tests equality within a small tolerance
- Use == directly; float comparison is always exact
- Convert both to strings and compare the strings
- Multiply both by 1000 and hope they match
Show answer
Answer: A. Use math.isclose(a, b), which tests equality within a small tolerance
Because floats carry tiny rounding errors, == often reports False for values that are mathematically equal. math.isclose(a, b) returns True when the values are within a small tolerance, which is the correct way to compare computed floats.
Q5. Why is Decimal("0.1") exact but Decimal(0.1) is not?
- There is no difference; both are exact
- Decimal(0.1) copies the float's error; the string is exact
- Decimal("0.1") is actually the inexact one
- Decimal(0.1) rounds to the nearest whole number
Show answer
Answer: B. Decimal(0.1) copies the float's error; the string is exact
Decimal(0.1) receives the float 0.1, which already holds 0.1000000000000000055..., and stores that error. Decimal("0.1") parses the string "0.1" and stores exactly one tenth. Always build money Decimals from strings.
Q6. What is special about Python's int type compared with integers in many other languages?
- Python ints are limited to 64 bits and overflow silently
- Python ints are stored as floats internally
- Python ints have arbitrary precision and never overflow
- Python ints can only hold values up to one million
Show answer
Answer: C. Python ints have arbitrary precision and never overflow
Python integers have arbitrary precision: they are exact and unbounded, growing to use as much memory as needed, so an expression like 2 ** 100 produces the full exact 31-digit result with no overflow.
Q7. A colleague uses Python's random module to generate password-reset tokens. What is the problem?
- random is too slow for generating tokens
- There is no problem; random is cryptographically secure
- random only produces whole numbers
- random is predictable; use the secrets module
Show answer
Answer: D. random is predictable; use the secrets module
The random module uses a predictable Mersenne Twister generator: given enough outputs its future values can be reconstructed. For anything that must be unguessable — tokens, passwords, one-time codes — use the secrets module, which draws from a cryptographically secure source.
Q8. What do the 52 mantissa bits and 11 exponent bits of a 64-bit float control, respectively?
- The mantissa controls precision; the exponent controls range
- The mantissa controls the sign and the exponent controls the color
- Both control only the range
- The mantissa controls range and the exponent controls precision
Show answer
Answer: A. The mantissa controls precision; the exponent controls range
In IEEE 754 double precision, the 52 mantissa (fraction) bits set the precision — about 15 to 17 significant decimal digits — while the 11 exponent bits set the scale, giving the enormous range up to about 1.8 x 10^308.
Glossary
- int
- Python's integer type: an exact whole number with arbitrary precision, meaning it grows as large as memory allows and never overflows.
- float
- Python's floating-point type: a number with a decimal point stored in 64 bits as an IEEE 754 double, fast and wide-ranging but limited to about 15 to 17 significant digits of precision.
- IEEE 754
- The international standard, published in 1985, that defines how binary floating-point numbers are laid out in bits and how arithmetic on them rounds, so the same program gives the same result on any conforming hardware.
- floating-point error
- The small, deterministic rounding difference that arises because most decimal fractions (like 0.1) have no exact representation in binary and must be stored as the nearest available value.
- mantissa
- The field of a floating-point number that holds its significant binary digits (also called the fraction); a 64-bit double has 52 mantissa bits, which set its precision.
- exponent
- The field of a floating-point number that sets the scale — which power of two the mantissa is multiplied by; a 64-bit double has 11 exponent bits, which set its range.
- floor division
- The // operator, which divides two numbers and rounds the result down to a whole number (toward negative infinity); 7 // 2 is 3.
- modulo
- The % operator, which returns the remainder left over after floor division; 7 % 2 is 1, and (a // b) * b + (a % b) always reconstructs a.
- Decimal
- A standard-library type (from the decimal module) that performs exact base-10 arithmetic to a chosen precision, correct for money; build it from a string, as Decimal("0.1"), never from a float.
- Fraction
- A standard-library type (from the fractions module) that stores a number as an exact ratio of two integers, so values like one third stay exact through arithmetic.
- math.isclose
- A function in the math module that reports whether two floats are equal within a small tolerance; the correct way to compare computed floats instead of using ==.
- rounding
- Reducing a number to a chosen number of digits; Python's built-in round uses banker's rounding (round half to even), so round(2.5) is 2 and round(3.5) is 4.
- quantization
- Storing numbers with fewer bits (for example a model's parameters in 8 bits instead of 32) to save memory and bandwidth, at the cost of coarser precision.
Sources and further reading
- Numeric Types — int, float, complex — Python Software Foundation (accessed 2026-07-12)
- decimal — Decimal fixed-point and floating-point arithmetic — Python Software Foundation (accessed 2026-07-12)
- Floating Point Arithmetic: Issues and Limitations — Python Software Foundation (accessed 2026-07-12)
- math — Mathematical functions — Python Software Foundation (accessed 2026-07-12)
- IEEE 754 — Wikipedia (accessed 2026-07-12)
Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.