Computing Foundations › Inside the Machine › Day 4

Day 4: Binary and Data Representation: Bits, Bytes, and Numbers

Day 4 of 365 — Binary and Data Representation: Bits, Bytes, and Numbers

After this lesson you will be able to convert numbers between decimal, binary, and hexadecimal by hand, add and negate binary numbers the way the hardware does, and explain overflow, floating point, and quantization well enough to size an AI model from its bit format.

Course
Computing Foundations
Category
Inside the Machine
Reading time
≈ 40 min
Practical time
≈ 30 min
Lesson duration
1h 10m
Last verified
2026-07-12

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/computing-foundations/day-004-binary-and-data-representation-bits-bytes

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path — section / subsection / week / day:
    cd labs/sections/computing-foundations/day-004-binary-and-data-representation-bits-bytes
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work — read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

Everything in a computer is a number, and every number is a choice about how many bits to spend.


The idea in plain language

Diagram: how a byte becomes two hexadecimal digits

Bits carry no meaning. The type decides what they say.

Diagram: one pattern of bits interpreted as an integer, a float, and text


Historical background

WhenWhat arrivedWhy it mattered
1703Leibniz publishes binary arithmeticTwo symbols suffice for any number
1937Shannon links Boolean algebra to circuitsBinary becomes buildable
1956Buchholz coins “byte”A named unit for a group of bits
1963ASCII standardised7 bits, 128 characters, English only
1985IEEE 754One floating-point standard everyone follows
1991Unicode 1.0One code point per character, for every script
2018–bfloat16 spreads through ML hardwareRange kept, precision traded, on purpose

What it is — and what it is not

It is a set of agreements about what patterns of bits mean.

It is not:


Why it was created and what problems it solves

Diagram: two voltage states compared with ten


How it works

Diagram: forming a negative number with two's complement

Place values

Diagram: the eight place values of a byte decoding 10110101 into 181

Negative numbers: two’s complement

Decimals: sign, exponent, mantissa

Diagram: the same 32 bits read as the integer 1078530011 and as the float 3.1415927

A float splits its bits three ways:

PartWhat it controlsfloat32float16bfloat16
SignPositive or negative111
ExponentRange — how large or small858
MantissaPrecision — significant digits23107

Why 0.1 + 0.2 is not 0.3


An everyday analogy

A number written on a card, with no note saying what kind of number it is.

The card saysIt could mean
030405The 3rd of April 2005
030405Three pounds four and fivepence
030405A phone extension
030405A product code

Examples in practice

Diagram: the sequence of the Ariane 5 flight 501 failure

The Ariane 5, flight 501, 4 June 1996.

Why your 1 TB drive shows 931 GB. The maker counts in powers of 1,000; the OS counts in powers of 1,024. 10^12 ÷ 1024^3 ≈ 931. Nothing is missing.


Implications: security, privacy, performance, scalability, and cost

Diagram: powers of 1000 compared with powers of 1024


Alternatives: free, open source, and commercial

As with Day 1, “alternatives” here means other excellent routes into the same material, at different depths and formats.

ResourceTypeWhat it offersCost
The Day 4 lab in this courseFreeConversion drills by hand plus a shell toolkit you complete yourselfFree
Crash Course Computer Science, episodes 4 and 5 (PBS Digital Studios)Free videoBinary, hexadecimal, and how the ALU adds — visual and fastFree
Wikipedia: “Binary number”, “Two’s complement”, “IEEE 754”Free referenceRigorous, well-cited treatments of exactly today’s three pillarsFree
”Code” by Charles Petzold (2nd ed.), middle chaptersBook (commercial)The gentlest deep telling of binary, bytes, and codes in printBook purchase
From Nand to Tetris (Schocken & Nisan)Open courseBuild the adder and ALU that execute this lesson’s arithmeticFree materials
A programmer’s calculator (macOS Calculator’s Programmer view, or bc)Built-in softwareInstant dec/hex/bin conversion for checking your hand workFree

If you read only one thing beyond this lesson, make it Petzold’s chapters on binary codes — and check every hand conversion you ever do against a programmer’s calculator until the two of you always agree.

Diagram: float16 compared with bfloat16

Often confusedThe actual difference
Bit vs byteOne binary digit vs a group of eight
KB vs KiB1,000 bytes vs 1,024 — the source of the “missing” disk space
Integer vs floatExact within a range vs approximate across a huge range
float16 vs bfloat16More precision, less range vs the reverse
Encoding vs encryptionMaking data readable by machines vs unreadable by people

That last row matters. Base64 is an encoding, not a security measure: it is reversible by anyone, with no key.


When to use it — and when not to

Diagram: comparing floats with equality compared with a tolerance

Think about representation when choosing a dtype, a model will not fit, values look subtly wrong, or you are comparing decimals.

Ignore it when writing ordinary logic. Languages provide sensible defaults, and hand-choosing an integer width in a data-cleaning script is a way to introduce a bug rather than avoid one.

The tell: a number that is close but not equal, a value that wrapped to negative, or a file that is exactly twice or half the size you expected.


The AI connection

Diagram: the memory footprint of one model at four precisions


Knowledge check

Try these from memory before looking back:

  1. Convert 77 to binary using either the subtraction or division method, showing each step, then convert your answer to hexadecimal by grouping bits.
  2. Add the bytes 00110101 and 01001011 by hand, tracking every carry, and check your work in decimal.
  3. The byte 11101100 is a two’s-complement signed integer. What decimal number is it? Explain the role the leftmost bit played in your answer.
  4. In one paragraph, retell the Ariane 5 failure: what representation was converted to what, why the value went out of range on Ariane 5 but never on Ariane 4, and why the backup computer did not save the flight.
  5. A colleague stores prices as floats and cannot reproduce a customer’s total. Explain, using 0.1’s binary expansion, what is happening and what representation to use instead.
  6. A “2 TB” external drive shows up as roughly 1.8 TiB. Show the arithmetic proving the drive is not defective.

Hands-on exercise

Convert numbers by hand, then watch the machine disagree with you about decimals. Worked through fully in the Day 4 lab directory.

What you are checkingCommand
Binary of a numberpython3 -c 'print(bin(181))'
Hex of a numberpython3 -c 'print(hex(181))'
What 0.1 really isprintf '%.20f\n' 0.1
Whether the sum matchespython3 -c 'print(0.1 + 0.2 == 0.3)'
Disk size, both conventionsdf -h / and df -H /

Expected output

0b10110101
0xb5
0.10000000000000000555
False

Validate your work

Troubleshooting

Common mistakes


Practice assignment

  1. Convert by hand. Turn 77 into binary using either subtraction or repeated division, showing each step, then check with python3 -c 'print(bin(77))'.

  2. Find the drift. Run printf '%.20f\n' 0.1 and the same for 0.2 and 0.3. Identify the first decimal digit that goes wrong.

  3. Do the size arithmetic. For a 70-billion-parameter model, compute the footprint in GiB at float32, bfloat16, int8 and 4 bits. Which fit in the RAM you measured on Day 1?


Extension challenge

Explain the bfloat16 trade in terms of place value alone.

  1. Compute the ranges. A 5-bit exponent versus an 8-bit exponent — what is the largest representable value in each?

  2. Compute the cost. Three exponent bits were bought with three mantissa bits. How many significant decimal digits does that lose?

  3. Write the argument. Two paragraphs on why a noisy statistical process can afford that trade, and why a bank’s ledger cannot.


Quiz

Q1. What decimal number does the binary pattern 101010 represent?

  1. 52, because the digits sum with place values 16, 8, 4
  2. 20, because there are two 1-bits and a trailing zero
  3. 42, because the 1-bits sit on the place values 32, 8, and 2
  4. 84, because each 1-bit doubles the previous total
Show answer

Answer: C. 42, because the 1-bits sit on the place values 32, 8, and 2

Binary place values run 1, 2, 4, 8, 16, 32 from the right; 101010 has 1-bits under 32, 8, and 2, and 32 + 8 + 2 = 42.

Q2. Why do engineers write bit patterns in hexadecimal?

  1. Hexadecimal avoids the rounding errors that binary introduces
  2. Processors compute faster on hexadecimal numbers than on binary ones
  3. Hexadecimal can represent negative numbers while binary cannot
  4. Each hex digit maps to exactly four bits, shortening long binary strings
Show answer

Answer: D. Each hex digit maps to exactly four bits, shortening long binary strings

Hex is purely a human convenience: because 16 = 2⁴, each hex digit stands for one four-bit group, so conversion is table lookup with no arithmetic. The processor itself only ever sees bits.

Q3. In 8-bit two's complement, how do you form the representation of −44 from 44 (00101100)?

  1. Flip every bit and add 1, giving 11010100
  2. Reverse the order of the bits, giving 00110100
  3. Subtract the pattern from 11111111, giving 11010011
  4. Set the leftmost bit to 1, giving 10101100
Show answer

Answer: A. Flip every bit and add 1, giving 11010100

Two's complement negation is invert-then-increment: 00101100 flips to 11010011, and adding 1 gives 11010100. Adding that to 00101100 produces a ninth bit that falls off the register, leaving exactly zero.

Q4. What actually went wrong in the software aboard Ariane 5 flight 501 in 1996?

  1. A floating-point division by zero crashed the main flight computer
  2. A 64-bit float was converted to a 16-bit integer it could not fit
  3. Cosmic radiation flipped a bit in the guidance program's memory
  4. The engines were commanded in imperial units while guidance computed in metric
Show answer

Answer: B. A 64-bit float was converted to a 16-bit integer it could not fit

Alignment code reused from Ariane 4 converted the horizontal-bias value to a 16-bit signed integer. Ariane 5's faster, flatter ascent pushed the value past 32,767 about 37 seconds after ignition; the operand-error exception halted both identical inertial units, and the vehicle broke up and self-destructed.

Q5. Why does 0.1 + 0.2 print as 0.30000000000000004 in most programming languages?

  1. The addition circuit in the ALU has a known rounding defect
  2. The decimal point character is being misread as a thousands separator
  3. One tenth has no finite binary form, so the sum is microscopically off
  4. Floating-point numbers can only store values greater than 1 exactly
Show answer

Answer: C. One tenth has no finite binary form, so the sum is microscopically off

0.1 in binary is 0.000110011001100… repeating forever, just as 1/3 repeats in decimal. Floats store the nearest representable values, and the sum of those approximations rounds to a number a hair above 0.3. The hardware adds flawlessly; the inputs were never exactly 0.1 and 0.2.

Q6. A 7-billion-parameter model is quantized from float32 to int8. What happens to its memory footprint for weights?

  1. It drops to about 875 MB, because int8 uses one bit per weight
  2. It halves from about 28 GB to about 14 GB, because int8 stores two weights per byte
  3. It stays the same, because quantization only changes computation speed, not storage
  4. It drops from about 28 GB to about 7 GB, because each weight shrinks from 4 bytes to 1 byte
Show answer

Answer: D. It drops from about 28 GB to about 7 GB, because each weight shrinks from 4 bytes to 1 byte

Quantization to int8 stores each weight in a single two's-complement byte (with a scale factor for reconstruction) instead of four bytes: 7 billion × 4 bytes ≈ 28 GB becomes 7 billion × 1 byte ≈ 7 GB.

Q7. Both float16 and bfloat16 use 16 bits. What is the key difference, and why does machine learning generally prefer bfloat16?

  1. bfloat16 keeps float32's exponent range, trading mantissa precision
  2. float16 is an integer format while bfloat16 is a floating-point format
  3. bfloat16 stores two numbers per 16 bits, doubling effective memory capacity
  4. bfloat16 rounds decimal fractions like 0.1 exactly, avoiding accumulation error
Show answer

Answer: A. bfloat16 keeps float32's exponent range, trading mantissa precision

float16 spends its bits on precision (10-bit mantissa, 5-bit exponent); bfloat16 spends them on range (7-bit mantissa, 8-bit exponent, matching float32). Gradients and activations routinely exceed float16's ~65,504 ceiling, so keeping the exponent field is the safer trade for learning.

Q8. A drive sold as "1 TB" shows about 931 GiB in the operating system. Why?

  1. The filesystem permanently reserves 7% of every drive for error correction
  2. The maker counts in powers of 1,000 and the OS in powers of 1,024
  3. The operating system is misreading the drive's firmware and the true size is 1,024 GiB
  4. The drive shipped partly full of preinstalled recovery software
Show answer

Answer: B. The maker counts in powers of 1,000 and the OS in powers of 1,024

Both numbers describe the same bytes. Marketing prefixes are decimal (kilo = 1,000), while KiB/MiB/GiB are binary (kibi = 1,024); the gap compounds to about 7% at the tera scale. No storage is missing.

Glossary

bit
A single binary digit, 0 or 1 — the smallest unit of information, physically realized as one two-state switch.
byte
A group of 8 bits, able to hold 256 distinct patterns; the smallest unit of memory most machines address, standardized at 8 bits by IBM's System/360 in 1964.
nibble
Half a byte — 4 bits — which is exactly the amount of information one hexadecimal digit represents.
word
The natural chunk of bits a processor handles in one operation, typically 32 or 64 bits on modern machines.
place value
The worth of a digit position in a positional numeral system: powers of 10 in decimal, powers of 2 in binary, powers of 16 in hexadecimal.
hexadecimal
Base-16 notation using digits 0–9 and A–F, written with the 0x prefix; each digit encodes exactly four bits, making it the standard human-readable shorthand for raw binary.
two's complement
The standard representation of signed integers, in which −n is the bit pattern of 2ᵏ − n (flip the bits, add 1); it gives one unique zero and lets a single adder circuit perform both addition and subtraction.
sign bit
The leftmost bit of a signed integer; in two's complement it carries a negative place value (−128 in a byte), so a 1 there makes the whole number negative.
overflow
What happens when an arithmetic result exceeds what a fixed-width register can hold: the value silently wraps around, as when an unsigned byte computes 255 + 1 = 0.
floating point
Binary scientific notation for storing fractional numbers as a sign, a mantissa, and an exponent, standardized by IEEE 754 in 1985; it trades exactness for enormous range.
mantissa
The significant-digits field of a floating-point number (also called the significand or fraction), which controls precision; float32 gives it 23 bits.
exponent
The field of a floating-point number that scales the mantissa by a power of two, controlling range; when it cannot hold a needed power, the number overflows.
quantization
Reducing the bits used per number — for example mapping float32 model weights onto 256 int8 levels plus a scale factor — shrinking memory and data movement at a small cost in precision.
endianness
The convention for which byte of a multi-byte number is stored first in memory: least-significant first (little-endian, x86 and most ARM) or most-significant first (big-endian, traditional in network protocols).
gibibyte
The binary giga-unit, GiB = 1,024³ bytes, defined by the IEC in 1998 to end the ambiguity with the decimal gigabyte (GB = 10⁹ bytes) that makes a "1 TB" drive report about 931 GiB.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.