What binary encoding actually means
Every piece of text your computer stores or transmits is ultimately a sequence of bytes — numbers in the range 0 to 255. Binary encoding is the system that maps human-readable characters to those numbers and back again. Without an agreed-upon encoding, a byte value of 233 could mean the letter “é” in one system, a completely different glyph in another, or nothing valid at all.
Understanding the full chain from character to bit is not just academic. It is the knowledge that lets you avoid the garbled text, the mysterious question-mark diamonds, and the broken string comparisons that plague so many applications when they cross system boundaries. You can experiment with any of these encodings directly using the Text to Binary tool, which shows you the exact bit pattern for any string you enter.
ASCII: the original 128
ASCII — the American Standard Code for Information Interchange — was defined in 1963 and finalized as a standard in 1968. It assigns a 7-bit code to 128 characters: the 26 uppercase and 26 lowercase English letters, the digits 0 through 9, punctuation marks like period, comma, and semicolon, and 33 non-printing control characters inherited from teleprinter conventions.
Those 33 control characters include familiar ones like tab (9), newline (10), carriage return (13), and the null byte (0), as well as historical relics like BEL (7, which rang a physical bell on a teletype machine) and ESC (27, which begins terminal escape sequences that are still in heavy use today).
The 7-bit design was deliberate. At the time, adding an eighth bit for data rather than parity was considered extravagant. In practice, 8-bit bytes quickly became the norm, and the eighth bit was often used to extend ASCII with 128 additional characters for accented Latin letters, line-drawing symbols, and other region-specific glyphs. This produced what is loosely called “extended ASCII,” though that term is misleading — there was never one single extended ASCII standard. Instead, dozens of incompatible code pages proliferated, each vendor or region defining the upper 128 slots differently.
The capital letter A is code point 65, or 01000001 in binary. The lowercase a is 97, or 01100001. The digit zero is 48, or 00110000. These mappings are foundational — every modern encoding scheme is built to be compatible with them for the basic Latin characters.
Extended ASCII and code pages
When personal computing spread internationally in the 1980s, ASCII’s 128 characters were simply not enough. IBM introduced Code Page 437 for the original PC, which filled the upper 128 slots with line-drawing characters, Greek letters, and accented characters useful for Western European languages. Microsoft later introduced Windows-1252, sometimes called “ANSI,” which diverged from CP437 and became the dominant encoding for Western European Windows systems through the 1990s and into the 2000s.
ISO published a series of 8-bit encodings called ISO 8859, with ISO 8859-1 (Latin-1) covering Western European languages and later variants covering other scripts. Each of these encodings was incompatible with the others in the upper 128 range, meaning a file encoded in Windows-1252 would display incorrectly when opened under the assumption of ISO 8859-2 or KOI8-R (a Cyrillic encoding common in Russia).
This is the root cause of one of the most common encoding bugs developers encounter: a file written on a Windows system in Windows-1252 gets uploaded, emailed, or committed to a repository and then read by software that assumes UTF-8. Characters like the em dash (0x97 in Windows-1252) or the left double quotation mark (0x93) have no legal representation in that byte range under UTF-8, producing the replacement character or outright parse errors.
Unicode: one encoding to rule them all
Unicode was conceived precisely to end the code page chaos. The Unicode Standard assigns a unique number — called a code point — to every character in every human writing system, past and present. The current version assigns code points to over 150,000 characters covering scripts from Latin and Cyrillic to Cuneiform, Egyptian Hieroglyphs, and mathematical symbols.
Code points are written in the form U+XXXX, where XXXX is a hexadecimal number. The capital A is U+0041, the euro sign is U+20AC, and the emoji for a smiling face is U+1F600. The total space of Unicode code points runs from U+0000 to U+10FFFF, which is roughly 1.1 million possible positions, though most are currently unassigned.
Knowing a code point tells you what character is intended. It does not, by itself, tell you how to store that code point as bytes. That is the job of a Unicode encoding.
UTF-16: the 16-bit encoding
UTF-16 encodes most common characters as 16-bit (2-byte) values. The Basic Multilingual Plane (BMP), which covers code points U+0000 through U+FFFF, can each be stored in exactly two bytes. This includes virtually all characters used in modern languages.
Characters outside the BMP — code points above U+FFFF — require a pair of 16-bit values called a surrogate pair. High surrogates are in the range 0xD800–0xDBFF and low surrogates in 0xDC00–0xDFFF. Together they encode a code point by a specific arithmetic formula. A lone surrogate without its partner is invalid.
UTF-16 also requires specifying byte order: should the two bytes of a 16-bit value be stored high byte first (big-endian) or low byte first (little-endian)? This is where the byte-order mark becomes important for UTF-16 files.
Windows internally uses UTF-16 for its string APIs, which is why many Windows APIs accept “wide strings” with the W suffix. JavaScript engines also store strings internally as UTF-16, which occasionally produces surprising behavior when you split a string containing emoji at a byte or code-unit boundary rather than a code-point boundary.
UTF-8: the variable-width encoding
UTF-8 is the encoding that won the web. As of recent surveys, over 98% of web pages use UTF-8. It encodes each Unicode code point as a sequence of one to four bytes according to the following rules.
Code points in the range U+0000 to U+007F — the first 128 characters, identical to ASCII — are stored as a single byte with the same value as the ASCII code. This means any ASCII file is automatically a valid UTF-8 file, which made UTF-8 easy to adopt in ASCII-dominated systems.
Code points from U+0080 to U+07FF, covering most Latin-script accented characters and many other common symbols, are stored as two bytes. The first byte begins with the bit pattern 110xxxxx and the second begins with 10xxxxxx. The x positions carry the actual bits of the code point.
Code points from U+0800 to U+FFFF, covering the rest of the Basic Multilingual Plane, are stored as three bytes with the pattern 1110xxxx 10xxxxxx 10xxxxxx.
Code points from U+10000 to U+10FFFF, covering supplementary planes including most emoji and many historic scripts, are stored as four bytes with the pattern 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx.
The leading byte patterns — 0xxxxxxx for single-byte, 110xxxxx for two-byte start, 1110xxxx for three-byte start, 11110xxx for four-byte start — make it unambiguous where each character begins. Any byte starting with 10xxxxxx is a continuation byte, never a start byte. This allows a parser to re-synchronize after corruption or truncation by simply scanning forward until it finds a non-continuation byte.
To see this in action with any text you like, paste a string into the Text to Binary tool and switch between UTF-8, ASCII, and Unicode modes. The per-character breakdown table shows exactly which bytes are assigned to each character.
The byte-order mark (BOM)
The Unicode code point U+FEFF is named “ZERO WIDTH NO-BREAK SPACE” but is used in a second role as a byte-order mark. When placed at the very beginning of a file, it signals the encoding and byte order of what follows.
In UTF-16, the BOM appears as either FE FF (big-endian) or FF FE (little-endian). A decoder that reads the first two bytes can determine byte order from which sequence it sees.
In UTF-8, the BOM is the three-byte sequence EF BB BF. Because UTF-8 is byte-order-independent, the UTF-8 BOM serves only to identify the encoding, not the byte order. The Unicode Standard says UTF-8 BOMs are neither required nor recommended, and in practice they cause more problems than they solve. Notepad on older versions of Windows was notorious for prepending the UTF-8 BOM to every file it saved, which broke shebang lines in shell scripts (the kernel expects the very first byte to be #, not EF), caused CSV parsers to include the BOM as part of the first column header, and generally confused any tool that was not expecting it.
Common encoding bugs developers encounter
The replacement character U+FFFD (rendered as a black diamond containing a question mark) is the most visible sign of an encoding mismatch. It appears when a UTF-8 decoder encounters a byte sequence that is not valid UTF-8. The most frequent cause is reading a Windows-1252 or Latin-1 file while the software assumes UTF-8 — the bytes 0x80 through 0x9F are defined in Windows-1252 but are illegal in UTF-8.
Mojibake is a related phenomenon where text has been decoded with the wrong encoding, producing a sequence of valid but wrong characters. The French phrase for “café” saved in UTF-8 produces the bytes 63 61 66 C3 A9. If those bytes are read as Windows-1252, the sequence C3 A9 displays as é rather than é. The result looks like garbage but is technically valid characters in the wrong encoding.
Double UTF-8 encoding happens when software encodes a string to UTF-8, treats the resulting bytes as a Latin-1 string, then encodes it to UTF-8 again. The two-byte sequence C3 A9 for é gets encoded as C3 83 C2 A9, producing four bytes where two were needed. The output reads as é instead of é even in a UTF-8 aware reader. This happens most often in database stacks where the connection charset and the table charset disagree.
String length confusion is subtler. In Python 2, the len() function on a UTF-8 string returned the byte count, not the character count. In Python 3 and JavaScript, len or .length counts UTF-16 code units, so a single emoji outside the BMP counts as 2. Understanding which layer you are measuring — bytes, code units, or code points — is essential for correct string manipulation.
For more context on how encodings interact with URLs, see the URL Encoding Guide, which covers the related challenge of encoding characters safely for use in HTTP requests.
Practical takeaways
Always declare your encoding explicitly rather than relying on defaults. In HTML that means <meta charset="UTF-8"> in the document head. In Python 3 it means specifying encoding="utf-8" when opening files rather than relying on the platform default. In MySQL it means using utf8mb4 rather than the poorly named utf8, which only supports three-byte sequences and silently drops emoji.
Store and process text in Unicode (UTF-8 preferred) and convert to other encodings only at the boundaries where external systems require it. Detect rather than guess encoding when consuming data from external sources — the chardet library in Python and the Encoding Sniffing Algorithm in browsers both use statistical heuristics to identify the most likely encoding from byte patterns, though no detection method is perfect.
Binary encoding is ultimately about contracts: the writer and the reader must agree on the rules. When they do not, the symptoms range from mildly annoying mojibake to data loss. Keeping the encoding explicit, consistent, and UTF-8 where possible eliminates most of these problems before they start.