The Invisible Characters Breaking Your Data

Two strings that look identical, compare unequal, and diff clean. Here is where they differ and how to see it.

By SimpleTools Daily editorial team · Published 2026-09-10 · Updated 2026-09-10 · How we check this

A support ticket arrives: a customer ID copied out of a web page will not match the same ID in the database. The two strings look identical on screen. They compare as equal length to the eye. Paste them into a diff tool and the diff shows nothing. Print their code points and one of them is nine characters long while the other is ten, because there is a U+200B ZERO WIDTH SPACE sitting between two digits, put there by whatever rendered the page to allow a line break.

This is a whole family of bugs, and they share one property: the evidence is invisible in every tool you would naturally reach for. Below is the cast list, what each character does, and—the part that surprises most people—the fact that your language's idea of "whitespace" disagrees with the next language's.

The usual suspects

• U+200B ZERO WIDTH SPACE. A line-break opportunity with no width. Category Cf (format), not a space character at all as far as Unicode's general category is concerned.
• U+00A0 NO-BREAK SPACE and U+202F NARROW NO-BREAK SPACE. These genuinely are spaces (category Zs) and render as one, but they are not U+0020. Word processors, CMSs and anything that has been through a rich-text editor produce them constantly.
• U+00AD SOFT HYPHEN. A conditional hyphenation point. Invisible unless the line breaks there. Category Cf.
• U+FEFF. At the start of a file it is a byte order mark. Anywhere else it is ZERO WIDTH NO-BREAK SPACE. The Unicode FAQ is explicit that an initial BOM "is only used as a signature" and that elsewhere U+FEFF "should normally not occur", recommending U+2060 WORD JOINER instead where a non-breaking join is actually wanted.
• U+200C ZERO WIDTH NON-JOINER and U+200D ZERO WIDTH JOINER. These are load-bearing: ZWJ builds multi-person and profession emoji sequences, and both control ligature and cursive joining in Arabic, Persian and Indic scripts. Do not strip these reflexively.
• U+200E LEFT-TO-RIGHT MARK and U+200F RIGHT-TO-LEFT MARK, plus the isolate characters defined in UAX #9. They change how text is displayed without changing the order of the code points stored. Reading rendered text is therefore not the same as reading the data.

Your whitespace is not their whitespace

Here is the finding that turns a nuisance into a genuine cross-service bug. Python and JavaScript disagree about which of these characters count as whitespace, and they disagree in opposite directions:

U+FEFF is in the ECMAScript definition of whitespace and is not in Python's. So "\uFEFFx".trim() returns "x" in Node, while the Python equivalent "\ufeffx".strip() returns the string unchanged. A sanitising function that works perfectly in your front end and is faithfully reimplemented in your Python backend will quietly stop working, on exactly the inputs that came from a file with a BOM.

Numeric parsing has the same asymmetry, and it fails in the more dangerous direction: int("\u00a01") returns 1 in Python, and Number("\u00A01") returns 1 in JavaScript. A no-break space in front of a number is silently tolerated. Put a zero width space there instead and Python raises ValueError while JavaScript yields NaN. One class of invisible character produces a loud error; another produces a plausible answer. The second is worse.

The BOM, specifically

Read a UTF-8 CSV written by Excel with a plain UTF-8 decoder and your first column header is not id—it is "\ufeffid", which prints as id in every log line and error message you will look at while debugging why row["id"] raises a KeyError. In Python the fix is to decode with the utf-8-sig codec, which consumes a leading BOM if present and is harmless if not. In JavaScript, JSON.parse('\uFEFF{}') throws Unexpected token, with the offending character rendered as nothing at all in the error message.

The general rule: strip a BOM at the point of decoding, once, based on knowing the file's encoding—not later with a string replace, and never by adding U+FEFF to a general "remove weird characters" list, because by then you may be deleting a legitimate word joiner from the middle of someone's text.

Two identical filenames that are not equal

Take the filename Nunez.txt with an acute accent on the u and a tilde on the n. There are two valid ways to encode it. In NFC (composed) form it is 9 code points and encodes to bytes including c3 ba and c3 b1. In NFD (decomposed) form it is 11 code points—plain u followed by U+0301 COMBINING ACUTE ACCENT, plain n followed by U+0303 COMBINING TILDE—encoding to 75 cc 81 and 6e cc 83. UAX #15 calls these canonically equivalent: they mean the same text, they render the same, and a conforming process should treat them as the same string.

File systems and version control compare bytes. macOS has historically stored filenames in a decomposed form, so a file created on a Mac and committed to git can arrive on a Linux checkout with a different byte sequence than the one in the index. The symptom is a repository that reports a file as both deleted and untracked, or two entries in a directory listing that look identical. Git carries a configuration option, core.precomposeunicode, precisely to normalise this on macOS clients—the existence of a platform-specific git setting for one normalisation form is a decent measure of how much trouble this causes.

The same problem appears without file systems anywhere you compare user-supplied strings for equality: usernames, tags, deduplication keys, cache keys. The fix is to normalise to a single form—NFC is the usual choice for storage and interchange—at the boundary where text enters your system, and to compare only normalised text.

One warning, because it is a common misunderstanding: normalisation is not a cleaning step. NFKC will fold a fullwidth A to A and the fi ligature to two letters, and it maps U+00A0 to an ordinary space. It does not remove U+200B, and it does not remove U+00AD. Both survive NFKC untouched. "Just NFKC everything" is not a defence against invisible characters.

Characters that look like other characters

The inverse problem is a visible character that is not the one you think. Replace the Latin a in apple.com with U+0430 CYRILLIC SMALL LETTER A and the two strings are visually indistinguishable in most fonts while being entirely different data—the Cyrillic version encodes to the punycode label xn--pple-43d.com. Greek omicron U+03BF, Cyrillic o U+043E and Latin o U+006F are three separate characters that render as the same circle.

Unicode addresses this in UTS #39, Unicode Security Mechanisms, which is the document to read before you invent your own defence. It defines confusable detection via a skeleton() transformation that maps characters to prototype forms so visually similar strings collapse together, distinguishes single-script, mixed-script and whole-script confusables, and defines restriction levels from ASCII-only upward. The important design insight there is that the workable defence is usually mixed-script detection—flagging an identifier that combines Latin and Cyrillic in one label—rather than a blocklist of dangerous characters, because the set of confusable pairs is far too large to enumerate by hand.

Finding them

The only reliable method is to stop looking at the text and look at the code points. Paste the string into the Characters to Unicode converter and read the list: an invisible character occupies a slot in that output like any other, and a U+200B between two digits is immediately obvious. The Unicode to Characters tool does the reverse, which is how you construct a test string containing a specific troublemaker to check that your validation actually rejects it.

On the command line, hexdump -C file | head will show a leading ef bb bf for a UTF-8 BOM. In a regex engine with Unicode property escapes—JavaScript with the u flag, or Python's third-party regex module, since the standard re module does not support \p{...}—the pattern [\p{Cf}\p{Zs}] catches format characters and non-ASCII spaces in one sweep. You can build and check patterns like that against sample strings in the Regex Tester before shipping them.

Stripping them, carefully

1. Normalise to NFC at the input boundary. One form, applied once, as early as possible.
2. Map every Zs character to U+0020, then collapse runs. This handles NBSP and its relatives without pretending they were never there.
3. Remove U+FEFF at decode time by choosing the right codec, not by string surgery afterwards.
4. Remove U+200B and U+00AD from fields that are identifiers, keys, or numbers. These have no legitimate role in such fields.
5. Do not blanket-delete category Cf. U+200D and U+200C are required by emoji sequences and by correct rendering of several scripts. A regex that strips all format characters will corrupt Persian, Hindi and any emoji built from a joiner sequence, which is a worse bug than the one you were fixing.
6. For identifiers, reject rather than clean. If a username, SKU or account number arrives with an invisible character in it, that is a signal about the input path, and silently repairing it hides the source while guaranteeing the same value arrives differently next time.

The unifying principle is that invisible characters are not a display problem to be papered over but a data-integrity problem with a specific location: the boundary where text enters your system. Normalise there, validate there, and be explicit about which characters you are removing and why—because the alternative is a sanitiser that grows one special case per incident and eventually deletes something a user needed.

Related tools

• Characters to Unicode — dump the code points of any pasted string
• Unicode to Characters — build a test string from code points
• Regex Tester — check a cleaning pattern before you ship it

Related tools

• Characters to Unicode
• Unicode to Characters
• Regex Tester