Text you did not type by hand almost always contains characters you cannot see. They arrive from word processors, from copying out of a web page or a PDF, from content management systems that helpfully tidy your spacing, and increasingly from anything that generated text for you.
They occupy no visible width, or exactly the width of a space, and they compare as different characters to everything that compares characters. That combination is what makes them expensive: the symptom never resembles the cause.
The ones that actually turn up
Non-breaking space (U+00A0). By far the most common. It looks exactly like a space and is not one. Word inserts them, web publishing tools insert them, and copying from a rendered page brings them with you. Every exact-match search, every string comparison and every CSV importer treats it as a different character.
Zero-width space (U+200B). No width at all. It sits inside a word, splits it in two as far as any search is concerned, and shows nothing. Often introduced by systems that wanted to allow a line break at a particular point.
Byte order mark (U+FEFF). Belongs at the very start of a file to declare its encoding, and frequently ends up in the middle of one after files are concatenated. As the first character of a CSV it corrupts the first column heading, which is why the first column is so often the one that will not map.
Soft hyphen (U+00AD). A suggestion of where a word may be broken if it needs breaking. Invisible when it is not needed, and permanently inside the word regardless.
Directional marks (U+200E, U+200F). Control how left-to-right and right-to-left text sit together. Necessary in mixed-direction writing and confusing everywhere else.
Zero-width joiner and non-joiner (U+200D, U+200C). Control whether adjacent letters connect. Also the glue inside emoji sequences, which is why a multi-person emoji becomes several separate people when text is cleaned carelessly.
The one that is not junk
It is worth stopping on this before reaching for a cleaner, because most advice on this subject is written as though every invisible character is a defect.
In Persian, Arabic and several Indic scripts, the zero-width non-joiner is a real part of the writing system. It stops two letters joining where they would otherwise join, and removing it changes the word — not its formatting, the word. The directional marks are similarly load-bearing in any text mixing Arabic or Hebrew with Latin script or numbers.
So the rule is not "strip everything invisible". It is: strip broadly in English prose, in code, and in data files, and strip selectively in anything else. A cleaner that lets you choose is doing you a favour even when it feels like a fussier tool.
Where they come from
Word processors. Non-breaking spaces around punctuation, soft hyphens from hyphenation, and non-breaking spaces holding numbers to their units.
Copying from a web page. You inherit whatever the page contained, including every its author or CMS inserted.
Copying from a PDF. Which brings its own problem — the line breaks — covered in copying out of a PDF without the line breaks.
Generated text. Output from a language model routinely carries non-breaking spaces and occasionally zero-width characters. This is inherited from the text it learnt from rather than inserted on purpose, and it is worth being precise about what that does and does not tell you: it is a reason to clean the text, and it is not a test for whether a person wrote it. Anything copied from a web page carries the same characters.
What breaks, and why the symptom is always misleading
Search finds nothing although the words are visibly present. There is a non-breaking space where your query has an ordinary one.
Two identical-looking strings compare as different. A deduplicator keeps both. A lookup fails. A password is rejected after being pasted.
Sorting comes out wrong. A leading zero-width character sorts before every letter, so one entry jumps to the top of an otherwise alphabetical list.
Code will not run with an error pointing at a line that looks perfect. A zero-width space inside an identifier is invisible in every editor that is not looking for it.
A CSV column will not map, usually the first one, which is the byte order mark.
In each case the visible evidence says the text is fine, so the time goes into re-checking the logic. Checking the bytes first is a habit worth acquiring.
How to find them
The quickest test needs no tool: select the suspect text and watch the selection highlight. A zero-width character shows up as a sliver of highlight where there is no letter. Count characters against what you expect — a counter reporting more characters than you can see is telling you something is there.
To find exactly what and where, a frequency breakdown lists every distinct character in the text with its count. Anything you do not recognise in that list is worth looking up before deciding what to do with it.
How to remove them
A whitespace cleaner handles the ordinary case: convert non-breaking spaces to ordinary ones, drop zero-width characters, collapse runs of spaces, and trim the ends of lines. That is the right default for English prose and for data.
For a specific character, find and replace with a regular expression is more precise, and the pattern is worth knowing: [\u200B-\u200D\uFEFF] matches the zero-width family and the byte order mark, and \u00A0 matches the non-breaking space specifically. Written as escapes rather than as the characters themselves — a pattern containing a literal zero-width space is a pattern nobody can read or copy correctly, which is the same problem one level up.
Do it in the browser rather than pasting into a website that processes it, for a reason that is specific here: the text people most often need to clean is text that was malfunctioning in a document they cared about.
The short version
When something impossible is happening to text — a search that fails on visible words, a comparison that disagrees with your eyes — check for invisible characters before you check anything else. It is nearly always that, and it is nearly always the last thing anyone tries.
Clean aggressively in English and in data. Clean selectively in anything written in a script where invisible characters do work.