Text in a file is bytes. Something has to decide which byte means which letter, and for about thirty years everybody decided differently — a different encoding per region, all using the same byte values for different characters.
Unicode fixed the first half by assigning a number to every character in every script. UTF-8 fixed the second half by choosing how to write those numbers as bytes, and it did it in a way that made adoption almost free.
Two ideas, kept separate
Unicode is a catalogue. The letter A is 65, the Arabic letter alef is 1575, an emoji is somewhere above 128,000. Just numbers, no storage.
UTF-8 is a way of storing those numbers. UTF-16 and UTF-32 are other ways. A file described only as "Unicode" has not told you the thing you need to read it, which is why that word on an encoding menu is unhelpful.
Why UTF-8 won
It uses one byte for the first 128 characters — exactly the ASCII set, with exactly the ASCII values.
Which means every ASCII file already was a UTF-8 file, unchanged, and every program that handled ASCII kept working. Nothing had to be converted and nothing had to be rewritten on the day of the switch. That backwards compatibility is the whole reason it displaced the alternatives.
Beyond ASCII it uses two bytes for most European and Middle Eastern scripts, three for most Asian ones, and four for emoji and rarer scripts. Text that is mostly English costs almost nothing; text that is mostly Chinese costs 50% more than UTF-16 would, which is the trade that was accepted.
It also has a property that keeps parsers honest: the leading byte of a character says how many bytes follow, so you can jump into the middle of a file and find the next character boundary. UTF-16 cannot do that.
The byte order mark
UTF-16 comes in two flavours depending on which end of a two-byte number goes first, so it needs a marker at the start of the file to say which. That marker is the byte order mark.
UTF-8 has single bytes and therefore no byte order to mark. It does not need one.
It got one anyway, as a signature: the three bytes EF BB BF, meaning "this file is UTF-8". And that turned out to be genuinely useful for a format with nowhere to declare its encoding — which is to say, for CSV.
Why Excel needs it and parsers hate it
Double-click a UTF-8 CSV without a BOM in Excel and non-ASCII text is mangled, because Excel assumes a legacy regional encoding. Add the BOM and it opens correctly. This is the single most practical fact in this article.
The cost is that everything else sees three unexpected bytes before the first character. A parser reading the header row finds the first column named \uFEFFid instead of id, and the lookup fails with an error that shows two identical-looking strings.
That is what utf-8-sig is for: read with it and the mark is stripped; write with it and the mark is added. Plain utf-8 leaves it alone in both directions.
So the rule is about the destination. A file for a spreadsheet gets a BOM; a file for a program does not. When it is both, add it and tell the program to expect it.
Mojibake, and why it is recoverable
When you see Ø£Øمد or café, UTF-8 bytes have been read as a single-byte encoding. Each real character has become the two or three characters its bytes mean in that other scheme.
The important part: nothing has been lost. The bytes on disk are correct and the reader is wrong, so telling the reader the right encoding restores the text exactly.
It becomes unrecoverable only when the mangled version is saved. At that point the wrong characters are the real content, and undoing it means reversing the specific misinterpretation — which works when you can identify it and not otherwise.
Practical rules
Declare it wherever you can: <meta charset="utf-8"> as the first thing in an HTML head, charset=utf-8 in a content type header, UTF-8 as the database and connection encoding.
In MySQL, use utf8mb4 rather than utf8. The one called utf8 is a three-byte subset that cannot store emoji or several scripts, and it is a long-standing trap.
Never assume the encoding of a file you were sent. Look at it, or ask.
The short version
Unicode numbers the characters; UTF-8 stores those numbers as bytes, staying byte-identical to ASCII for the first 128. A BOM is an optional three-byte signature that Excel relies on and parsers trip over. And mojibake is a reading error rather than damage — until someone saves it.