Broken Characters After Migration: Check Your Encoding

Why broken characters only surface after the migration

A file does not contain text, it contains a sequence of bytes. Which characters come out of it is decided by the encoding someone reads those bytes with. UTF-8 encodes characters from the U+0000 to U+10FFFF range in sequences of one to four octets (RFC 3629); a “ü” (U+00FC) takes two bytes, a “€” (U+20AC) takes three. Read the same bytes with a single-byte character set and each of those bytes becomes a character of its own, so one “ü” turns into two visible characters. That is exactly what broken characters are. The bytes themselves are usually untouched; what is wrong is the instruction for how to read them.

Before the migration that instruction was mostly correct without anyone having stated it. The old web server set a default, the old database set a matching one, and the result looked right. A migration replaces precisely those tacit defaults: a different setting in the web server, a different default character set in the database, an export filter that makes a different assumption. That is why the break rarely appears everywhere at once. What comes from the database is hit first, while statically stored menu labels stay untouched. The home page therefore often looks fine even when several hundred detail pages do not. What else drifts measurably during a migration is covered in the article on monitoring a website migration.

• 1 to 4 octets are used per character from the U+0000 to U+10FFFF range in UTF-8 (RFC 3629).
• 1024 bytes is the limit within which an HTML document's encoding declaration must appear in full (HTML Living Standard).
• Exactly one meta-based encoding declaration per document is permitted (HTML Living Standard).
• windows-1252 is what the standard suggests as the fallback for every locale not named separately in its table; the normative fallback itself stays implementation-defined (HTML Living Standard).
• U+FFFD is the replacement character a decoder emits for invalid byte sequences in the “replacement” error mode (Encoding Standard).
• One to three bytes per character are stored by the utf8mb3 character set, with four bytes for supplementary characters only in utf8mb4 (MariaDB documentation).

These values come from standards texts and from the documentation of an open-source database, not from a survey. They therefore say nothing about how often a fault occurs, only what is supposed to hold and what limits the parts involved have. For fault-finding that is the more useful kind of number, because it can be checked against any individual case. How many pages in your own site are affected is a question only a measurement across the whole site can answer, and that is what this article is really about.

The chain: five places where an encoding is decided

Between the stored byte and the displayed character there are several places where an encoding is declared or assumed. First, storage: character set and collation of the database column. Second, the connection between application and database, which negotiates an encoding of its own. Third, the application itself, including the encoding of its template files. Fourth, transport, meaning the Content-Type header field of the HTTP response. Fifth, the document with its own declaration. If all five agree, the case is boring. A migration typically changes two or three of them, and what remains is a chain in which one link claims something different from the rest.

The header field of the HTTP response is the place that looks different after a server change particularly often. RFC 9110 states that a sender generating a message containing content should generate a Content-Type header field unless the intended media type is unknown to it (RFC 9110, section 8.3); the same section notes that in practice origin servers are not consistently configured to deliver the correct Content-Type (RFC 9110). A Content-Type without a charset parameter is permitted and consequential all the same, because the decision then drops one level. A Content-Type with the wrong charset parameter is the more awkward case, because it beats the declaration inside the document.

Precedence: which declaration wins when two disagree

The HTML Living Standard describes the order in which a document's encoding is determined as a procedure of its own. For fault-finding what matters most is that the order is fixed and that earlier steps override later ones.

1. Byte order mark: if the byte stream starts with a BOM, the supplied encoding argument is ignored and the BOM itself is skipped (HTML Living Standard). A BOM in a program file also produces output before the first header field, which is a second, unrelated fault.
2. User override: if someone has forced an encoding in the browser, the user agent may return that encoding; in the standard this step follows the BOM check and is worded as optional (HTML Living Standard). It explains why a page looks right on one workstation and wrong on every other.
3. Transport layer: if the transport layer specifies a supported encoding, that encoding is returned with confidence “certain” (HTML Living Standard). This is the charset parameter in Content-Type.
4. Document prescan: only then is the document itself examined. Authoring conformance requirements limit encoding declarations to the first 1024 bytes, and browsers are expected to prescan exactly that range (HTML Living Standard).
5. Fallback: if everything is still open, an implementation-defined default encoding applies that depends on the locale; for every locale not named separately the table gives windows-1252 (HTML Living Standard).

This order produces the finding that occurs particularly often after a server move: the document carries the right declaration, the header field carries the wrong one, and the rendering follows the header field. The declaration inside the document is therefore no protection against a wrong declaration in the header, only the safety net for the case where no charset is sent at all. Conversely, exceeding the limit of 1024 bytes — because a long comment, a consent layer or an embedded script sits before the head section — loses the declaration inside the document silently. The standard also permits exactly one meta-based declaration per document (HTML Living Standard); two contradictory declarations are not a safety net but a fault.

Patterns: what the broken text says about the cause

Broken characters come in a small number of easily distinguished forms. Reading the pattern correctly saves the detour through guesswork. The decisive question is whether the damage is only in the rendering or already in the stored byte.

Two of these rows deserve an explanation. The third is the only case in which the information is genuinely gone: the Encoding Standard prescribes that in the “replacement” error mode a decoder emits U+FFFD for an invalid byte sequence (Encoding Standard). The original byte generally cannot be recomputed from the result, because different faulty sequences produce the same replacement character. The fourth row is the case a second repair makes worse: re-encoding already double-encoded text creates a third layer. The way back here runs through the state before the migration, not through another pass.

What a post-migration check actually has to measure

A status light does not answer the encoding question. Status code 200, unchanged load time and a valid certificate are perfectly compatible with completely unreadable product copy; uptime monitoring is the baseline, not the answer. What has to be measured is the content itself, across the whole site rather than three spot checks, because the fault sits precisely where nobody looks often.

What also belongs in this picture is what changed between two states at all. A visual comparison shows the break where it is noticeable in the layout; how that method works and where its limits are is described in the article on detecting website changes by visual comparison. For the text side the article on detecting content changes applies, because it asks the same question from the direction of unauthorised edits.

Measuring it yourself: from the header field to the byte

The measurement needs no special equipment. Three requests narrow the fault down to one layer, and they do so in the same order a browser uses to decide.

The first request shows the situation that often follows a server change: a Content-Type without a charset parameter. The decision then drops one level, and the second request answers whether anything is declared there at all and whether it sits within the first 1024 bytes, because only that range counts (HTML Living Standard). The third request counts a character that hardly ever occurs in ordinary German or English prose and that sits inside almost every umlaut in mis-decoded UTF-8. A count well above zero is a strong indication; zero is not an acquittal, because stored data can also contain replacement characters.

# Classifies a delivered byte sequence: clean, mis-decoded or lost
TRACES = ("Ã", "Â", "â€", "")

def classify(raw: bytes) -> str:
    try:
        text = raw.decode("utf-8")
    except UnicodeDecodeError as error:
        return f"not a valid UTF-8 sequence from byte {error.start}"

    if "�" in text:
        return "replacement characters present: the loss is already in the data"

    if any(trace in text for trace in TRACES):
        # reverse via windows-1252: works only if the text was mis-decoded exactly once
        original = text.encode("cp1252", errors="strict").decode("utf-8")
        return "mis-decoded once, reversible: " + original

    return "unremarkable"


print(classify("Grüße".encode("utf-8")))
print(classify("Grüße".encode("utf-8").decode("cp1252").encode("utf-8")))

The script separates the three cases that can be told apart from the result: a byte sequence that is not valid UTF-8 at all, a text containing replacement characters, and a text whose way back through windows-1252 succeeds. The last branch is the useful one, because it supplies the proof along with the verdict: if the reverse calculation yields readable text again, the stored data was fine and only the reading instruction was wrong. If the reverse calculation fails, the text contains characters that do not exist in windows-1252 and the cause lies elsewhere.

An encoding fault does not announce itself. It waits until somebody searches for an order with an umlaut in the name and finds none.

— XICTRAQ

The stored data is the second scene

If the rendering is set up correctly and the text stays broken, the fault is in the stored data. There it has two common causes: the character set of the column and the encoding of the connection the application reads and writes through. Both are easily overlooked during a migration, because an export followed by an import can silently reinterpret either side. On top of that comes a limit that is documented and still regularly surprises people: the utf8mb3 character set stores one to three bytes per character and holds no supplementary characters, whereas utf8mb4 uses the same encoding with four bytes for supplementary characters (MariaDB documentation). Characters outside the basic plane, such as symbols and rare scripts, therefore fit into one column and not into the other.

• Check character set and collation per table and per column, not just the database default; individual columns often still carry the state from before the last conversion.
• Check the encoding of the connection the application reads and writes through; it is independent of the column's character set and tends to be reset when the runtime environment changes.
• Check the encoding of template and translation files; a single file in the old character set produces a fault that appears on only a few pages and is therefore overlooked for a long time.
• Read backups and exports back: a backup that already contains mis-decoded text makes the way back worthless.
• Check output paths separately — page, interface, invoice document, export file; each path makes its own assumption.
• Take a backup before any re-encoding run and measure how long the way back takes instead of estimating it.

Forms, search and file names: the quiet part

Visibly broken characters are the harmless part. What happens when a form is submitted is worse. Forms send their values in the document's encoding, and for the case where a character cannot be represented in that encoding the Encoding Standard defines an error mode of its own. The standard states explicitly that this mode causes a sequence to be emitted that cannot be distinguished from legitimate input and can therefore lead to silent data loss (Encoding Standard). In practice that means a surname with an umlaut is turned into a digit sequence, the application stores it as text, and nobody notices until somebody searches for that name.

The same chain affects three further places that are rarely checked along with it. First, search: an index that took in text in one encoding and receives queries in another returns zero results without reporting an error. Second, sorting: collation rules decide whether “Öl” comes before or after “Ozean”, and a conversion changes the order of lists somebody assumed to be stable. Third, file names transmitted in header fields; the name of a downloaded invoice document is a common place for broken characters, because it takes a different route than the page content. If a confirmation email belongs to the same process, the article on monitoring the double opt-in flow applies as well, because the same chain runs there over one more delivery path.

The sequence before and after a migration

The preceding sections turn into an order of work. It assumes that the state before the migration is recorded, because without a reference state it is no longer decidable after the switch whether a character was already wrong beforehand.

1. Before the migration, take stock: note the encoding per layer — column, connection, template files, header field, document declaration. Five lines per application, in writing.
2. Before the migration, reference sample: record a fixed list of pages that contain umlauts, an eszett, a euro sign, typographic quotation marks and at least one character outside the basic plane. That list becomes the yardstick later.
3. During the transition, measure the preview: check the new state under a preview address first and hold header field, first 1024 bytes and stored data there against the reference sample.
4. After the switch, header and document: measure the header field and the document declaration before anything in the content is changed. That measurement decides whether the repair belongs in the configuration or in the data.
5. After the switch, the data: run the text comparison across all pages and group the hits by pattern; a break that affects exactly one template looks different from one that affects half the database.
6. After the switch, input paths: re-check form flow, search, sorting and file names with the same test record.
7. Permanently: keep the text comparison and the check of the encoding declaration running in normal operation instead of switching them off after two weeks.

Encoding belongs in continuous monitoring, not on the migration checklist

A migration is the most conspicuous trigger, but not the only one. The same chain breaks when the runtime environment changes, when an update replaces a template, when a new output path is added and when a cache keeps serving an old state; on that, see the article on finding stale content after a deploy. The calendar plays a part too: comparing analyses across a clock-change day shows differences that are none, as the article on daylight saving time in monitoring shows. And anyone consuming content through interfaces gets the encoding question along with deprecation deadlines from the other side; how that is announced is covered in the article on reading sunset and deprecation headers.

The standards are unambiguous here, and that makes continuous monitoring straightforward. The Encoding Standard requires authors to use UTF-8 and to identify it with the “utf-8” label (Encoding Standard). The HTML Living Standard requires that the actual character encoding used to encode a document be UTF-8 regardless of whether a declaration is present (HTML Living Standard), and it requires authoring tools to default to UTF-8 for newly created documents (HTML Living Standard). An encoding declaration is required even when all characters are in the ASCII range, because it is needed for form input and for URLs generated by scripts (HTML Living Standard). Turn those four sentences into check rules and you have a test that can be automated; how technical checks of that kind run permanently is described in the article on automating technical SEO monitoring. How the findings come together in one view is shown in the demo environment.