ParsingGENERALSIMPLIFIEDPLATFORM-SPECIFIC

The HTML Tokenizer

Bytes become characters become tokens, through a specified state machine that has no fatal errors and that the tree builder can reach in and reconfigure.

What actually happens What it costs the browser How to build it

The intent, the obvious build, and why it breaks

Every lesson starts where the work starts: someone wanted an outcome, and the first implementation that comes to mind has a problem.

The question

How does a stream of bytes become the start tags, attributes and text the browser works with — and why does malformed HTML never throw?

The user intent

Someone requested a page. They expect to see it. They will never see a parse error, because the platform has decided that a broken page rendering imperfectly beats a broken page rendering not at all.

The obvious build

The browser reads the HTML, splits it on angle brackets, and builds elements. Valid HTML parses; invalid HTML produces some kind of error the browser reports.

Why it breaks

There is no error to report. HTML defines *parse errors* but requires the parser to recover from every one of them and keep going, so <p><div>Hello</p> produces a document rather than a diagnostic. Your linter has opinions; the parser has none.

How it breaks in a real browser

There is no error to report. HTML defines *parse errors* but requires the parser to recover from every one of them and keep going, so <p><div>Hello</p> produces a document rather than a diagnostic. Your linter has opinions; the parser has none.
Splitting on angle brackets is wrong inside <script> and <style>, where the tokenizer switches to a mode in which < is ordinary text. A string literal containing </script> closes your script element mid-expression, and the rest of your JavaScript becomes page content.
It is wrong inside <textarea> and <title> too, which are in a third mode where tags are text but character references are still decoded — which is why & shows up as & in a textarea and <b> does not go bold.
Before any of that, the bytes have to become characters. Get the encoding wrong and every non-ASCII character in the document is wrong, in a way that looks like a font problem and is not (Internationalization).
The tokenizer is not a standalone lexer you can reason about in isolation: the tree builder switches its state. What <a> means to the tokenizer depends on whether the tree builder is currently inside SVG (Tree Construction).

Intent→Event→State→UI Logic→DOM Work→Network→Layout / Paint→Pixels→Feedback

What is actually happening

In the browser, not in the framework.

Decode. A byte stream decoder turns bytes into a character stream. The encoding comes from the Content-Type header, then a byte order mark, then a prescan of the first kilobyte of the document for a <meta charset>. If a later <meta charset> contradicts what was assumed, the parser restarts from the beginning with the new encoding (The Head: Metadata That Changes Rendering).
Tokenize. A state machine consumes characters and emits tokens: DOCTYPE, start tag, end tag, comment, character, end-of-file. Start tag tokens carry a name, an attribute list and a self-closing flag.
Switch modes. The tokenizer has several content models. *Data* is normal markup. *RCDATA* (title, textarea) treats tags as text but decodes character references. *RAWTEXT* (style, and historically others) treats everything as text. *Script data* is RAWTEXT with extra escaping states inherited from the era of <!-- inside scripts.
Decode references. &, — and the named-reference table are resolved by the tokenizer, in the states where that is allowed. The list is long, legacy-laden and specified exactly, which is why &notit; is a real trap.
Recover. Every malformed construct has a defined outcome: an unquoted attribute value ends at whitespace, a stray < in text is a character, an unterminated tag at end of file is dropped. The spec calls these parse errors and then says what to do anyway.
Hand off. Tokens go to the tree construction stage one at a time, and that stage can push the tokenizer into a new state — most visibly when a <script> start tag switches it to script data.

What this makes the browser do

And which of it is avoidable.

Decoding bytes to characters, once — or twice, if a late <meta charset> forces a restart. The restart throws away tokenizing and tree building already done.
Running the state machine over every character in the document. This is fast, tuned, mostly off the critical path of your own code, and it is not where slow pages come from.
Resolving character references against the named-reference table, which is why & in attribute values is more work than a plain character.
Repeatedly re-entering the tokenizer as chunks arrive, because the parser is incremental and must be able to stop mid-token and resume (Streaming HTML).
Avoidable: the re-decode. Send a correct charset in the Content-Type header and the prescan never has to guess.

Bytes do not arrive as characters

Before any tokenizing happens, something has to decide what a byte means. HTTP can say it in the Content-Type header, a byte order mark can say it, and failing both, the browser prescans the first part of the document looking for a <meta charset> declaration. Only then does decoding begin.

The consequence of getting this wrong is not a subtle one. Every multi-byte character in the document decodes to the wrong thing, and because the mistake is uniform it looks like a font or a database problem rather than a parsing one. The second consequence is worse: if a <meta charset> turns up later than the prescan window and contradicts the assumption, the parser starts over from byte zero.

From response bytes to a token stream

1
Determine encoding
HTTP Content-Type charset, then a byte order mark, then a prescan of the first part of the document for <meta charset>, then a locale-influenced default.
fails by No declaration anywhere: the browser guesses, and the guess depends on the browser and the user's locale — so the same page can be readable for one user and garbled for another.
2
Decode to characters
Applies the chosen encoding to produce a stream of code points, with defined replacement behaviour for invalid sequences.
fails by A late, contradicting <meta charset> forces a full restart, discarding the tokenizing and tree building already done.
3
Tokenize
Runs the state machine, emitting DOCTYPE, start tag, end tag, comment, character and EOF tokens.
fails by Never fatally. Malformed input takes a defined recovery path and still emits tokens.
4
Resolve character references
Turns &, — and named references into characters, in the states where references are allowed.
fails by Legacy rules bite: an unterminated reference like &notit; resolves the &not prefix in some contexts, so text renders with a stray negation sign.
5
Emit to the tree builder
Delivers one token at a time to tree construction, which may push the tokenizer into a different content model.
fails by Not really a failure, but the source of the surprise: the same characters tokenize differently depending on what element the tree builder is currently inside.

Only the first two steps can produce a page that is wrong everywhere at once. The rest fail locally.

Four content models, and why `</script>` is special

GENERALThe content models and the </script> rule are in the HTML Standard and behave identically in every mainstream engine. What differs is tooling: some bundlers and template engines escape this for you automatically and some do not, so the same source can be safe in one build pipeline and broken in another.

The tokenizer does not treat < the same way everywhere. In the data state — ordinary markup — a < starts a tag. Inside <script> and <style> it does not: those elements have a raw content model in which the only thing that ends the element is a matching end tag. That is the entire reason a JavaScript string containing the six characters </script> terminates your script.

Between those two extremes sits RCDATA, used by <title> and <textarea>, where tags are text but character references are still decoded. This is why the default value of a textarea can contain & and show &, but cannot contain a <b> that goes bold. Knowing which model an element uses tells you exactly which escaping the content needs.

The escape that is a tokenizer rule, not a JavaScript rule

1<!-- Broken: the tokenizer ends the script inside the string literal. -->
2<script>
3  const closer = "</script>";
4  render(closer);
5</script>
6 
7<!-- Fixed: the backslash is meaningless to JavaScript inside a string,
8     but it stops the tokenizer matching the end tag. -->
9<script>
10  const closer = "<\/script>";
11  render(closer);
12</script>
13 
14<!-- Same problem, more common shape: server-injected JSON. -->
15<script id="state" type="application/json">
16  {"bio": "I write about the <\/script> tag"}
17</script>

The second and third forms are valid JavaScript and valid JSON respectively — \/ is just /. The escape exists purely so the HTML tokenizer does not see a matching end tag. Serialising state into a page is the usual way this reaches production (Hydration).

Input:   <p class="lead">Tea &amp; cake</p>

  data state          "<"        -> tag open state
  tag open state      "p"        -> tag name state          [start tag: p]
  tag name state      " "        -> before attribute name
  attribute name      "class"    -> after attribute name
  before value        '"'        -> attribute value (double-quoted)
  attribute value     "lead"     -> after attribute value   [attr class="lead"]
  after value         ">"        -> data state              EMIT <p class="lead">
  data state          "Tea "                                EMIT characters
  data state          "&amp;"    -> character reference     EMIT "&"
  data state          " cake"                               EMIT characters
  data state          "</"       -> end tag open            EMIT </p>

Same six characters, four content models:

  <p>a </script> b</p>              -> text "a </script> b"     (data: "<" starts a tag, "</s" is a bogus end tag -> recovered as text)
  <title>a </script> b</title>      -> text "a </script> b"     (RCDATA: only </title> ends it)
  <style>a </script> b</style>      -> text "a </script> b"     (RAWTEXT: only </style> ends it)
  <script>var s = "</script>";      -> SCRIPT ENDS HERE         (script data: </script> always wins)
                       ";  ... "    -> becomes page text

There is no such thing as a syntax error

The single most useful fact about this parser is that it always succeeds. The specification enumerates parse errors, and then specifies precisely what to do after each one. The browser will not stop, will not warn the user, and will not produce anything you can catch. Every string of bytes has a document.

That is a deliberate and, on balance, correct decision for a platform where most documents are generated by software of unknown quality. It also means "it renders" carries no information about whether the markup says what you meant. The rows below are the recoveries that most often produce a page that looks broken for reasons nobody can find in the source.

Malformed input and what the tokenizer actually does

Trigger	Symptom	Cause	Response
Unterminated attribute quote: `<a href="/x class="btn">`	The rest of the document disappears below that point	The tokenizer stays in the attribute-value state, consuming everything up to the next `"` as part of `href`	Look at the Elements panel: the giant attribute value is visible there and invisible in the source. Escape quotes in interpolated values (Cross-Site Scripting).
`</script>` inside a string or JSON blob	Script ends early; the remainder of the code appears as text on the page	Script data state ends at any matching end tag, regardless of JavaScript syntax	Escape as `<\/script>` at serialisation time, not by hand.
Self-closing syntax on a normal element: `<div />`	Following siblings become children of the div	The self-closing flag is ignored for non-void HTML elements; the div is simply open	Write the end tag. JSX is not HTML; the file extension does not change the parser.
Unquoted attribute value with a space: `<img alt=Two words>`	`alt` is "Two"; a spurious `words` attribute appears	Unquoted values end at the first whitespace, and the remainder tokenizes as another attribute name	Quote every attribute value. This one silently degrades accessibility (Semantics Are Behaviour).
A stray `<` in body text: `if (a < b)` outside a script	Renders as text — or eats the following word	`<` followed by a letter starts a tag; followed by anything else it is a character token	Escape as `<` in prose. This is why templating engines escape by default.
A late `<meta charset>` after a large head	Non-ASCII text is garbled, then the page appears to reload itself	The prescan window was exceeded, so the parser restarts with the corrected encoding	Send `charset` in the `Content-Type` header and put the meta first in `<head>` (The Head: Metadata That Changes Rendering).

How to build it

Most important first.

Declare UTF-8 in the HTTP response header, and also as the first thing in <head>. Header first, meta as a fallback for when the document is loaded from a file or a cache that lost the header.
Never build HTML by string concatenation with untrusted values. The tokenizer is doing exactly what it was told; the vulnerability is that you told it something an attacker wrote (Cross-Site Scripting, Sanitization and Trusted HTML).
Inside inline <script>, escape any literal </script> sequence — commonly as <\/script>. This is a tokenizer rule, not a JavaScript rule, and no amount of correct JavaScript avoids it.
Quote attribute values. Unquoted values are legal and the recovery is specified, but the boundary rules differ from what most people assume and the failure is silent.
Let the parser be the parser. Regular expressions over HTML fail on exactly the constructs the tokenizer has explicit states for; if you need to inspect markup, parse it (The DOM Is Not Your HTML).

Keyboard, focus, semantics, announcement

A required field on every lesson in this domain, not a section added when there is room.

Everything assistive technology sees is derived from the tree this stage feeds. A mis-tokenized attribute means a lost alt, aria-label or lang, and the accessibility tree simply has no name for that node (The Accessibility Tree).
lang on the root element tells the browser and the screen reader which language to pronounce. It is an attribute the tokenizer must survive to produce, which is one more reason unbalanced quotes are an accessibility bug and not only a rendering one.
Character references are how  ,  and typographic punctuation reach the document. A screen reader announces the decoded character, so a run of   used for visual spacing is announced, or produces unnatural pauses, where CSS margin would be silent.
A page that fails to render below a tokenizer accident is not degraded for assistive technology — it is absent. There is no partial reading of content that never became nodes.

What can go wrong

Failure modes

Encoding declared late or not at all: the prescan guesses, the guess is wrong, and every accented character in the page is mojibake. Worse, a late correction restarts parsing and discards work.
A </script> inside a JavaScript string or a JSON blob embedded in the page: the script element ends there, the remaining JavaScript is rendered as text, and the console error points at a line that looks fine.
Server-side templating that emits an unbalanced quote in an attribute: the tokenizer swallows the rest of the document into that attribute value, and the page goes blank below the error with no message.
Assuming a self-closing slash works. <div /> is a non-void element with a self-closing flag that HTML ignores, so the div stays open and everything after it becomes its child. JSX habits leak into .html files this way.
The mitigation failing: an HTML escaper that handles <, > and & but not quotes, applied to a value that lands inside an unquoted attribute. The escaping ran; the injection worked anyway.

Security

The tokenizer is the sink at the end of most cross-site scripting. Injected text that reaches the data state as <img src=x onerror=...> becomes a real element with a real event handler, and the browser has no way to know it was not intended (Cross-Site Scripting).
Context decides the escape. A value inside an attribute, inside a <script>, inside a URL and inside a comment need four different escapes, because the tokenizer is in four different states. One escapeHtml helper applied everywhere is a false sense of safety.
The browser enforces nothing here on your behalf. A Content-Security-Policy can stop the injected script from *executing*, which is why CSP is a second line rather than the first (Content Security Policy).
Trusted Types, where available, move the check to the sink itself — assigning a plain string to innerHTML throws instead of parsing. Support is not universal; treat it as defence in depth rather than the plan (Sanitization and Trusted HTML).

Misreads

"Invalid HTML breaks the page." Invalid HTML produces a defined, often surprising tree. The break is that the tree is not the one you meant.
"The tokenizer is just a lexer, so I can model it as a regular language." The tree builder changes its state, so it is not context-free in the usual sense and definitely not regular. This is exactly where HTML departs from the textbook front end (Tree Construction).
"XHTML was stricter, so it parsed better." XHTML served as XML has fatal errors: a single malformed byte yields a yellow screen instead of a page. The web chose recovery deliberately.
"Minifying HTML by removing optional tags is dangerous." It is specified — the recovery rules that reinsert them are the same everywhere. It is unreadable, which is a different objection (Minification Is Not Compression).

Measuring it, and what changes in the field

How you would see this

View the served bytes, not the rendered DOM. The Network panel response tab shows what the parser was given; the Elements panel shows what it built. When the two disagree, the recovery rules are the explanation (A Mental Model of the Devtools).
The console reports mismatched-encoding warnings and some malformed-markup cases in most browsers, but the reporting is inconsistent — treat its absence as no evidence.
An HTML validator tells you where the parse errors are. It does not tell you the page is broken; it tells you that you are relying on recovery rules you did not choose.
In the Performance panel, HTML parsing appears as its own work on the main thread, interleaved with script. If it is a visible share of load time, the document is unusually large (Reading a Network Waterfall).

Slow device, slow network, large data, old tab

On a slow network, the document arrives in many small chunks and the tokenizer is entered and re-entered far more often. Behaviour does not change; the interleaving with script and rendering does (Streaming HTML).
On a very large document — a generated report, a table with tens of thousands of rows — tokenizing and tree building are genuinely a main-thread cost, and the browser will interrupt them to paint what it has.
On a slow device the absolute cost rises with everything else, but the parser is rarely the dominant term. JavaScript compile and execute usually are (The Real Cost of JavaScript).
When markup is generated by a template engine that escapes for a different language than the surrounding context, the failures are input-dependent: they appear only for the user whose name contains a quote.

What this costs

Relying on error recovery costs nothing today and everything the day you move markup between contexts. innerHTML on a <div> and a document parse of the same string do not build the same tree (Tree Construction).
Escaping correctly per context is more work than a single helper, and it is the only approach that survives a value moving from body text into an attribute.
Declaring encoding in two places duplicates a fact. It is worth it: the two places fail under different circumstances.

Where this applies

Frontend advice ages badly and fragments across engines. These labels say what each claim is specific to, and where a different browser, device or framework would differ.

GENERALThe tokenizer states, the content models and the error-recovery behaviour are specified byte for byte in the HTML Standard, so Blink, Gecko and WebKit agree here in a way they do not agree about layout or devtools. Where they differ is in reporting: what appears as a console warning is entirely a browser choice.
SIMPLIFIEDThe real state machine has dozens of states, including a set of script-data escaped states that exist only to handle <!-- inside scripts. This lesson teaches four content models, which predicts the behaviour you will meet; it will not let you reimplement the parser.
PLATFORM-SPECIFICDocuments served as application/xhtml+xml use an XML parser instead, where a well-formedness error is fatal and produces a parse-error document. That path exists but is rare on the open web; everything else in this lesson describes the HTML parser only.

Where the depth lives

This domain teaches the browser-side mechanism and hands the rest off.

Backendparsing-http parse-dont-validate

Securityxss input-validation content-security-policy

DSAstring stack

Domains that do not exist yet

— Compilers & Programming Languages — lexical analysis, finite automata and the classical separation of scanner from parser. HTML is the famous counterexample: the tree builder reaches back and changes the tokenizer's state, so the two stages cannot be specified independently.
— Compilers & Programming Languages — error recovery strategies. Most language front ends recover in order to report more than one diagnostic; HTML recovers in order to never report any.

OS & Networkingeverything-is-io