How Markdown Parsers Work: Abstract Syntax Trees (AST), CommonMark, and Unified Remark Pipelines

The Problem: Ambiguity in Original 2004 Markdown#

The original 2004 Markdown implementation was a loose Perl script (`Markdown.pl`) accompanied by an informal syntax description. Because it lacked a formal grammar, different implementations produced wildly divergent HTML outputs for nested lists, emphasis markers, and indented code blocks.

This inconsistency led to the creation of the CommonMark initiative, an unambiguous, mathematically specified standard with hundreds of edge-case test suites.

CommonMark & GitHub Flavored Markdown (GFM) Standardization#

CommonMark provides the rigorous baseline for standard Markdown constructs (headings, paragraphs, blockquotes, lists, links, images, and code blocks).

GitHub Flavored Markdown (GFM) extends CommonMark with crucial developer extensions: pipe-delimited ASCII tables, task list checkboxes (`- [x] Done`), strikethrough (`~~text~~`), autolinks, and fenced code block syntax highlighting annotations.

The Two-Phase Parsing Architecture: Block vs Inline#

Compliant CommonMark parsers process text in two distinct, sequential phases:

• Phase 1: Block Structure Parsing. The parser scans the document line-by-line, constructing the container hierarchy (documents, headings, list items, blockquotes, and code fences).
• Phase 2: Inline Phase Parsing. Once block containers are established, the parser processes the text inside blocks to resolve delimiters into emphasis (*italic*), strong (**bold**), inline code (`code`), and hyperlinks.

Abstract Syntax Trees: MDAST to HAST Pipeline#

Modern JavaScript tools (such as Unified, Remark, and Rehype) represent documents using formal Abstract Syntax Trees:

Markdown String
       │
       ▼ (remark-parse)
Markdown AST (MDAST) ──> Plugins: autolink-headings, gfm-tables, math
       │
       ▼ (remark-rehype)
HTML AST (HAST)      ──> Plugins: rehype-highlight, rehype-slug
       │
       ▼ (rehype-sanitize / rehype-stringify)
Sanitized HTML Output

Security: Sanitizing Raw HTML inside Markdown Content#

Because standard Markdown allows embedded raw HTML, accepting user-submitted Markdown (in forum posts or comments) introduces severe Cross-Site Scripting (XSS) risks if a user injects `<script>` tags or `<img onerror="...">` payloads.

Production pipelines must always pass the generated HTML through a strict DOM sanitizer (such as `DOMPurify` or `rehype-sanitize`) with a strict whitelist of permitted tags and attributes.

In-Browser Markdown Processing Without Server Overhead#

Client-side Markdown formatters and real-time live preview editors run entirely within the browser’s JavaScript engine. Processing AST transforms locally ensures zero latency, offline capability, and absolute privacy for proprietary technical documents.