fix(html2md): retain complex tables and referenced fragments

moli-html2md/README.md

@@ -68,20 +68,27 @@ context, keeping links on one line. Unicode spaces and literal entity text are
 retained; URLs use destination escaping. GFM supplies
 missing trailing cells in short rows; the converter does not allocate a padded grid.
 
-Tables with `role="presentation"` or `role="none"` retain ordinary block expansion,
-in DOM order with blank lines between cells, even when structurally simple. Role
-matching ignores ASCII case and surrounding HTML whitespace. The same fallback
-applies to complex tables: non-unit spans, multiple `thead` rows, rows wider than
-an explicit header, or cells containing nested tables, lists, headings, quotes
-or preformatted blocks. Inner simple tables can still produce GFM when their
-containing layout table is expanded. Other roles, borders and CSS do not select
-another conversion path. Complex layouts remain readable blocks instead of raw
-HTML tables.
+Tables with `role="presentation"` or `role="none"` retain ordinary block
+expansion, in DOM order with blank lines between cells, even when structurally
+simple. Role matching ignores ASCII case and surrounding HTML whitespace. Nested
+layout wrappers expand into readable blocks. Tables with header cells, captions,
+explicit data roles, header relationships or real cell spans retain their
+authored structure. These tables use content-only HTML for non-unit spans,
+multiple `thead` rows, rows wider than an explicit header, or cells containing
+nested tables, lists, headings, quotes or preformatted blocks. This retains
+header, cell and nested-table associations that Markdown pipe tables cannot
+express. Inner simple tables can still produce GFM when a presentational or
+layout-only containing table is expanded. Scripts, styles and event handlers are
+omitted from HTML fallback output.
 
 Supported output includes headings, paragraphs, emphasis, strikethrough, links,
-images, lists, blockquotes, hard breaks, media source links, readable form values,
-separated form choices, and fenced code. Inline HTML is used when
+images, lists, blockquotes, hard breaks, media source links, separated form choices,
+readable form values, and fenced code. Inline HTML is used when
 Markdown delimiters cannot express an emphasis boundary.
+Superscripts and subscripts retain their HTML markers. Complete author-supplied
+TeX spans retain their commands and indices. Native MathML retains mathematical
+markup and presentation attributes, with alternate annotations omitted, so
+fractions, roots, matrices and powers do not collapse into ambiguous plain text.
 Script/style/head/noscript/template subtrees are omitted.
 
 `max_depth` counts the supplied root as depth zero; nodes at or beyond the limit
@@ -94,25 +101,27 @@ spaces and blank lines. Empty blocks still separate surrounding text.
 
 This is a structural content dump. It omits non-rendered serialized state blobs
 but retains content in collapsed panels and inactive tabs. It does not evaluate
-stylesheets, layout, or JavaScript, and cannot preserve all HTML presentation (for example, arbitrary
-ordered-list numbering or table spanning). DOM snapshots used by Moli's optional
+stylesheets, layout, or JavaScript. Callers may supply computed block boundaries
+through the optional DOM query. The converter
+cannot preserve all HTML presentation (for example, arbitrary ordered-list numbering).
+DOM snapshots used by Moli's optional
 strip/base/frame transformations remain the responsibility of the caller.
 
 ## Verification
 
 `cargo test -p moli-html2md` covers an independent arena DOM, immutability,
 whitespace, escaping, code, tables, depth limits, and 20,000 nested nodes on a
 64 KiB thread stack. The [Turndown differential suite](tests/turndown/README.md)
-pins 147 upstream fixtures and 50 additional cases. It compares rendered meaning,
-checks DOM immutability, and explicitly asserts 16 documented differences that
-preserve Moli's handling of literal text, code, emphasis, and list boundaries.
-Targeted regressions also cover empty elements, links, attributes in headings and
-tables, code whitespace and language names, list numbering, and loose lists.
-An operation-count regression checks nested table fallback at 64, 128 and 256
-levels with 256 text bytes per level, both with and without headers. It bounds
-bytes copied during fragment materialization by output size; completed blocks
-move into parent buffers without copying their text. A repeated flatten/copy
-regression fails without timing thresholds.
+pins 147 upstream fixtures and 50 additional cases. It compares rendered
+meaning, checks DOM immutability, and explicitly asserts 16 documented
+differences that preserve Moli's handling of literal text, code, emphasis, and
+list boundaries. Targeted regressions also cover empty elements, links,
+attributes in headings and tables, code whitespace and language names, list
+numbering, and loose lists. An operation-count regression checks nested table
+fallback at 64, 128 and 256 levels with 256 text bytes per level, both with and
+without headers. It bounds bytes copied during fragment materialization by
+output size; completed blocks move into parent buffers without copying their
+text. A repeated flatten/copy regression fails without timing thresholds.
 Dropping 20,000 nested fragments is also tested on a 64 KiB thread stack.
 Emphasis tests cover adjacent links that can use Markdown delimiters and
 intraword link boundaries that still require inline HTML. Combining marks and
@@ -140,9 +149,7 @@ JavaScript and inline HTML document URLs are excluded.
 
 SVG title children are tooltip metadata, not body text. They supply a name only
 when an SVG has no emitted content; title elements outside SVG are omitted.
-Author-supplied TeX keeps its commands across transparent inline wrappers.
-MathML and superscript/subscript retain their structure; alternate visual copies
-do not duplicate an equation. Code omits embedded copy controls and scripts.
+Code omits embedded copy controls and scripts.
 
 Backslash-delimited inline TeX becomes inline dollar math. Invisible separators
 keep adjacent formulas distinct across empty wrappers. They also separate a
@@ -155,3 +162,10 @@ HTML emphasis when nested Markdown markers would pair across those tags.
 MathML deduplication retains links, formatting and surrounding whitespace,
 including Unicode spaces; filtered containers keep their grouping, and `maction`
 keeps its first element.
+
+Linked preformatted source retains HTML when needed to keep link and line-anchor
+positions. Its language marker and physical newlines are preserved; a standalone
+`pre` HTML block accepts blank lines. Complex HTML tables use physical row and
+cell boundaries without blank lines. Plain heading wrappers remain Markdown
+headings. Captured headings, list items and quotations contain their own fragment
+targets rather than appending them to the preceding block.

moli-html2md/src/anchors.rs

@@ -0,0 +1,199 @@
+use std::collections::HashSet;
+
+use crate::{Dom, NodeKind};
+
+pub(crate) struct Targets {
+    referenced: HashSet<String>,
+    header_ids: HashSet<String>,
+    ids: HashSet<String>,
+}
+
+impl Targets {
+    pub(crate) fn is_empty(&self) -> bool {
+        self.referenced.is_empty() && self.header_ids.is_empty()
+    }
+
+    pub(crate) fn targets<'a, D: Dom + ?Sized>(&self, dom: &'a D, node: D::NodeId) -> Vec<&'a str> {
+        let mut targets = Vec::new();
+        if let Some(id) = dom.attribute(node, "id").filter(|id| {
+            self.referenced.contains(*id)
+                || (dom.node_kind(node) == NodeKind::Element("th") && self.header_ids.contains(*id))
+        }) {
+            targets.push(id);
+        }
+        if dom.node_kind(node) == NodeKind::Element("a")
+            && let Some(name) = dom
+                .attribute(node, "name")
+                .filter(|name| self.referenced.contains(*name) && !self.ids.contains(*name))
+            && !targets.contains(&name)
+        {
+            targets.push(name);
+        }
+        targets
+    }
+}
+
+pub(crate) fn referenced<D: Dom + ?Sized>(dom: &D, root: D::NodeId, limit: usize) -> Targets {
+    let mut available = HashSet::new();
+    let mut element_ids = HashSet::new();
+    let mut header_ids = HashSet::new();
+    let mut fragments = Vec::new();
+    let mut headers = HashSet::new();
+    let fragment_links = dom.may_have_fragment_links();
+    walk(dom, root, limit, |node| {
+        if fragment_links
+            && let Some(fragment) = dom
+                .attribute(node, "href")
+                .and_then(|href| href.strip_prefix('#'))
+                .filter(|fragment| !fragment.is_empty())
+        {
+            fragments.push(fragment);
+        }
+        if matches!(dom.node_kind(node), NodeKind::Element("td" | "th"))
+            && let Some(value) = dom.attribute(node, "headers")
+        {
+            headers.extend(
+                value
+                    .split(crate::writer::is_space)
+                    .filter(|id| !id.is_empty()),
+            );
+        }
+        false
+    });
+    if fragments.is_empty() && headers.is_empty() {
+        return Targets {
+            referenced: available,
+            header_ids,
+            ids: element_ids,
+        };
+    }
+    walk(dom, root, limit, |node| {
+        for attribute in ["id", "name"] {
+            if attribute == "name" && dom.node_kind(node) != NodeKind::Element("a") {
+                continue;
+            }
+            if let Some(value) = dom.attribute(node, attribute) {
+                available.insert(value.to_owned());
+                if attribute == "id" {
+                    element_ids.insert(value.to_owned());
+                    if dom.node_kind(node) == NodeKind::Element("th") && headers.contains(value) {
+                        header_ids.insert(value.to_owned());
+                    }
+                }
+            }
+        }
+        false
+    });
+    let mut referenced = HashSet::new();
+    for fragment in fragments {
+        if available.contains(fragment) {
+            referenced.insert(fragment.to_owned());
+        } else {
+            let decoded = decode_fragment(fragment);
+            if available.contains(&decoded) {
+                referenced.insert(decoded);
+            }
+        }
+    }
+    Targets {
+        referenced,
+        header_ids,
+        ids: element_ids,
+    }
+}
+
+pub(crate) fn contains<D: Dom + ?Sized>(
+    dom: &D,
+    root: D::NodeId,
+    limit: usize,
+    ids: &Targets,
+) -> bool {
+    walk(dom, root, limit, |node| !ids.targets(dom, node).is_empty())
+}
+
+pub(crate) fn within<D: Dom + ?Sized>(
+    dom: &D,
+    root: D::NodeId,
+    limit: usize,
+    ids: &Targets,
+) -> Vec<String> {
+    let mut found = Vec::new();
+    walk(dom, root, limit, |node| {
+        for id in ids.targets(dom, node) {
+            // Every caller deduplicates with its shared emitted-target set.
+            found.push(id.to_owned());
+        }
+        false
+    });
+    found
+}
+
+fn walk<D: Dom + ?Sized>(
+    dom: &D,
+    root: D::NodeId,
+    limit: usize,
+    mut found: impl FnMut(D::NodeId) -> bool,
+) -> bool {
+    let mut stack = vec![(Some(root), 0, false)];
+    while let Some((node, depth, siblings)) = stack.pop() {
+        let Some(node) = node else {
+            continue;
+        };
+        if siblings {
+            stack.push((dom.next_sibling(node), depth, true));
+        }
+        if depth >= limit {
+            continue;
+        }
+        if matches!(
+            dom.node_kind(node),
+            NodeKind::Other
+                | NodeKind::Element("head" | "script" | "style" | "noscript" | "template")
+        ) {
+            continue;
+        }
+        if found(node) {
+            return true;
+        }
+        stack.push((dom.first_child(node), depth + 1, true));
+    }
+    false
+}
+
+pub(crate) fn decode_fragment(fragment: &str) -> String {
+    let bytes = fragment.as_bytes();
+    let mut decoded = Vec::with_capacity(bytes.len());
+    let mut offset = 0;
+    while offset < bytes.len() {
+        if bytes[offset] == b'%' && offset + 2 < bytes.len() {
+            let hex = |byte: u8| (byte as char).to_digit(16);
+            if let (Some(high), Some(low)) = (hex(bytes[offset + 1]), hex(bytes[offset + 2])) {
+                decoded.push((high * 16 + low) as u8);
+                offset += 3;
+                continue;
+            }
+        }
+        decoded.push(bytes[offset]);
+        offset += 1;
+    }
+    String::from_utf8(decoded)
+        .unwrap_or_else(|error| String::from_utf8_lossy(error.as_bytes()).into_owned())
+}
+
+pub(crate) fn markup(id: &str) -> String {
+    let mut result = String::from("<span id=\"");
+    for character in id.chars() {
+        match character {
+            '&' => result.push_str("&amp;"),
+            '"' => result.push_str("&quot;"),
+            '<' => result.push_str("&lt;"),
+            '>' => result.push_str("&gt;"),
+            '\n' => result.push_str("&#10;"),
+            '\r' => result.push_str("&#13;"),
+            '|' => result.push_str("&#124;"),
+            character => result.push(character),
+        }
+    }
+    result.push_str("\"></span>");
+    result
+}

moli-html2md/src/content.rs

@@ -1,5 +1,34 @@
 use crate::{Dom, NodeKind};
 
+/// Readable content is independent of serialized tags and navigation targets.
+#[derive(Default)]
+pub(crate) struct Content {
+    revision: usize,
+}
+
+pub(crate) struct Html {
+    pub(crate) markup: String,
+    pub(crate) has_content: bool,
+}
+
+impl Content {
+    pub(crate) fn mark(&self) -> usize {
+        self.revision
+    }
+
+    pub(crate) fn text(&mut self, text: &str) {
+        self.revision += usize::from(text.chars().any(|ch| !ch.is_whitespace()));
+    }
+
+    pub(crate) fn object(&mut self) {
+        self.revision += 1;
+    }
+
+    pub(crate) fn merge(&mut self, child: &Self) {
+        self.revision += child.revision;
+    }
+}
+
 pub(crate) fn fallback_text<D: Dom + ?Sized>(
     dom: &D,
     node: D::NodeId,
@@ -102,3 +131,36 @@ pub(crate) fn image_label<D: Dom + ?Sized>(dom: &D, node: D::NodeId) -> &str {
         })
         .unwrap_or_default()
 }
+
+/// Interactive descendants of code are UI controls, not source text.
+pub(crate) fn is_code_control<D: Dom + ?Sized>(dom: &D, node: D::NodeId) -> bool {
+    matches!(
+        dom.node_kind(node),
+        NodeKind::Element("button" | "input" | "select" | "textarea")
+    ) || dom
+        .attribute(node, "role")
+        .is_some_and(|role| role.eq_ignore_ascii_case("button"))
+}
+
+/// The same authored/default language policy serves fenced and HTML code blocks.
+pub(crate) fn code_language<'a, D: Dom + ?Sized>(
+    dom: &'a D,
+    node: D::NodeId,
+    default: Option<&'a str>,
+) -> Option<&'a str> {
+    let mut child = dom.first_child(node);
+    while let Some(id) = child {
+        if dom.node_kind(id) == NodeKind::Element("code")
+            && let Some(language) = class_language(dom.attribute(id, "class"))
+        {
+            return Some(language);
+        }
+        child = dom.next_sibling(id);
+    }
+    class_language(dom.attribute(node, "class")).or(default)
+}
+pub(crate) fn class_language(class: Option<&str>) -> Option<&str> {
+    class?
+        .split_ascii_whitespace()
+        .find_map(|part| part.strip_prefix("language-").filter(|s| !s.is_empty()))
+}

moli-html2md/src/copy_tests.rs

@@ -28,7 +28,7 @@ fn nested_table_fallback_copies_text_linearly() {
             assert_eq!(actual.matches(&after).count(), depth);
             assert_eq!(
                 support::rendered_html(&actual).matches("<table>").count(),
-                1
+                if heading.is_empty() { 1 } else { depth }
             );
             // Count actual bytes copied when materializing fragments; completed
             // blocks move into parents without copying their text. This measures