WIP: Update to mistune v2 - #57
Open
CrossNox wants to merge 110 commits into
Open
Conversation
|
What is the state of this? |
|
@CrossNox any update on this ? |
|
Further updates here #66 Unfortunately I'm going to have to release our version to pypi as we need it to build our project for an urgent release (we were using poetry git= requirements before) but doesn't work on pypi release. Happy to take it down if we integrate the changes and deploy a newer version of m2r2. Happy to become an active mantainer too - as you like |
is_sphinx is an internal renderer detail for Sphinx integration. The Sphinx codepath uses M2R2.from_sphinx_config() directly, never convert(). Updated test to use M2R2 class directly.
The config mapping duplicated information already expressed by the M2R2 constructor signature. Callers now construct M2R2 directly with explicit config attribute access. setup() inlines add_config_value calls.
is_sphinx was introduced during the migration to switch between :: and .. code-block:: for unlabeled code blocks. The original m2r2 always used .. code-block::, and :: is standard RST anyway (not Sphinx-specific). Removed from M2R2, RestRenderer, and all callers.
The old codebase supported 'python -m m2r2' but the package restructuring broke it. Add the standard __main__.py entry point and update the CLI test to use sys.executable -m m2r2 instead of the bare 'm2r2' command for portability.
The old extension registered 'no_underscore_emphasis' without the m2r_ prefix. Register both names and migrate the old value via a config-inited handler that emits a DeprecationWarning, so existing conf.py files keep working while nudging users to the new name.
parse_from_file was part of the old public API (importable from m2r2 directly). Re-export it from __init__.py and fix the circular import this introduces by having cli/m2r2.py import from m2r2.m2r2 instead of the package root.
The hasattr(md.inline, '_rules') branch was dead code — the _rules attribute does not exist in mistune v3. Removing from md.inline.rules is sufficient. Also add an iteration limit (10) to the raw-html merge loop to prevent infinite loops if the regex ever produces a fixpoint.
Remove references to nonexistent files (constants.py, typing.py, rst/directives.py, rst/parser.py), add sphinx/directives.py and __main__.py, and list test_sphinx.py in the testing section.
Trim from 324 to 59 lines by removing sphinx-quickstart boilerplate comments, unused LaTeX/man/texinfo sections, dead sys.path.append hack, and broken intersphinx_mapping. Update copyright to CrossNox.
Mirrors the convert() signature for a consistent, self-documenting API instead of blindly forwarding arbitrary keyword arguments.
~~text~~ was passing through unconverted because the strikethrough plugin was never registered. Add the mistune plugin and a RestRenderer method that renders it as :raw-html-m2r:`<del>text</del>`.
Pass instantiated Reader, RstParser, and Writer to Publisher instead of using set_components() with string names — the latter is deprecated and will be removed in docutils 2.0. Also add proper assertions to test_strikethrough now that the plugin is enabled.
Runs the CLI end-to-end via subprocess with --dry-run and verifies the output matches the expected .rst fixture.
mistune v3 combines all inline patterns into one large regex via named
groups and alternation. Using m.group(1) returned None because group 1
referred to a different pattern's capture group. Switch to named groups
((?P<emph_text>...) and (?P<strong_text>...)) and m.group('emph_text')
to always capture the correct match.
Passing plugins=some_list to M2R2() would append 4 items (rst_directives, table, footnotes, strikethrough) to the caller's list, mutating it unexpectedly. Copy with list(plugins) to avoid side effects.
Bug fixes:
- Fix code block info string: split on whitespace to extract language only
(e.g. 'python title=test' -> 'python')
- Fix _raw_html() to escape backticks as ` preventing broken RST roles
- Remove dead paragraph condition: text.strip().startswith('\n\n') was
unreachable; keep only the image_link path check
- Wrap list() item processing in try/finally to always restore state
Dead code removal:
- Remove finalize(): not in mistune v3 RSTRenderer, never called
- Remove table_cell(): never dispatched to (table_head/table_row call
render_children directly)
- Remove render_referrences/inline_images dead code from __call__
Comment improvements:
- Add semantic comment explaining why table_head and table_row are separate
- Update footnote comments: .lower() normalizes mistune v3's uppercasing
- Guard app.connect('config-inited', ...) with try/except KeyError for
Sphinx < 1.8 which doesn't have the config-inited event
- Invert add_source_parser try/except: try modern API first (single arg),
fall back to old 2-arg API for Sphinx < 4.0
- Add --version flag showing package version - Add --use-mermaid flag to render mermaid blocks as directives - Add try/except for FileNotFoundError/OSError in run_m2r2 with stderr output and exit code 1 - Guard interactive input() with sys.stdin.isatty() check to prevent blocking in non-interactive environments (pipes, CI) - Update tests to mock sys.stdin.isatty for overwrite/decline tests - Add test for multiple input files
- docs/conf.py: source_suffix .md -> ['.rst', '.md'], master_doc -> root_doc (Sphinx 4.0+), copyright year 2025 -> 2026, htmlhelp_basename M2Rdoc -> M2R2doc - pyproject.toml: remove follow_imports='skip' that neutered mypy - README.md: fix import path from m2r2.cli to m2r2
- Add 'check' job that runs lint and tests before deployment - Both untagged_deploy and tagged_deploy now need: [check] - Fix branches syntax: 'master' -> [master] (YAML list required)
- no_underscore_emphasis: asterisk emphasis/strong, underscore passthrough, mixed emphasis in sentence (regression for #1) - Code block info string with extra metadata (regression for #6) - Inline HTML with backtick escaping (regression for #23) - image_title: add actual assertions for image directive output - Complex text: add assertions for code blocks, headings, thematic break - Heading levels 2-6: verify all underline characters (-, ^, ~, ", #) - Footnote mixed-case: verify key normalization to lowercase
… link fallback - Set tok['prev'] = prev_tok in iter_tokens to maintain the parent RSTRenderer's contract (parent's block_quote accesses token['prev']) - Remove dead state.env['inline_images'] initialization from __call__ (was needed by removed render_referrences code, never read anywhere) - Fix relative links with path+fragment (e.g. page.md#section): fall back to regular link instead of silently discarding the anchor, since :doc: cannot target specific anchors - Fix rest_code_block plugin token to use 'raw' key instead of 'text' to skip unnecessary inline parsing on empty content
… config precedence CLI: - Validate all input files exist before processing any, rather than failing on first error and skipping remaining valid files - Send skip/warning messages to stderr instead of stdout to avoid corrupting piped output (e.g. with --dry-run) Sphinx: - Fix deprecated config migration: if the user explicitly sets the new m2r_no_underscore_emphasis config, it takes precedence over the deprecated no_underscore_emphasis value
CI security: - Add permissions: contents: read to all workflows (least privilege) - Add permissions: contents: write to tagged_deploy job (needs gh-pages push) - Pin ad-m/github-push-action to commit SHA (57116acb) - Pin pypa/gh-action-pypi-publish to release/v1.13 Config: - Fix mypy: replace global ignore_missing_imports with per-module override for mistune.* only, so real import errors are caught - Add py.typed marker file for PEP 561 downstream type checking - Fix CHANGES.md: docutils version 0.21.2+ -> 0.19+ to match pyproject.toml - Fix CONTRIBUTING.md: update exports list and cli description
CLI tests:
- Rewrite file-writing tests to use tempdir instead of mutating the
shared test.rst fixture (prevents corruption on crash/kill)
- Use StringIO for stdout/stderr capture instead of mock.patch('builtins.print')
- Add tests: --version flag, --parse-relative-links, --use-mermaid,
missing files validation (all reported before exit), non-interactive skip
Renderer tests:
- Strengthen test_strikethrough: assert full role content
- Update test_relative_link_with_anchor for new fallback behavior
- Add TestEdgeCases: empty input, whitespace-only, multiple newlines
- Add TestInstanceReuse: list/table/footnote state isolation across calls
Fixtures:
- Expand test.md/test.rst pair to cover lists (nested, ordered),
inline code, code blocks, images, block quotes, tables, horizontal
rules, and footnotes
- Escape link/title attrs with html.escape() to prevent XSS injection - Fix _RAW_HTML_MERGE_PATTERN: allow colons but exclude newlines to prevent cross-line merges that consume unrelated RST content - Simplify post_process newline normalization to lstrip + prepend - Add PackageNotFoundError fallback for __version__ - Remove unused tight param from _render_list_item and bullet key from list tokens in plugins.py - Fix paragraph() docstring accuracy - Improve test assertions: backtick escape, codespan role, version regex - Pin pypa/gh-action-pypi-publish to SHA, add concurrency group, clear gh-pages before deploy, add -j auto to Sphinx CI builds - Standardize CHANGES.md heading levels - Rewrite CONTRIBUTING.md to reference pre-commit hooks
… :doc: The migration changed relative links with anchors (e.g. [text](page.md#anchor)) to fall back to plain hyperlinks, which don't resolve in Sphinx builds. Restore the original m2r behavior: drop the fragment and emit :doc: instead, since the :doc: directive does not support anchors.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.