Add --issue-timeline flag to capture cross-reference events (#168)

The issue events endpoint never returns cross-references, so an issue
mentioned from another issue or pull request leaves no trace in the
backup. GitHub only exposes those through the timeline endpoint. Full
backups miss them as well as incremental ones, and incremental runs
never revisit the issue at all, because a cross-reference does not
change its updated_at.

Adds --issue-timeline, which stores the timeline under a separate
timeline_data key alongside the unmodified event_data. The two
endpoints are complementary rather than nested: the timeline adds
cross-references, commits and reviews, while events carries
referenced entries the timeline drops, so neither replaces the other.

Timeline entries for comments are filtered out, since --issue-comments
already saves the bodies. Everything else is stored verbatim. The flag
is included in --all.

Closes #168

Author: Iamrodos

README.rst

@@ -41,7 +41,8 @@ CLI Help output::
                   [--incremental-by-files]
                   [--starred] [--all-starred] [--starred-skip-size-over MB]
                   [--watched] [--followers] [--following] [--all]
-                  [--issues] [--issue-comments] [--issue-events] [--pulls]
+                  [--issues] [--issue-comments] [--issue-events]
+                  [--issue-timeline] [--pulls]
                   [--pull-comments] [--pull-reviews] [--pull-commits]
                   [--pull-details]
                   [--labels] [--hooks] [--milestones] [--security-advisories]
@@ -96,6 +97,8 @@ CLI Help output::
       --issues              include issues in backup
       --issue-comments      include issue comments in backup
       --issue-events        include issue events in backup
+      --issue-timeline      include issue timeline in backup (cross-references,
+                            commits and reviews; complements --issue-events)
       --pulls               include pull requests in backup
       --pull-comments       include pull request review comments in backup
       --pull-reviews        include pull request reviews in backup
@@ -347,6 +350,27 @@ For finer control, avoid using ``--assets`` with starred repos, or use ``--skip-
 
 Alternatively, consider just storing links to starred repos in JSON format with ``--starred``.
 
+
+About issue events and the issue timeline
+-----------------------------------------
+
+Use ``--issue-events`` and ``--issue-timeline`` with ``--issues``; on their own they back up nothing.
+
+``--issue-events`` backs up GitHub's issue events endpoint under each issue's ``event_data`` key. That endpoint returns a restricted set of event types and **never** includes cross-references, so an issue that was mentioned from another issue or pull request shows no sign of it. GitHub only exposes those through the timeline.
+
+``--issue-timeline`` backs up the timeline endpoint under a separate ``timeline_data`` key. The two flags are independent: ``event_data`` keeps its existing contents and shape whether or not the timeline is enabled, so nothing that already reads it is affected.
+
+Use both flags together for a complete picture. Neither endpoint is a superset of the other, so they are complementary rather than alternatives. Only the timeline reports ``cross-referenced``, ``committed`` and ``reviewed`` entries. Only the events endpoint reliably reports ``referenced`` entries, where a commit elsewhere in the repository mentions the issue; the timeline omits many of these, substituting the issue's own ``committed`` entries instead.
+
+Note that timeline entries are not all the same shape as events. ``cross-referenced`` and ``committed`` entries have no ``id`` field, so anything consuming ``timeline_data`` should not assume one is present. ``commented`` entries are excluded from the backup because they embed the full comment body, which ``--issue-comments`` already saves.
+
+``--issue-timeline`` is included in ``--all``. On issues with many cross-references it can be noticeably larger and slower than ``--issue-events`` alone, because each ``cross-referenced`` entry embeds the referencing issue in full. On a heavily cross-referenced issue this can mean several times the API requests and many times the JSON on disk. Ordinary repositories see little difference.
+
+If you enable ``--issue-timeline`` on an existing incremental backup, run once without ``--incremental`` to backfill it. There is no automatic backfill, so an incremental run only revisits recently updated issues and every older issue keeps a JSON file with no ``timeline_data`` until something touches it. The same applies to ``--incremental-by-files``.
+
+Incremental backups select issues by ``updated_at``, and a cross-reference does not change the referenced issue's ``updated_at``. An issue whose only new activity is being referenced elsewhere is therefore not revisited on that run. It corrects itself the next time the issue is touched, since each issue's timeline is fetched in full, so the only lasting gap is an issue that is referenced and then never touched again. Run occasionally without ``--incremental`` if that matters to you.
+
+
 About pull request reviews
 --------------------------
 
@@ -450,14 +474,14 @@ Quietly and incrementally backup useful Github user data (public and private rep
     FINE_ACCESS_TOKEN=SOME-GITHUB-TOKEN
     GH_USER=YOUR-GITHUB-USER
 
-    github-backup -f $FINE_ACCESS_TOKEN --prefer-ssh -o ~/github-backup/ -l error -P -i --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --pulls --pull-comments --pull-reviews --pull-commits --labels --milestones --security-advisories --discussions --repositories --wikis --releases --assets --attachments --pull-details --gists --starred-gists $GH_USER
+    github-backup -f $FINE_ACCESS_TOKEN --prefer-ssh -o ~/github-backup/ -l error -P -i --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --issue-timeline --pulls --pull-comments --pull-reviews --pull-commits --labels --milestones --security-advisories --discussions --repositories --wikis --releases --assets --attachments --pull-details --gists --starred-gists $GH_USER
 
 Debug an error/block or incomplete backup into a temporary directory. Omit "incremental" to fill a previous incomplete backup. ::
 
     FINE_ACCESS_TOKEN=SOME-GITHUB-TOKEN
     GH_USER=YOUR-GITHUB-USER
 
-    github-backup -f $FINE_ACCESS_TOKEN -o /tmp/github-backup/ -l debug -P --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --pulls --pull-comments --pull-reviews --pull-commits --labels --milestones --discussions --repositories --wikis --releases --assets --pull-details --gists --starred-gists $GH_USER
+    github-backup -f $FINE_ACCESS_TOKEN -o /tmp/github-backup/ -l debug -P --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --issue-timeline --pulls --pull-comments --pull-reviews --pull-commits --labels --milestones --discussions --repositories --wikis --releases --assets --pull-details --gists --starred-gists $GH_USER
 
 Pipe a token from stdin to avoid storing it in environment variables or command history (Unix-like systems only)::
 

github_backup/github_backup.py

@@ -278,6 +278,12 @@ def parse_args(args=None):
         dest="include_issue_events",
         help="include issue events in backup",
     )
+    parser.add_argument(
+        "--issue-timeline",
+        action="store_true",
+        dest="include_issue_timeline",
+        help="include issue timeline in backup (cross-references, commits and reviews; complements --issue-events)",
+    )
     parser.add_argument(
         "--pulls",
         action="store_true",
@@ -2610,6 +2616,7 @@ def backup_issues(args, repo_cwd, repository, repos_template):
     )
     comments_template = _issue_template + "/{0}/comments"
     events_template = _issue_template + "/{0}/events"
+    timeline_template = _issue_template + "/{0}/timeline"
     for number, issue in list(issues.items()):
         issue_file = "{0}/{1}.json".format(issue_cwd, number)
         if args.incremental_by_files and os.path.isfile(issue_file):
@@ -2629,6 +2636,16 @@ def backup_issues(args, repo_cwd, repository, repos_template):
         if args.include_issue_events or args.include_everything:
             template = events_template.format(number)
             issues[number]["event_data"] = retrieve_data(args, template)
+        if args.include_issue_timeline or args.include_everything:
+            template = timeline_template.format(number)
+            # "commented" timeline entries embed the full comment body, which
+            # --issue-comments already backs up; storing them twice invites the
+            # two copies to diverge.
+            issues[number]["timeline_data"] = [
+                item
+                for item in retrieve_data(args, template)
+                if item.get("event") != "commented"
+            ]
         if args.include_attachments:
             download_attachments(
                 args, issue_cwd, issues[number], number, repository, item_type="issue"

tests/test_issue_timeline.py

@@ -0,0 +1,123 @@
+"""Tests for issue timeline backup (GitHub issue #168).
+
+The ``/issues/{number}/events`` endpoint never returns cross-reference events
+("mentioned in #22"); GitHub only exposes those via ``/issues/{number}/timeline``.
+``--issue-timeline`` backs up the timeline alongside the events, so those
+references are captured.
+"""
+
+import json
+
+from github_backup import github_backup
+
+
+def _issue(number=1, **extra):
+    issue = {
+        "number": number,
+        "title": "an issue",
+        "updated_at": "2026-07-01T00:00:00Z",
+    }
+    issue.update(extra)
+    return issue
+
+
+def _fake_retrieve(responses, calls):
+    """Return a retrieve_data stub serving ``responses`` keyed by URL suffix."""
+
+    def _retrieve(args, template, query_args=None, **kwargs):
+        calls.append(template)
+        for suffix, payload in responses.items():
+            if template.endswith(suffix):
+                return payload
+        return []
+
+    return _retrieve
+
+
+def _run(create_args, tmp_path, monkeypatch, responses, **arg_overrides):
+    args = create_args(include_issues=True, since=None, **arg_overrides)
+    calls = []
+    monkeypatch.setattr(
+        github_backup, "retrieve_data", _fake_retrieve(responses, calls)
+    )
+    github_backup.backup_issues(
+        args, str(tmp_path), {"full_name": "owner/repo"}, "https://api.github.com/repos"
+    )
+    issue_file = tmp_path / "issues" / "1.json"
+    assert issue_file.is_file(), "issue was not written to disk"
+    return json.loads(issue_file.read_text()), calls
+
+
+def test_parse_args_issue_timeline_flag():
+    # The create_args fixture sets attributes directly, so only this test
+    # catches the flag being wired to the wrong dest.
+    args = github_backup.parse_args(["--issue-timeline", "testuser"])
+    assert args.include_issue_timeline is True
+
+
+TIMELINE = [
+    {"id": 1, "event": "labeled", "label": {"name": "bug"}},
+    {"event": "cross-referenced", "source": {"type": "issue", "issue": {"number": 22}}},
+    {"id": 2, "event": "commented", "body": "a comment body"},
+    {"event": "committed", "sha": "abc123"},
+]
+
+
+def test_comments_filtered_and_everything_else_stored_verbatim(
+    create_args, tmp_path, monkeypatch
+):
+    saved, calls = _run(
+        create_args,
+        tmp_path,
+        monkeypatch,
+        {"/issues": [_issue()], "/1/timeline": TIMELINE},
+        include_issue_timeline=True,
+    )
+
+    assert any(c.endswith("/1/timeline") for c in calls)
+    # "commented" is dropped because --issue-comments already covers it.
+    # Everything else is stored exactly as GitHub returned it, including the
+    # fat source object on cross-references.
+    assert saved["timeline_data"] == [TIMELINE[0], TIMELINE[1], TIMELINE[3]]
+
+
+def test_timeline_not_fetched_without_flag(create_args, tmp_path, monkeypatch):
+    saved, calls = _run(
+        create_args,
+        tmp_path,
+        monkeypatch,
+        {"/issues": [_issue()], "/1/timeline": TIMELINE},
+    )
+
+    assert not any(c.endswith("/1/timeline") for c in calls)
+    assert "timeline_data" not in saved
+
+
+def test_events_unchanged_by_timeline_flag(create_args, tmp_path, monkeypatch):
+    """--issue-events keeps reading /events, so event_data's shape is untouched."""
+    events = [{"id": 1, "event": "labeled", "label": {"name": "bug"}}]
+    saved, calls = _run(
+        create_args,
+        tmp_path,
+        monkeypatch,
+        {"/issues": [_issue()], "/1/events": events, "/1/timeline": TIMELINE},
+        include_issue_events=True,
+        include_issue_timeline=True,
+    )
+
+    assert any(c.endswith("/1/events") for c in calls)
+    assert saved["event_data"] == events
+    assert len(saved["timeline_data"]) == 3
+
+
+def test_timeline_included_in_all(create_args, tmp_path, monkeypatch):
+    saved, calls = _run(
+        create_args,
+        tmp_path,
+        monkeypatch,
+        {"/issues": [_issue()], "/1/timeline": TIMELINE},
+        include_everything=True,
+    )
+
+    assert any(c.endswith("/1/timeline") for c in calls)
+    assert len(saved["timeline_data"]) == 3