Sitemap includes non-content files (assets, scripts, robots.txt)

[stephanoodle]

Context

While fixing a sitemap 500 error on the support site (dnsimple/dnsimple-support#1958), @weppos flagged that both sites should be kept in sync. This issue tracks the equivalent problem on the developer site.

What's wrong

The live sitemap at https://developer.dnsimple.com/sitemap includes non-content files that should not be indexed:

• assets/css/style.css
• assets/images/border.jpg (and 15+ other image files)
• ddns/ddns.sh
• robots.txt
• llms.txt
• v2/openapi.yml
• The sitemap page itself (/sitemap)

These inflate the sitemap with low-value URLs and can signal poor sitemap quality to crawlers.

Additionally, all entries share the same lastmod date (2026-05-25), which reflects the last deploy timestamp rather than actual content changes. Google has said it ignores or distrusts lastmod when it appears inaccurate or uniform across all URLs — which is exactly the case here.

What already exists

lib/default.rb has a sitemap_items helper that's already doing some filtering — it excludes is_hidden items and the /v2/platform/ subtree. The fix builds on that by also excluding items that don't resolve to page URLs.

Suggested fix

In lib/default.rb, update sitemap_items to reject items whose path is nil or doesn't end with / (pages on this site use trailing-slash URLs; assets and files resolve to extensions):

def sitemap_items
  @items.reject do |item|
    item[:is_hidden] ||
      platform_sitemap_excluded?(item.identifier.to_s) ||
      item.path.nil? ||
      !item.path.end_with?('/')
  end
end

This would keep all legitimate content pages (/v2/domains/, /getting-started/, etc.) while excluding CSS, images, scripts, text files, and the sitemap itself.

References

• Fix sitemap.xml 500 error dnsimple-support#1958 — the support site fix this is paired with
• nanoc XMLSitemap helper docs
• nanoc issue #1082 — documents the recursive dependency failure mode from bare xml_sitemap calls

[stephanoodle]

Update: support site fix is now landed — here's what consistency actually means

The support site PR (dnsimple/dnsimple-support#1958) has been approved with @sbastn's revision. The final fix on the support site is:

<%= xml_sitemap(rep_select: ->(rep) { rep.path == "/" || rep.path.end_with?("/") }) %>

It uses the rep_select option (which filters item representations by their URL path) rather than items:. Only pages with trailing-slash URLs are included — articles, sections, the homepage. Everything else — assets, scripts, text files, the sitemap itself — is excluded because those paths don't end with /.

How the developer site already differs

The developer site already has more filtering logic than the support site via the sitemap_items helper in lib/default.rb:

def sitemap_items
  @items.reject do |item|
    item[:is_hidden] || platform_sitemap_excluded?(item.identifier.to_s)
  end
end

Two extra rules that the support site doesn't have:

1. is_hidden exclusion — items with is_hidden: true in frontmatter are kept out of the sitemap. The support site has no hidden items currently, but the support site PR adds support for this pattern going forward.
2. /v2/platform/ subtree exclusion — these pages are intentionally excluded from the sitemap but remain in on-site search. This is developer-site-specific logic with no equivalent on the support site.

What consistency looks like in practice

To match the support site pattern (inline rep_select in the template, no lib helper), the developer site template would become:

<%= xml_sitemap(rep_select: ->(rep) {
  (rep.path == "/" || rep.path.end_with?("/")) &&
  !rep.item[:is_hidden] &&
  !platform_sitemap_excluded?(rep.item.identifier.to_s)
}) %>

platform_sitemap_excluded? would stay in lib/default.rb since it's a named helper, but sitemap_items itself would go away.

Alternatively, keep sitemap_items as a helper but switch the template to use rep_select instead of items: — so the mechanism matches the support site even if the helper stays:

# lib/default.rb
def sitemap_rep_select
  ->(rep) {
    (rep.path == "/" || rep.path.end_with?("/")) &&
    !rep.item[:is_hidden] &&
    !platform_sitemap_excluded?(rep.item.identifier.to_s)
  }
end

<%= xml_sitemap(rep_select: sitemap_rep_select) %>

Either approach fixes the junk-URL problem and aligns the filtering mechanism with the support site. The choice is mainly about where to keep the developer-specific logic.