feat: implement rubygems registry [CM-1241]

[mbani01]

This pull request introduces a new RubyGems worker to the packages_worker service, adding support for ingesting and processing RubyGems package data. The main changes include Docker Compose and script updates for the new worker, as well as the implementation of RubyGems batch processing logic for both core and critical packages. The code adds API clients, data normalization, and database integration for RubyGems packages, versions, and maintainers.

RubyGems Worker Integration:

• Added Docker Compose configuration (rubygems-worker.yaml) to run and develop the new RubyGems worker service with appropriate environment variables, volumes, and network settings.
• Updated package.json scripts to support starting and developing the RubyGems worker locally and in Docker.

RubyGems Batch Processing Implementation:

• Added processRubyGemsCoreBatch and processRubyGemsCriticalBatch activities, which handle batch processing for core and critical RubyGems packages, respectively. [1] [2]
• Implemented batch processing logic in runRubyGemsCoreLoop.ts and runRubyGemsCriticalLoop.ts for fetching, normalizing, and upserting package, version, and maintainer data into the database, including error handling and retry logic. [1] [2]

RubyGems Registry Integration:

• Added API client methods to fetch package metadata, versions, and owners from the RubyGems registry, with error classification for common HTTP errors.
• Introduced normalization utilities to convert RubyGems registry responses into the service's internal data format.

Configuration:

• Added getRubyGemsConfig to manage batch size, concurrency, and delay settings for RubyGems batch processing, with environment variable support.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[Copilot]

Pull request overview

This PR adds a new RubyGems registry ingestion capability to the packages_worker service, following the established multi-entry-point worker pattern (most closely mirroring the existing NuGet worker). It introduces an API client for rubygems.org, normalization utilities, two Temporal batch loops (a "core" metadata/downloads pass and a "critical" versions/maintainers pass), their activities/workflows/schedules, a dev/prod Docker Compose file, and supporting data-access-layer helpers.

Changes:

• New rubygems/ worker module: client.ts, normalize.ts, types.ts, runRubyGemsCoreLoop.ts, runRubyGemsCriticalLoop.ts, activities.ts, workflows.ts, schedule.ts, and bin/rubygems-worker.ts, wired into activities.ts, workflows/index.ts, and getRubyGemsConfig.
• New DAL helpers: osspckgs/rubygems.ts (package selection queries) and a generic osspckgs/downloads.ts (recordDownloadSnapshot), plus new root exports.
• Ops wiring: scripts/services/rubygems-worker.yaml and package.json start/dev scripts (inspect port 9244).

Reviewed changes

Copilot reviewed 16 out of 17 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
services/libs/data-access-layer/src/osspckgs/rubygems.ts	Keyset/priority selection queries for core and critical RubyGems packages.
services/libs/data-access-layer/src/osspckgs/downloads.ts	Generic download-snapshot recorder (duplicate of the NuGet variant).
services/libs/data-access-layer/src/index.ts	Exports new osspckgs/types, downloads, rubygems modules.
services/apps/packages_worker/src/workflows/index.ts	Exports RubyGems workflows.
services/apps/packages_worker/src/rubygems/workflows.ts	Core/critical continue-as-new workflows.
services/apps/packages_worker/src/rubygems/types.ts	Fetch/normalized RubyGems type definitions.
services/apps/packages_worker/src/rubygems/schedule.ts	Daily cron schedules for both workflows.
services/apps/packages_worker/src/rubygems/runRubyGemsCriticalLoop.ts	Versions/maintainers batch loop (non-draining on skip/error).
services/apps/packages_worker/src/rubygems/runRubyGemsCoreLoop.ts	Metadata/downloads batch loop with error marking.
services/apps/packages_worker/src/rubygems/normalize.ts	Pure normalization helpers (untested).
services/apps/packages_worker/src/rubygems/client.ts	Axios client with 404/429 error classification.
services/apps/packages_worker/src/rubygems/activities.ts	Activity wrappers over the two loops.
services/apps/packages_worker/src/config.ts	Adds getRubyGemsConfig (used only by the core loop).
services/apps/packages_worker/src/bin/rubygems-worker.ts	Worker entrypoint registering both schedules.
services/apps/packages_worker/src/activities.ts	Registers the new RubyGems activities.
services/apps/packages_worker/package.json	start/dev scripts for the RubyGems worker.
scripts/services/rubygems-worker.yaml	Compose service (omits CROWD_TEMPORAL_NAMESPACE).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 18 changed files in this pull request and generated 2 comments.