Indonesian Language & Quote Scraper API

A REST API for Indonesian language data built with Node.js, Express 5, and TypeScript. The service scrapes KBBI for dictionary definitions and Wikiquote for Indonesian proverbs plus Indonesian figure profiles, photos, descriptions, and quotes.

Features

• KBBI word search with structured headwords, word classes, and definitions.
• Anonymous word visit tracking using X-Visitor-Id.
• Top visited words API backed by Supabase aggregation.
• Paginated Indonesian proverb list, search, and detail endpoints.
• Paginated Indonesian figure summary, search, and detail endpoints.
• Request tracing with X-Request-Id, centralized error handling, and request logging with Pino.
• IP-based rate limiting for public routes and stricter scraper-backed search endpoints.
• Controller-service architecture with focused Vitest coverage.
• Vercel-compatible serverless deployment configuration.

API Documentation

See docs/API.md for the human-readable endpoint reference and docs/openapi.yaml for the OpenAPI contract. When the server is running, interactive Swagger UI is available at http://localhost:3000/docs.

Quick examples:

curl http://localhost:3000/search/demokrasi
curl -H "X-Request-Id: local-debug-1" http://localhost:3000/search/demokrasi
curl -H "X-Visitor-Id: anonymous-client-id" http://localhost:3000/search/demokrasi
curl http://localhost:3000/words/top?limit=10
curl "http://localhost:3000/proverb/search?q=air&page=1&limit=5"
curl "http://localhost:3000/figure/search?q=soekarno"
curl "http://localhost:3000/figure/Soekarno"

Every response includes an x-request-id header. Provide X-Request-Id to preserve a client-generated correlation ID, or omit it and the API will generate one.

Requirements

• Node.js compatible with the versions required by the dependencies in package.json.
• npm.
• Supabase project for word visit tracking and top visited words.

The scraping endpoints can run without Supabase, but word visit tracking and /words/top require Supabase configuration.

Environment Variables

Create a .env file in the project root:

PORT=3000
BASE_URL=http://localhost:3000
RATE_LIMIT_GLOBAL_WINDOW_MS=900000
RATE_LIMIT_GLOBAL_MAX=300
RATE_LIMIT_SCRAPER_WINDOW_MS=900000
RATE_LIMIT_SCRAPER_MAX=30
WIKIQUOTE_CACHE_TTL_MS=3600000
KBBI_FETCH_TIMEOUT_MS=45000
# Optional. Reserved for future salted visitor hashing support.
VISITOR_HASH_SALT=replace-with-random-secret

# Optional for scraping endpoints. Required together for visit tracking and /words/top.
SUPABASE_URL=https://your-project.supabase.co
SUPABASE_ANON_KEY=your-anon-key
# Required in production for visit tracking. Used before SUPABASE_ANON_KEY when present.
SUPABASE_SERVICE_ROLE_KEY=your-service-role-key

Variable	Required	Description
PORT	No	Positive integer server port. Defaults to 3000. Invalid values fail startup.
BASE_URL	No	Valid URL used in the root endpoint examples. Defaults to http://localhost:3000. Invalid values fail startup.
RATE_LIMIT_GLOBAL_WINDOW_MS	No	Positive integer global rate limit window in milliseconds. Defaults to 900000 (15 minutes).
RATE_LIMIT_GLOBAL_MAX	No	Positive integer global request limit per IP per window. Defaults to 300.
RATE_LIMIT_SCRAPER_WINDOW_MS	No	Positive integer scraper/search endpoint rate limit window in milliseconds. Defaults to 900000 (15 minutes).
RATE_LIMIT_SCRAPER_MAX	No	Positive integer scraper/search request limit per IP per window. Defaults to 30.
WIKIQUOTE_CACHE_TTL_MS	No	Positive integer TTL for Wikiquote proverb and figure list/detail caches in milliseconds. Defaults to 3600000.
KBBI_FETCH_TIMEOUT_MS	No	Positive integer timeout for each upstream KBBI HTML fetch in milliseconds. Defaults to 45000 (45 seconds).
SUPABASE_URL	For visit tracking	Valid Supabase project URL. If provided, either SUPABASE_ANON_KEY or SUPABASE_SERVICE_ROLE_KEY is required.
SUPABASE_ANON_KEY	Local/dev fallback	Supabase anon key. Only use for local development unless you add explicit public RLS policies.
SUPABASE_SERVICE_ROLE_KEY	Production	Server-only key for visit tracking. Takes precedence over SUPABASE_ANON_KEY and must never be exposed publicly.
VISITOR_HASH_SALT	No	Reserved for future salted visitor hashing support. Not required by the current implementation.

Configuration is validated at startup. Missing Supabase variables are allowed so scraping endpoints can run without visit tracking, but partial Supabase configuration fails startup with an explicit error.

Wikiquote proverb and Indonesian figure list/detail responses are cached in process memory until WIKIQUOTE_CACHE_TTL_MS expires. Requests before expiry reuse cached data; the first request after expiry refreshes the data from Wikiquote. The cache is process-local, resets on restart, and is not shared across multiple deployed instances.

Installation

git clone https://github.com/your-username/kbbi-api.git
cd kbbi-api
npm install

Database Setup

Hosted Supabase

1. Create a Supabase project from the Supabase dashboard.
2. Open Project Settings > API.
3. Copy the project URL into SUPABASE_URL.
4. Copy the service role key into SUPABASE_SERVICE_ROLE_KEY. Keep this key server-side only.
5. Open SQL Editor and run the migration files in this order:

supabase/migrations/001_create_word_visits.sql
supabase/migrations/002_create_top_word_visits_view.sql
supabase/migrations/20260722022955_secure_word_visits_rls.sql

The word_visits table stores one unique visit per word, visitor_hash, and visited_date. The top_word_visits view powers GET /words/top. Row-Level Security is enabled and direct anon/authenticated access is revoked because this API accesses Supabase through the backend service role.

Local Supabase CLI

This repository includes supabase/config.toml for local Supabase development.

Start Supabase locally:

supabase start

Reset the local database and apply migrations:

supabase db reset

Use the local API URL and anon key printed by supabase start in .env:

SUPABASE_URL=http://127.0.0.1:54321
SUPABASE_ANON_KEY=your-local-anon-key

Migration Files

supabase/migrations/001_create_word_visits.sql
supabase/migrations/002_create_top_word_visits_view.sql
supabase/migrations/20260722022955_secure_word_visits_rls.sql

The first migration creates public.word_visits and an index on word. The second migration creates public.top_word_visits, an aggregate view used by the top visited words endpoint. The third migration enables RLS and restricts table/view access to the backend service role.

Verify Supabase

After configuring .env and starting the API, verify the connection:

curl http://localhost:3000/health/supabase

Then verify visit tracking by sending a stable visitor ID:

curl -H "X-Visitor-Id: local-test-user" http://localhost:3000/search/demokrasi
curl http://localhost:3000/words/top?limit=10

Development

Start the development server with auto-reload:

npm run dev

The API will be available at http://localhost:3000 unless PORT is changed.

Before opening a pull request, run the full local quality gate:

npm run check

Rate Limiting

The API applies a loose global limit to all routes and a stricter limit to scraper-backed search routes. By default, each IP can make 300 total requests per 15 minutes and 30 requests per 15 minutes to these endpoints:

GET /search/:word
GET /proverb/search
GET /figure/search

Requests over the limit return HTTP 429:

{
  "success": false,
  "message": "Too many requests"
}

Standard RateLimit headers are included where supported. The default limiter uses in-memory storage, so limits are tracked per Node.js process or serverless runtime instance.

Indonesian Figures

GET /figure and GET /figure/search return paginated summaries by default. Summary items contain name, slug, and sourceUrl; use GET /figure/:slug for full photo, description, and quotes.

For compatibility with older detailed list responses, pass includeDetails=true to /figure or /figure/search. This opt-in mode fetches detail pages for the current page items, so it is slower than the default summary response.

Available Scripts

Command	Description
npm run dev	Start the TypeScript development server with auto-reload.
npm run build	Compile TypeScript into dist/.
npm start	Run the compiled server from dist/server.js.
npm test	Run the Vitest test suite.
npm run lint	Run ESLint on source and test TypeScript files.
npm run format	Format source, test, JSON, and Markdown files with Prettier.
npm run format:check	Check formatting without writing changes.
npm run typecheck	Run TypeScript type checking without emitting files.
npm run check	Run type checking, linting, format check, tests, and build.

Testing

Run all tests:

npm test

This runs the unit tests and integration tests. Integration tests exercise the real Express app through HTTP requests with external KBBI, Wikiquote, and Supabase dependencies mocked for deterministic local runs.

Run a production build check:

npm run build

The test suite covers parser fixtures, controller behavior, error responses, middleware, word visit tracking, top visited words behavior, and HTTP integration behavior.

Code Quality

This project uses Vitest for tests, ESLint flat config with TypeScript support for linting, and Prettier for formatting. ESLint formatting conflicts are disabled so Prettier owns code style while ESLint focuses on code quality.

Run the full local verification before merging dependency updates or behavior changes:

npm run check

Project Structure

src/
├── config/             # Environment and Supabase configuration
├── controllers/        # Express request handlers
├── interfaces/         # TypeScript response and domain types
├── lib/                # Shared HTTP, logging, and async utilities
├── middlewares/        # Request logging and error handling
├── routes/             # API route definitions
├── services/           # Scraping, parsing, and persistence logic
├── app.ts              # Express application setup
└── server.ts           # Server entry point

docs/
├── API.md              # Endpoint reference
└── openapi.yaml        # OpenAPI contract

supabase/
└── migrations/         # Database schema and view migrations

test/
├── fixtures/           # HTML parser fixtures
└── *.test.ts           # Vitest tests

Deployment

The repository includes vercel.json configured to route all requests to src/server.ts with @vercel/node.

For production deployment:

1. Configure the environment variables in the hosting provider.
2. Apply Supabase migrations.
3. Deploy the app.
4. Confirm GET /health/supabase reports the expected Supabase connection status.

Visit Tracking Behavior

Clients may send X-Visitor-Id when calling GET /search/:word. The API hashes this value before storage and counts one unique visit per word, visitor, and day. Search responses include visitorCount; it is null when the header is missing or Supabase tracking is unavailable.

License

This project is licensed under the ISC License.