sparanoid/laplace-event-fetcher repository overview

⁠LAPLACE Event Fetcher

An event fetcher for bilibili live that retrieves events and stores them as LaplaceEvent[]

⁠Available Tags

• local, dockerhub, <branch>, edge, sha-<hash>, latest

⁠Docker Compose Example

Minimal example. Please feel free to modify it as needed.

services:
  lef:
    image: sparanoid/laplace-event-fetcher:latest
    environment:
      DATABASE_URL: postgresql://lef:lef@lef-pg:5432/lef
      ROOMS: 25034104,456117
      TZ: Asia/Shanghai # recommended, this ensure all cron tasks are executed in CST
    depends_on:
      - lef-pg
      - lef-redis
    restart: always

  lef-pg:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: lef
      POSTGRES_USER: lef
      POSTGRES_PASSWORD: lef
    volumes:
      - lef-db:/var/lib/postgresql/data
    restart: always
    healthcheck:
      test: pg_isready -U lef -h 127.0.0.1
      interval: 5s

  # Redis is optional for serving recent chat messages
  lef-redis:
    image: redis:latest
    volumes:
      - lef-redis:/data
    restart: always

volumes:
  lef-db:
  lef-redis:

⁠Deploy: Koyeb + Neon

First, you need to create a PostgresQL database from Supabase:

• Create a new database in the Supabase dashboard
• Change the region to AWS US East (N. Virginia)
• Remember the database password and write down your connection string from Connection Details - Connection string

Then, we need to deploy the event fetcher on Koyeb:

• Create a new app and choose Docker as your deployment method
• Use docker.io/sparanoid/laplace-event-fetcher as your image. Leave blank or type edge for the latest beta images
• Change the region to WAS (Close to the database you created)
• Choose the eMicro instance (a larger instance can handle more rooms)
• Click the Advanced button and add the following environment variables: ROOMS and DATABASE_URL. The DATABASE_URL variable should be something like postgresql://postgres:<DB_PASSWORD>@xxxxxxxxxxxxxxxxxxxx.us-east-2.aws.neon.tech/neondb?sslmode=require you got from the previous step
• Click Deploy

⁠Tested Serverless Platforms

The following combinations are tested and working:

• Koyeb + Supabase
• Koyeb + Neon
• Render
• Render + Neon

⁠WebSocket API (Bridge Mode)

The server provides a WebSocket mode at the root path (/) for real-time event streaming. This feature can be enabled by setting WEBSOCKET_BRIDGE=1 or WEBSOCKET_BRIDGE=true. When enabled, the server will accept WebSocket connections from LAPLACE Event Bridge SDK.

⁠Authentication

If WEBSOCKET_BRIDGE_AUTH is configured, clients must authenticate using the Sec-WebSocket-Protocol header. For example:

const ws = new WebSocket('ws://localhost:8080/', ['client', 'auth-token'])

Alternatively, you can pass the token via the token query parameter:

const ws = new WebSocket('ws://localhost:8080/?token=auth-token')

Then you'll receive real-time LaplaceEvent as they are processed by the server.

⁠Testing

A test client is included at websocket-test.html that you can open in your browser to test the WebSocket connection and see live events. The test page also includes a Mock Event Sender panel that allows you to send test events through the /mock endpoint.

You can also use wscat to test the WebSocket connection:

# Via Sec-WebSocket-Protocol header
wscat -c ws://localhost:8080 -s client -s <auth-token>

# Via token query parameter
wscat -c 'ws://localhost:8080/?token=<auth-token>'

To send mock events via curl:

curl -X POST http://localhost:8080/mock \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-admin-key-at-least-12-chars" \
  -d '{"type": "message", "message": "Test message", "username": "TestUser"}'

⁠OpenTelemetry

The service includes optional OpenTelemetry support for distributed tracing and observability. When configured, it automatically instruments PostgreSQL operations.

⁠Axiom Logging (Optional)

If you're using Axiom and want structured logging in addition to tracing, the service can automatically configure Pino logger to send logs to Axiom when it detects Axiom credentials in the OTLP headers:

• Logs are automatically sent when both X-Axiom-Dataset and Authorization are present in OTEL_EXPORTER_OTLP_HEADERS
• Log level is set to debug by default

⁠Instrumented Components

When OpenTelemetry is enabled, the following components are automatically instrumented:

• PostgreSQL: All database queries and operations
• HTTP requests: All incoming HTTP requests and responses

⁠API

You can access the API documentation at /openapi for your instance.

• GET /events/<room_id>: Get events from the database by room id
  • queries:
    • ?full=1: Get all events
• POST /upload: Upload events and store them in the database
  • headers:
    • Authorization: Bearer <ADMIN_KEY> (required, token must be at least 12 characters)
  • body:
    • Raw JSON LaplaceEvent[] or LAPLACE Chat Archive (.lca)
• POST /mock: Mock a LaplaceEvent and broadcast it to all connected WebSocket clients (useful for testing). Examples can be found at @laplace.live/event-types⁠
  • headers:
    • Authorization: Bearer <ADMIN_KEY> (required, token must be at least 12 characters)
  • body:
    • JSON object with at least type field (e.g., {"type": "message", "message": "Test message"})
    • All fields from the event will be broadcast with additional mock: true flag and current timestamps
• GET /ping: Check if the server is running. This will connect to the database and return pong in JSON format with a 200 status code, or a 500 status code if it fails.

⁠Environment Variables

• PORT (optional): Server port to listen. Default: 8080
• ROOMS: Rooms to fetch. Multiple rooms can be separated by commas. Default: 456117. It's recommended the rooms you added are less than 10 per node, otherwise Bilibili will prevent you from getting events.
• DATABASE_URL: Database to connect. ie. postgresql://username:password@pg:5432/lef
• REDIS_URL (optional): Redis to connect. ie. redis://username:password@redis:6379
• EVENTS_KEEP (optional): Events older than this value will be discarded. Default: 72 (hours)
• REDIS_EVENT_LIMIT (optional): Maximum number of events to store in Redis per room. Default: 100
• RESTART_WAIT (optional): Time to wait before re-establishing connections. Default: 2000 (ms)
• RESTART_INTERVAL (optional): Connection restart interval in cron format. Default: 0 6,18 * * * (Every day at 6:00 AM and 6:00 PM)
• LOGIN_SYNC_TOKEN (optional): Token from LAPLACE Login Sync extension. Default: undefined. Multiple keys can be separated by commas.
• LOGIN_SYNC_SERVER (optional): Custom sync server. Default: undefined
• AUTH_KEY (optional): A long random string as credentials to access the REST API. Useful if you only want to expose the API to authorized users. Default: undefined. Multiple keys can be separated by commas.
• ADMIN_KEY (optional): A long random string as credentials to upload events to the cloud and send mock events. Must be at least 12 characters long. Default: undefined. You must set this if you want to upload events or use the mock endpoint. You can use openssl rand -hex 32 to generate a random string for this.
• WEBSOCKET_BRIDGE (optional): Enable WebSocket bridge mode for real-time event streaming. Set to false or 0 to disable. Default: undefined (disabled)
• WEBSOCKET_BRIDGE_AUTH (optional): Password for WebSocket authentication. When set, clients must provide this password to connect to the WebSocket endpoint. Default: undefined
• OTEL_EXPORTER_OTLP_ENDPOINT (optional): The OpenTelemetry endpoint URL (e.g., https://api.example.com). When not set, OpenTelemetry is disabled.
• OTEL_EXPORTER_OTLP_HEADERS (optional): Headers to send with OpenTelemetry requests in comma-separated key=value format (e.g., Authorization=Bearer <token>,X-Dataset=my-dataset).
• OTEL_TRACE_SAMPLE_RATE (optional): The sample rate for OpenTelemetry traces. Default: 0.2 (20%)

⁠Development

bun run dev # session 1
bunx drizzle-kit studio # session 2

# init db and create migrations
bunx drizzle-kit migrate

# Update schemas
bunx drizzle-kit generate

⁠TODOs

• Fix drizzle typing errors