hatamiarash7/redis-watcher repository overview

⁠Redis Watcher

Redis Watcher is a small, production-minded daemon that subscribes to a Redis server's MONITOR stream, parses every command it observes and forwards the result to logs, Prometheus metrics, and alert channels (Telegram, generic webhooks, Prometheus Pushgateway).

⁠Features

• Connects to Redis over unix socket or TCP
• Streams MONITOR events with automatic exponential-backoff reconnect
• Parses timestamp, DB number, source IP/port, command + arguments
• Multiple outputs in parallel:
  • rotated file (lumberjack)
  • stdout (JSON or text)
  • UDP/TCP forwarder (Fluent Bit, Fluentd, syslog, …)
• Prometheus metrics (commands, per-IP, per-DB, alerts, drops, reconnects)
• Alerts on suspicious commands (FLUSH*, CONFIG, ACL, KEYS, EVAL, SCRIPT, SHUTDOWN, DEBUG, …) with per-(command, IP) rate limiting; delivered via Telegram, webhook, or Pushgateway
• Sentry integration for runtime error visibility
• Drop-on-full backpressure policy to protect the MONITOR connection

⁠Architecture

                        +--------------------+
                        |   Redis (MONITOR)  |
                        +---------+----------+
                                  |
                                  v
                       +----------+----------+
                       |  monitor.Client     |
                       |  (RESP, reconnect)  |
                       +----------+----------+
                                  |
                          events chan *Event
                                  |
                                  v
                         +--------+--------+
                         |   Dispatcher    |
                         +--------+--------+
                                  |
        +-------------------------+-------------------------+
        |                         |                         |
        v                         v                         v
+---------------+        +-----------------+        +-----------------+
| Prom metrics  |        |    Outputs      |        |  Alert engine   |
| /metrics:9100 |        | file/stdout/    |        |  Telegram /     |
|               |        | UDP / TCP       |        |  Webhook / PGW  |
+---------------+        +-----------------+        +-----------------+

Each sink runs in its own goroutine with a bounded buffered channel. When pipeline.drop_on_full: true (recommended), a slow downstream cannot back up the MONITOR consumer.

⁠Quick start

⁠Local Go build

git clone https://github.com/hatamiarash7/Redis-Watcher.git
cd Redis-Watcher
cp config.example.yaml config.yaml      # edit to taste
make build
./bin/redis-watcher --config config.yaml

⁠Docker Compose (recommended for kicking the tires)

cp config.example.yaml config.yaml
make docker-compose-up
curl http://localhost:9100/metrics      # metrics
docker logs -f rw-watcher               # process logs

The compose stack starts a Redis instance with both a TCP port and a unix socket (shared via a named volume) so the watcher can connect over either transport.

⁠Configuration

Configuration is loaded in this order (later sources override earlier ones):

1. Built-in defaults
2. YAML file passed via --config (or REDIS_WATCHER_CONFIG)
3. Environment variables prefixed with REDIS_WATCHER_

See config.example.yaml⁠ for an exhaustive, commented example.

⁠Useful environment variables

⁠Sentinel-aware role detection

In Redis Sentinel deployments the primary may move between hosts at any time. Running Redis Watcher on every node would duplicate audit trails across the fleet and trigger spurious alerts from the replication command stream itself. Also, the real source IP:PORT will be shown only when the instance is master

Redis Watcher therefore ships with a built-in role detector. It probes the upstream Redis with INFO replication every few seconds:

• While the instance is the primary, the pipeline runs normally.
• When the instance becomes a replica (e.g. after a Sentinel failover) the MONITOR connection is dropped immediately and outputs/metrics/alerts pause until the role flips back.

role_check:
  enabled: true
  interval: 5s
  dial_timeout: 3s
  read_timeout: 3s
  allow_replica: false   # set to true only for debugging or non-Sentinel setups

Observability:

redis_watcher_redis_is_master                       # gauge: 1=master, 0=replica
redis_watcher_redis_role_info{role="master"}        # 1 for the currently observed role
redis_watcher_redis_role_transitions_total{from,to} # counter of role flips

Deployment pattern: install Redis Watcher as a sidecar on every node that can become a primary (e.g. as a DaemonSet, or alongside each Redis unit in your service manager). With role_check enabled this is safe: only one Redis Watcher in the cluster will actively audit at any time, and the active one automatically follows the primary across failovers.

⁠Filtering noisy commands

Busy production hosts emit a lot of PING, INFO, AUTH, SELECT, SUBSCRIBE and similar housekeeping traffic that is almost never worth auditing. Use the top-level filter section to drop these commands as early as possible:

filter:
  ignored_commands:
    - PING
    - INFO
    - AUTH
    - SELECT
    - HELLO
    - COMMAND
    - SUBSCRIBE
    - UNSUBSCRIBE
    - PSUBSCRIBE
    - PUNSUBSCRIBE

Matching is case-insensitive on the command name (the first token of the Redis command). Listing a parent command like CLIENT also silences every CLIENT <subcommand> invocation.

Filtered events do not reach outputs, metrics or alerts. They are counted separately in redis_watcher_ignored_events_total{command="..."} so you can still verify the filter is doing what you expect.

⁠Prometheus metrics

All metrics are exposed at /metrics on the configured metrics.address (default :9100). Notable series:

Health endpoints /healthz and /readyz are also exposed for liveness / readiness probes.

⁠Alerts

The alert engine matches each event against:

1. alerts.suspicious_commands — exact command names (e.g. FLUSHALL, CONFIG). For commands that route by subcommand (CONFIG, CLIENT, ACL, SCRIPT, …) the watcher exposes the joined name (e.g. CONFIG SET) as the alert title.
2. alerts.patterns — case-insensitive regular expressions applied to the reconstructed command line.

Rate limiting is applied per (command, source_ip) tuple. When the rate is exceeded the event is recorded in metrics but not pushed downstream.

⁠Production checklist

• Run on a Redis replica (the primary should not pay the MONITOR tax). Replicas still see every write because of replication.
• Use network: unix when watching the local instance — no port, no network round-trip, easier filesystem permissions.
• Set metrics.track_source_ip: false if clients use many IPs.
• Keep pipeline.drop_on_full: true. Back-pressuring MONITOR causes Redis to drop the watcher.
• Scrape /metrics, alert on: rate(redis_watcher_dropped_events_total[5m]), rate(redis_watcher_monitor_reconnects_total[5m]) > 0, rate(redis_watcher_suspicious_commands_total[5m]).
• Capture Sentry events for the monitor, metrics_server, and outputs components.
• Make sure the unix socket has restricted permissions (unixsocketperm 770 in redis.conf).

⁠Development

make tidy            # go mod tidy
make fmt             # go fmt + goimports
make vet lint        # static checks
make test            # unit tests with race detector
make test-cover      # unit tests + coverage report
make test-integration  # against a running Redis instance
make docker          # build the OCI image

See Makefile⁠ for the full list of targets.

⁠💛 Support

⁠🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch: git checkout -b feature/my-new-feature
3. Install development dependencies: make install-dev
4. Make your changes and add tests
5. Run checks: make check
6. Commit your changes: git commit -am 'Add some feature'
7. Push to the branch: git push origin feature/my-new-feature
8. Submit a pull request

⁠🐛 Issues

Found a bug or have a suggestion? Please open an issue⁠.