tinsaetadesse/t-databus

By tinsaetadesse

•Updated 10 days ago

A custom container Apache Kafka image for KRaft-based deployments, with SSL-secured clients.

Image

Message queues

Data science

Monitoring & observability

155

Overview Tags

tinsaetadesse/t-databus repository overview

⁠T-DataBus Image

A custom, containerized Apache Kafka⁠ image designed for KRaft-based deployments, with SSL-secured client access and persistent Kafka data.

⁠Features

The T-DataBus image comes with the following features:

Java 21 runtime — Temurin JRE 21 is included in the image.
KRaft mode — The DATABUS cluster has no ZooKeeper dependency.
Separate listeners - Broker, controller, and client traffic are handled using separate listeners.
SSL/TLS enabled — Client communication uses an SSL listener on port 9192.
Mutual TLS support — Client communication is only allowed after authentication.
Non-root execution — The application runs as the t-soft user.
Configurable through environment variables — DATABUS settings are supplied through DATABUS_* environment variables.
Multi-architecture build support — The image supports AMD64 and ARM64 architectures.

⁠Container Image

The image contains the T-DATABUS runtime and its supporting components, so end users do not need to install Java or Kafka inside the container.

⁠Main image components

Component	Version / Location
Base OS	Alpine Linux 3.21
Java	Temurin JRE 21.0.10+7
Kafka	Apache Kafka 4.2.1
Scala	2.13
Runtime user	`t-soft`
Application home	`/usr/local/t-soft/t-databus`
Configuration	`/etc/t-soft/t-databus`
Secrets	`/etc/t-soft/t-databus/secrets`
Data	`/data/t-soft/t-databus`
Logs	`/var/log/t-soft/t-databus`

⁠Configuration

T-DataBus configuration is supplied through DATABUS_* environment variables.

Example:

environment:
  DATABUS_NODE_ID: 4
  DATABUS_PROCESS_ROLES: 'broker'
  DATABUS_LISTENERS: 'BROKER://0.0.0.0:9092,CLIENT://0.0.0.0:9192'
  DATABUS_LOG_DIRS: '/data/t-soft/t-databus'

This allows the same image to be reused for different:

Controller nodes
Broker nodes
Node IDs
Hostnames
Listener configurations
Data locations
SSL configurations

without rebuilding the image.

⁠Listener Configuration

Instances can define multiple listener types for inter-broker communication, KRaft controller communication, and client communication (such as communication with producers and consumers):

The configuration below shows how a listener is created for inter-broker communication:

DATABUS_LISTENERS: 'BROKER://broker-N.tsoft.internal:9092'
DATABUS_INTER_BROKER_LISTENER_NAME: 'BROKER'

The settings for KRaft controller communication are:

DATABUS_LISTENERS: 'CONTROLLER://controller-N.tsoft.internal:9093'
DATABUS_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'

Producer and consumer client communication can use the listener configured for this purpose:

DATABUS_LISTENERS: 'CLIENT://broker-N.tsoft.internal:9192'

The listener protocol mapping looks:

DATABUS_LISTENER_SECURITY_PROTOCOL_MAP: 'BROKER:PLAINTEXT,CONTROLLER:PLAINTEXT,CLIENT:SSL'

The client listener uses SSL/TLS.

⁠SSL / mTLS Configuration

The client listener is SSL-enabled and requires client authentication:

DATABUS_SSL_CLIENT_AUTH: required

This means a client connecting to the broker must present a certificate trusted by the broker. All the required keysore, truststore, and credential files used to secure the broker listener can be configured using DATABUS_SSL_* environment variables:

DATABUS_SSL_KEYSTORE_LOCATION: /etc/t-soft/t-databus/secrets/databus-broker.keystore.jks
DATABUS_SSL_TRUSTSTORE_LOCATION: /etc/t-soft/t-databus/secrets/star.tsoft.internal.truststore.jks
DATABUS_SSL_KEYSTORE_CREDENTIAL_FILE: databus-broker.ssl.keystore.credential
DATABUS_SSL_KEY_CREDENTIAL_FILE: databus-broker.ssl.key.credential
DATABUS_SSL_TRUSTSTORE_CREDENTIAL_FILE: star.tsoft.internal.ssl.truststore.credential

Hostname verification is enabled with:

DATABUS_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: HTTPS

⁠Data Persistence

The metadata used in the T-DataBus container is stored at DATA_DIR configured in the image. This can be overridden by specifying the DATABUS_LOG_DIRS environment variable in the compose script:

DATABUS_LOG_DIRS: '/data/t-soft/t-databus'

Application logs, however, are stored under /var/log/t-soft/t-databus by default.

⁠Quick Start

⁠1. Prepare SSL secrets

The broker configuration expects files such as:

include/secrets/
├── databus-broker.keystore.jks
├── star.tsoft.internal.truststore.jks
├── databus-broker.ssl.keystore.credential
├── databus-broker.ssl.key.credential
└── star.tsoft.internal.ssl.truststore.credential

Use the exact filenames referenced by your deployment.

Do not commit private keys, keystores, or credential files to source control.

⁠2. Deployment Example

The compose script below is an example template you can use to quickly spin up T-DataBus containers.

services:
  controller-1:
    image: docker.io/tinsaetadesse/t-databus:latest
    container_name: controller-1
    hostname: controller-1.tsoft.internal
    environment:
      DATABUS_NODE_ID: 1
      # This instance acts as a controller
      DATABUS_PROCESS_ROLES: 'controller'
      # This broker uses BROKER, and CONTROLLER as listener names 
      DATABUS_LISTENER_SECURITY_PROTOCOL_MAP: 'BROKER:PLAINTEXT,CONTROLLER:PLAINTEXT'
      DATABUS_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      DATABUS_INTER_BROKER_LISTENER_NAME: 'BROKER'
      # This instance listens through one (1) port for connecting with brokers
      DATABUS_LISTENERS: 'CONTROLLER://0.0.0.0:9093'
      DATABUS_CONTROLLER_QUORUM_VOTERS: '[email protected]:9093,[email protected]:9093,[email protected]:9093'
      DATABUS_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      DATABUS_LOG_DIRS: '/data/t-soft/t-databus'
    networks:
      my-network:
        aliases:
          - controller-1.tsoft.internal

  controller-2:
    image: docker.io/tinsaetadesse/t-databus:latest
    container_name: controller-2
    hostname: controller-2.tsoft.internal
    environment:
      DATABUS_NODE_ID: 2
      # This instance acts as a controller
      DATABUS_PROCESS_ROLES: 'controller'
      # This broker uses BROKER, and CONTROLLER as listener names 
      DATABUS_LISTENER_SECURITY_PROTOCOL_MAP: 'BROKER:PLAINTEXT,CONTROLLER:PLAINTEXT'
      DATABUS_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      DATABUS_INTER_BROKER_LISTENER_NAME: 'BROKER'
      # This instance listens through one (1) port for connecting with brokers
      DATABUS_LISTENERS: 'CONTROLLER://0.0.0.0:9093'
      DATABUS_CONTROLLER_QUORUM_VOTERS: '[email protected]:9093,[email protected]:9093,[email protected]:9093'
      DATABUS_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      DATABUS_LOG_DIRS: '/data/t-soft/t-databus'
    networks:
      my-network:
        aliases:
          - controller-2.tsoft.internal

  controller-3:
    image: docker.io/tinsaetadesse/t-databus:latest
    container_name: controller-3
    hostname: controller-3.tsoft.internal
    environment:
      DATABUS_NODE_ID: 3
      # This instance acts as a controller
      DATABUS_PROCESS_ROLES: 'controller'
      # This broker uses BROKER, and CONTROLLER as listener names 
      DATABUS_LISTENER_SECURITY_PROTOCOL_MAP: 'BROKER:PLAINTEXT,CONTROLLER:PLAINTEXT'
      DATABUS_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      DATABUS_INTER_BROKER_LISTENER_NAME: 'BROKER'
      # This instance listens through one (1) port for connecting with brokers
      DATABUS_LISTENERS: 'CONTROLLER://0.0.0.0:9093'
      DATABUS_CONTROLLER_QUORUM_VOTERS: '[email protected]:9093,[email protected]:9093,[email protected]:9093'
      DATABUS_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      DATABUS_LOG_DIRS: '/data/t-soft/t-databus'
    networks:
      my-network:
        aliases:
          - controller-3.tsoft.internal

  broker-1:
    image: docker.io/tinsaetadesse/t-databus:latest
    container_name: broker-1
    hostname: broker-1.tsoft.internal
    ports:
      # Listner ports (port 19092 on the host is mapped to 9192 of the container port)
      - '19092:9192'
    environment:
      DATABUS_NODE_ID: 4
      # This instance acts as a broker
      DATABUS_PROCESS_ROLES: 'broker'
      # This broker uses BROKER, CONTROLLER, and CLIENT as listener names 
      DATABUS_LISTENER_SECURITY_PROTOCOL_MAP: 'BROKER:PLAINTEXT,CONTROLLER:PLAINTEXT,CLIENT:SSL'
      DATABUS_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      DATABUS_INTER_BROKER_LISTENER_NAME: 'BROKER'
      # This instance listens through two (2) ports for connecting with brokers, and external clients(producer/consumer)
      DATABUS_LISTENERS: 'BROKER://0.0.0.0:9092,CLIENT://0.0.0.0:9192'
      # This broker exposes it's container name for internal communications, and 
      # 'docker.io' for connections from clients(producer/consumer) running on the host
      DATABUS_ADVERTISED_LISTENERS: 'BROKER://broker-1.tsoft.internal:9092,CLIENT://broker-1.tsoft.internal:9192'
      DATABUS_CONTROLLER_QUORUM_VOTERS: '[email protected]:9093,[email protected]:9093,[email protected]:9093'
      DATABUS_SSL_KEYSTORE_LOCATION: '/etc/t-soft/t-databus/secrets/databus-broker.keystore.jks'
      DATABUS_SSL_KEYSTORE_CREDENTIAL_FILE: 'databus-broker.ssl.keystore.credential'
      DATABUS_SSL_KEY_CREDENTIAL_FILE: 'databus-broker.ssl.key.credential'
      DATABUS_SSL_TRUSTSTORE_LOCATION: '/etc/t-soft/t-databus/secrets/star.tsoft.internal.truststore.jks'
      DATABUS_SSL_TRUSTSTORE_CREDENTIAL_FILE: 'star.tsoft.internal.ssl.truststore.credential'
      DATABUS_SSL_CLIENT_AUTH: 'required'
      DATABUS_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: 'HTTPS'
      DATABUS_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      DATABUS_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      DATABUS_TRANSACTION_STATE_LOG_MIN_ISR: 1
      DATABUS_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      DATABUS_LOG_DIRS: '/data/t-soft/t-databus'
    volumes:
      - ./include/secrets:/etc/t-soft/t-databus/secrets:ro,z
    depends_on:
      - controller-1
      - controller-2
      - controller-3
    networks:
      my-network:
        aliases:
          - broker-1.tsoft.internal

  broker-2:
    image: docker.io/tinsaetadesse/t-databus:latest
    container_name: broker-2
    hostname: broker-2.tsoft.internal
    ports:
      # Listner ports (port 29092 on the host is mapped to 9192 of the container port)
      - '29092:9192'
    environment:
      DATABUS_NODE_ID: 5
      # This instance acts as a broker
      DATABUS_PROCESS_ROLES: 'broker'
      # This broker uses BROKER, CONTROLLER, and CLIENT as listener names 
      DATABUS_LISTENER_SECURITY_PROTOCOL_MAP: 'BROKER:PLAINTEXT,CONTROLLER:PLAINTEXT,CLIENT:SSL'
      DATABUS_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      DATABUS_INTER_BROKER_LISTENER_NAME: 'BROKER'
      # This instance listens through two (2) ports for connecting with brokers, and external clients(producer/consumer)
      DATABUS_LISTENERS: 'BROKER://0.0.0.0:9092,CLIENT://0.0.0.0:9192'
      # This broker exposes it's container name for internal communications, and 
      # 'docker.io' for connections from clients(producer/consumer) running on the host
      DATABUS_ADVERTISED_LISTENERS: 'BROKER://broker-2.tsoft.internal:9092,CLIENT://broker-2.tsoft.internal:9192'
      DATABUS_CONTROLLER_QUORUM_VOTERS: '[email protected]:9093,[email protected]:9093,[email protected]:9093'
      DATABUS_SSL_KEYSTORE_LOCATION: '/etc/t-soft/t-databus/secrets/databus-broker.keystore.jks'
      DATABUS_SSL_KEYSTORE_CREDENTIAL_FILE: 'databus-broker.ssl.keystore.credential'
      DATABUS_SSL_KEY_CREDENTIAL_FILE: 'databus-broker.ssl.key.credential'
      DATABUS_SSL_TRUSTSTORE_LOCATION: '/etc/t-soft/t-databus/secrets/star.tsoft.internal.truststore.jks'
      DATABUS_SSL_TRUSTSTORE_CREDENTIAL_FILE: 'star.tsoft.internal.ssl.truststore.credential'
      DATABUS_SSL_CLIENT_AUTH: 'required'
      DATABUS_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: 'HTTPS'
      DATABUS_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      DATABUS_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      DATABUS_TRANSACTION_STATE_LOG_MIN_ISR: 1
      DATABUS_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      DATABUS_LOG_DIRS: '/data/t-soft/t-databus'
    volumes:
      - ./include/secrets:/etc/t-soft/t-databus/secrets:ro,z
    depends_on:
      - controller-1
      - controller-2
      - controller-3
    networks:
      my-network:
        aliases:
          - broker-2.tsoft.internal

  broker-3:
    image: docker.io/tinsaetadesse/t-databus:latest
    container_name: broker-3
    hostname: broker-3.tsoft.internal
    ports:
      # Listner ports (port 39092 on the host is mapped to 9192 of the container port)
      - '39092:9192'
    environment:
      DATABUS_NODE_ID: 6
      # This instance acts as a broker
      DATABUS_PROCESS_ROLES: 'broker'
      # This broker uses BROKER, CONTROLLER, and CLIENT as listener names 
      DATABUS_LISTENER_SECURITY_PROTOCOL_MAP: 'BROKER:PLAINTEXT,CONTROLLER:PLAINTEXT,CLIENT:SSL'
      DATABUS_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      DATABUS_INTER_BROKER_LISTENER_NAME: 'BROKER'
      # This instance listens through two (2) ports for connecting with brokers, and external clients(producer/consumer)
      DATABUS_LISTENERS: 'BROKER://0.0.0.0:9092,CLIENT://0.0.0.0:9192'
      # This broker exposes it's container name for internal communications, and 
      # 'docker.io' for connections from clients(producer/consumer) running on the host
      DATABUS_ADVERTISED_LISTENERS: 'BROKER://broker-3.tsoft.internal:9092,CLIENT://broker-3.tsoft.internal:9192'
      DATABUS_CONTROLLER_QUORUM_VOTERS: '[email protected]:9093,[email protected]:9093,[email protected]:9093'
      DATABUS_SSL_KEYSTORE_LOCATION: '/etc/t-soft/t-databus/secrets/databus-broker.keystore.jks'
      DATABUS_SSL_KEYSTORE_CREDENTIAL_FILE: 'databus-broker.ssl.keystore.credential'
      DATABUS_SSL_KEY_CREDENTIAL_FILE: 'databus-broker.ssl.key.credential'
      DATABUS_SSL_TRUSTSTORE_LOCATION: '/etc/t-soft/t-databus/secrets/star.tsoft.internal.truststore.jks'
      DATABUS_SSL_TRUSTSTORE_CREDENTIAL_FILE: 'star.tsoft.internal.ssl.truststore.credential'
      DATABUS_SSL_CLIENT_AUTH: 'required'
      DATABUS_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: 'HTTPS'
      DATABUS_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      DATABUS_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      DATABUS_TRANSACTION_STATE_LOG_MIN_ISR: 1
      DATABUS_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      DATABUS_LOG_DIRS: '/data/t-soft/t-databus'
    volumes:
      - ./include/secrets:/etc/t-soft/t-databus/secrets:ro,z
    depends_on:
      - controller-1
      - controller-2
      - controller-3
    networks:
      my-network:
        aliases:
          - broker-3.tsoft.internal

networks:
  my-network:
    external: true

⁠Production Recommendations

Use persistent storage for T-DataBus data.
Keep secret mounts read-only.
Run T-DataBus as the supplied non-root t-soft user.
Use certificates with correct SANs for every advertised broker hostname.
Restrict access to the SSL client and controller ports.
Review whether plaintext inter-broker communication is appropriate for your security requirements.
Use production-appropriate topic replication and ISR settings.
Monitor KRaft controller quorum health.
Monitor broker availability and disk usage.
Back up data according to your recovery requirements.

Tag summary

Recent tags

Content type

Image

Digest

sha256:1a0759e76…

Size

365 MB

Last updated

10 days ago

docker pull tinsaetadesse/t-databus