kinetica/sqlassist repository overview

⁠SQLAssist Docker

SqlAssist provides a Kinetica AI capability that generates SQL based on questions asked in natural language. The functionality integrated into the core database engine using SQL syntax so it can be leveraged by any Kinetica powered application.

• 1. Overview⁠
• 2. Prerequisites⁠
  • 2.1. GPU Requirements⁠
  • 2.2. Docker requirements on RHEL⁠
  • 2.3. Nvidia Container Toolkit⁠
  • 2.4. Testing the runtime⁠
• 3. SQLAssist Images⁠
  • 3.1. Pulling an image⁠
  • 3.2. Optional: Offline transport of images⁠
• 4. Launching the contanier⁠
  • 4.1. Setting the engine type⁠
  • 4.2. Testing the container with Huggingface engine⁠
  • 4.3. Testing the container with the VLLM engine⁠
  • 4.4. Testing with external Llama paramters⁠
  • 4.5. Testing with remote API⁠
  • 4.6. Launch Detached⁠
  • 4.7. Confirm conectivity to API⁠
• 5. Kinetica configuration⁠
• 6. OpenAI compatible /v1/chat/completions⁠
• 6. Release Notes⁠
• 7. References⁠

⁠1. Overview

The SQLAssist container will provide a REST API endpoint that is used when the GENERATE SQL syntax is invoked. The container has a process for 2 services:

• webui: This process manages the the LLM model and will copy it into VRAM at startup. It provides a web interface on port 8050 for direct testing of queries but it should not typically be needed.
• api: The proxy exposes a REST API on port 8050 for servicing requests from Kinetica.

This service is intended to be called only by a Kinetica instance. See the Kinetica user documentation⁠ for more information. For an introduction on programmatic usage of the container see the Kinetica Provider documentation for Langchain⁠.

⁠2. Prerequisites

⁠2.1. GPU Requirements

Kinetica uses a 16 billion parameter transformer model and you will need to run it on a node with GPU's that have at least a cumulative 40GB of VRAM. If you are using a GPU with less than this amount then you can specify multiple GPU's. Some recommended configurations GPU configurations are:

• 4x Nvidia V100 (64GB)
• 1x Nvidia A100 (40GB)

The CPU and RAM requirements are not significant but you will need enough disk for the 50GB docker image.

⁠2.2. Docker requirements on RHEL

If you are using Redhat or CentOS then the default Docker version 1.13.1 is too old.

$ docker --version
Docker version 1.13.1, build 7d71120/1.13.1

As an alternative you can follow the installation instructions for Docker CE Engine for CentOS⁠. After installing you should see a version like this:

$ docker --version
Docker version 25.0.3, build 4debf41

⁠2.3. Nvidia Container Toolkit

The LLM container requires an installation of the Nvidia Container Toolkit so that the GPU's are accessible by the docker runtime. You can check for the installed package as shown. Depending on your OS you will need to follow the yum or apt installation instructions⁠.

If the runtime hook is installed then it should be accessible from the path⁠.

$ which nvidia-container-runtime-hook
/usr/bin/nvidia-container-runtime-hook

Make sure you follow the post installation steps⁠ including the configuration with nvidia-ctk.

sudo nvidia-ctk runtime configure --runtime=docker

⁠2.4. Testing the runtime

Launch the nvidia-smi container to test the docker runtime. You will need to use the --gpus option to make GPU's available in the container⁠. Use all or device=N to indicate which GPU's should be exposed. Verify that the nvidia-smi container displays GPU's you intend to expose.

$ sudo docker run --rm --gpus '"device=0"' ubuntu nvidia-smi

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 535.86.10    Driver Version: 535.86.10    CUDA Version: 12.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            On   | 00000000:00:1E.0 Off |                    0 |
| N/A   34C    P8     9W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

⁠3. SQLAssist Images

⁠3.1. Pulling an image

Images are available based on criteria:

• The Base OS of the container
• The base ML model
• Are parameters included in the image?

You will need to pull a copy of the kinetica/sqlassist⁠ from DockerHub.

$ docker pull kinetica/sqlassist
Using default tag: latest
latest: Pulling from kinetica/SQLAssist
Digest: sha256:f569436fb4d9f1ecbd54d99f36a371030ca08c5bde204a5bf9907011d212c9a2
Status: Image is up to date for kinetica/SQLAssist:latest
docker.io/kinetica/SQLAssist:latest

⁠3.2. Optional: Offline transport of images

If your target environment has no connectivity then you can export the file.

docker save kinetica/sqlassist:latest | gzip > SQLAssist_1.1.tar.tgz

Load the transferred file with:

docker load -i ./SQLAssist_1.1.tar.tgz

⁠4. Launching the contanier

There are multiple ways to launch the container depending on how you have downloaded it. These include:

• Use Huggingface or VLLM inference engine?
• Are your parameters embedded or external?
• Are you running the conteiner interactive or detached?
• Do you shard the model on some or all GPU's?

The examples provided here are not comprehensive and you will need to adapt them depending on your configuration.

⁠4.1. Setting the engine type

The default inference engine is Huggingface. Alternate engine implementations can be selected with the SA_ENGINE_TYPE variable.

⁠4.2. Testing the container with Huggingface engine

Launch the container interactively and verify that it starts up properly. The default inference engine is Huggingface. It may a couple minutes to load the entire model into VRAM. Use CTRL-C to terminate the container.

$ docker run --name SQLAssist-hf \
    --rm --interactive --tty \
    --gpus all \
    --publish 8050:8050 \
    kinetica/sqlassist:1.8.0-1.1-ub22
[...]
INFO:     2024-03-01 03:24:20 [on] Application startup complete.
INFO:     2024-03-01 03:24:20 [server] Uvicorn running on http://0.0.0.0:8050 (Press CTRL+C to quit)

⁠4.3. Testing the container with the VLLM engine

The container can optionally use VLLM⁠ as the inferencing engine instead of HuggingFace transformers. To invoke this pass the SA_USE_VLLM variable and --shm-size options as shown.

docker run --name SQLAssist-vllm \
    --rm --interactive --tty \
    --gpus all \
    --publish 8050:8050 \
    --shm-size=512mb \
    --env SA_ENGINE_TYPE=VLLM \
    kinetica/sqlassist:1.8.0-1.1-ub22

⁠4.4. Testing with external Llama paramters

If the container does not have embedded parameters (e.g. llama-rh8-noparam) then you will need to mount them as an external volume.

docker run --name SQLAssist-llama \
    --rm --interactive --tty \
    -v /opt/sqlassist/models/Meta-Llama-3.1-8B-Instruct:/opt/sqlassist/model \
    --gpus '"device=0"' \
    --publish 8050:8050 \
    kinetica/sqlassist:1.7.3-llama-rh8-noparam 

⁠4.5. Testing with remote API

With SA_ENGINE_TYPE=REMOTE there are no internal parameters loaded and requests will be proxied to an remote host using the /v1/completions API.

For example you could start an external VLLM instance.

vllm serve \
    ${MODEL_PATH} \
    --port 5001 \
    --seed 42

When starting the container you should pass SA_REMOTE_URL and SA_REMOTE_MODEL_PATH with values needed to call the remote API.

docker run --name SQLAssist-remote \
    --rm -it \
    --publish 8050:8050 \
    --env SA_ENGINE_TYPE=REMOTE \
    --env SA_REMOTE_URL=http://172.17.0.1:5001 \
    --env SA_REMOTE_MODEL_PATH=${MODEL_PATH} \
    kinetica/sqlassist:1.9.2-noparam-ub22

The SA_REMOTE_MODEL_PATH is passed as the model parameter when calling /v1/completions and so it must match the path you used to start VLLM.

⁠4.6. Launch Detached

After confirming correct startup you can re-launch in detached mode.

$ docker run --name SQLAssist \
    --rm --detach \
    --gpus all \
    --publish 8050:8050 \
    kinetica/sqlassist:1.8.0-1.1-ub22
c91d84cd25d550ac46cf18e5e9564cc2283c26b26215f06983a8e30ecf3d49b7
$

The command will not display the log but you can check it with:

docker container logs SQLAssist --follow

⁠4.7. Confirm conectivity to API

You can test the service health by calling the /sql/status API.

$ curl -sS -X 'GET' \
  "http://0.0.0.0:8050/sql/status" \
  -H 'accept: application/json' | python -m json.tool
{
    "sa_version": "1.8.0",
    "model": "sqlassist",
    "engine_type": "VLLM",
    "model_type": "gpt_bigcode",
    "gpus": [
        "0=[Tesla V100-PCIE-16GB]",
        "1=[Tesla V100-PCIE-16GB]",
        "2=[Tesla V100-PCIE-16GB]",
        "3=[Tesla V100-PCIE-16GB]"
    ],
    "engine_class": "LLM",
    "tokenizer_class": "CachedGPT2TokenizerFast",
    "vllm_version": "0.6.4.post1",
    "num_gpu_blocks": 18165,
    "block_size": 16,
    "num_gpu_tokens": 290640,
    "correlation_id": "643c2fc0177a",
    "cache_size": 0,
    "vram": {
        "used_gb": 54.72,
        "total_gb": 63.06,
        "free_gb": 8.34
    }
}

⁠5. Kinetica configuration

By default the Kinetica server is configured to communicate with ChatGPT. You will need to configure it to connect to the SQLAssist API running in the container.

Edit the gpudb.conf as follows where ${SQLAssist_HOST} is the host running the container.

# AI API provider type.
ai.api.provider = kineticallm

# AI API URL. The default is "https://sqlgpt.io/api/sql/suggest"
ai.api.url = http://${SQLASSIST_HOST}:8050/sql/suggest

As an alternative you can use the ALTER syntax from Workbench.

ALTER SYSTEM SET PROPERTIES ('ai_api_provider' = 'kineticallm');

ALTER SYSTEM SET PROPERTIES ('ai_api_url' = 'http://172.31.31.13:8050/sql/suggest');

⁠6. OpenAI compatible /v1/chat/completions

If a llama model is running then you can access the model using an endpoint compatible with/v1/chat/completions.

from langchain_openai import ChatOpenAI
from langchain_core.messages import AIMessage, HumanMessage, SystemMessage

llm = ChatOpenAI(
    api_key="???", # You can provide any fake key. It is not checked.
    base_url="http://${SQLASSIST_HOST}:8050/sql/v1",
    model="meta-llama/Llama-3.3-70B-Instruct-fast", # this is not checked
    temperature=0,
    max_tokens=None,
    timeout=None,
    max_retries=2)

messages = [
    SystemMessage("You are a helpful assistant that translates English to French. Translate the user sentence"),
    HumanMessage("I love programming."),
]
response: AIMessage = llm.invoke(messages)
print(f"content: {response.content}")

⁠6. Release Notes

Docker labels follow a syntax of ${code_version}-${model_version}

• 1.9.4

  • Upgrade libs including torch=2.8.0.

• 1.9.3

  • Add support for /v1/chat/completions API.

• 1.9.2

  • Add support for remote API engine.

• 1.9.0

  • Add support for generation of embeddings with llama. (experimental)

• 1.8.0

  • Add support for Llama 3.1 base model.
  • Add Redhat 8 base OS image.
  • Update python libraries.

• 1.6.0-1.1

  • Update Nvidia runtime to cuda:12.6.0
  • Add support for VLLM.
  • Add new parameters to /sql/status

• 1.5.9-1.1

  • Update max_new_tokens=512

• 1.5.8-1.1

  • Update to transformers 4.41.2
  • Add /sql/count_tokens API

• 1.5.7-1.1

  • Update to 12.5.0-runtime-ubuntu22.04
  • Update to requests 2.32.3
  • Various python library updates

• 1.5.6-1.1

  • Update transformers to 4.40.2
  • Update runtime to 12.4.1-runtime-ubuntu22.04
  • Fix bug in beam search
  • Various package updates

• 1.5.4-1.1

  • Enhanced caching
  • Extend context size to 4096

⁠7. References

• Kinetica User Documentation⁠
• SQLAssist DockerHub⁠
• Docker Command Line Reference⁠