modular/max-openai-api repository overview

⁠MAX containers

Modular Accelerated eXecution (MAX)⁠ provides a high-performance, flexible platform for AI workloads, leveraging modern GPUs to deliver accelerated generative AI performance while maintaining portability across different hardware configurations and cloud providers.

The MAX container is compatible with the OpenAI API specification and optimized for deployment on GPUs. For more information on container contents and instance compatibility, see MAX containers⁠ in the MAX documentation.

⁠Quick Start

You can run an LLM on GPU using the latest MAX container with the following command:

docker run --gpus 1 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_HUB_ENABLE_HF_TRANSFER=1" \
    --env "HF_TOKEN=<secret>" \
    -p 8000:8000 \
    modular/max-openai-api:<version> \
    --model-path <model-provider/model-id>

You can run a MAX-optimized model⁠⁠ by referencing its Hugging Face model ID. For example, modularai/Llama-3.1-8B-Instruct-GGUF

You can also use the MAX container to run a variety of LLMs hosted on Hugging Face, such as Qwen/Qwen2.5-1.5B-Instruct.

For more information on quickly deploying popular models with MAX, see MAX Builds⁠.

⁠Tags

Supported tags are updated to the latest MAX versions, which include the latest stable release and more experimental nightly releases. The latest tag provides you with the latest stable version and the nightly tag provides you with the latest nightly version.

Stable

• max-openai-api:25.1.1

Nightlies

• max-openai-api:25.2.0.devYYYYMMDD

⁠Documentation

For more information on Modular and its products, visit the Modular documentation site⁠.

⁠Community

To stay up to date with new releases, sign up for our newsletter⁠, check out the community⁠, and join our forum⁠.

If you're interested in becoming a design partner to get early access and give us feedback, please contact us⁠.

⁠License

This container is released under the NVIDIA Deep Learning Container license⁠.