langtechbsc/vllm-with-params repository overview

This Docker image is adapted from vllm/vllm-openai⁠. It includes an argument called PARAMS that can be passed to the command line.

To run the image, use the following command:

docker run langtechbsc/vllm-with-params:0.6.3 -e PARAMS --model model_name --[other arguments]

Note: The x.y.z.1 tag includes a fixed version of the hermes-tool-parser. To enable it, use the --tool-call-parser hermes_fixed option.

Make sure to replace model_name and [other arguments] with the appropriate values when running the command.