Deploying Open-Source LLMs Locally: vLLM, Ollama & Private Cloud GPU Inference

While proprietary APIs like OpenAI and Anthropic offer incredible performance, banks, healthcare networks, and government agencies cannot allow unencrypted customer records to leave their private infrastructure.

With modern open-weight models like Meta Llama 3 and Mistral, enterprises can match proprietary performance while running entirely on-premises or within an isolated private cloud VPC.

Below is a production Docker Compose definition running vLLM with FP8/AWQ quantization and continuous batching across NVIDIA GPUs.

version: '3.8'

services:
  vllm-engine:
    image: vllm/vllm-openai:latest
    runtime: nvidia
    restart: always
    environment:
      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
    command: >
      --model meta-llama/Meta-Llama-3.1-70B-Instruct
      --quantization bitsandbytes
      --load-format bitsandbytes
      --tensor-parallel-size 2
      --max-model-len 8192
      --gpu-memory-utilization 0.92
      --port 8000
    ports:
      - "8000:8000"
    volumes:
      - /mnt/fast-nvme/huggingface_cache:/root/.cache/huggingface
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Because vLLM serves an OpenAI-compatible API, you simply point your existing LangChain, LlamaIndex, or OpenAI SDK client to http://internal-gpu.local:8000/v1 without altering application logic.