Image to text

Multimodal large language models are capable of processing images and text together in a single request. They can describe visual content, answer questions about images, and support tasks such as image captioning, document analysis, chart interpretation, optical character recognition (OCR), and content moderation.

Endpoint

You can interact with a multimodal LLM through the v1/chat/completions endpoint by including image inputs alongside text in the request. This allows you to provide an image URL or base64-encoded image as part of the conversation, enabling use cases such as image captioning, asking questions about a photo, requesting a chart summary, or combining text prompts with visual context.

Within the v1/chat/completions request body, the "messages" array might look similar to the following for image-to-text:

"messages": [
  {
    "role": "user",
    "content": [
      {
        "type": "text",
        "text": "What is in this image?"
      },
      {
        "type": "image_url",
        "image_url": {
          "url": "https://example.com/path/to/your-image.jpg"
      }
    ]
  }
]

Quickstart

In this quickstart, learn how to set up and run Llama 3.2 11B Vision Instruct, which excels at tasks such as image captioning and visual question answering.

GPU required

To run Llama 3.2 11B Vision Instruct, your system must have a compatible GPU.

System requirements:

Mac

Linux

WSL

GPU

Set up your environment

Create a Python project to install our APIs and CLI tools:

pixi

1. If you don't have it, install pixi:

   curl -fsSL https://pixi.sh/install.sh | sh

   Then restart your terminal for the changes to take effect.

2. Create a project:

   pixi init vision-quickstart \
     -c https://conda.modular.com/max-nightly/ -c conda-forge \
     && cd vision-quickstart

   Tip: You can skip the -c options if you add these channels as defaults.

3. Install the modular conda package:
   Nightly

   pixi add modular

   Stable

   pixi add "modular=25.5"

4. Start the virtual environment:

   pixi shell

uv

1. If you don't have it, install uv:

   curl -LsSf https://astral.sh/uv/install.sh | sh

   Then restart your terminal to make uv accessible.

2. Create a project:

   uv init vision-quickstart && cd vision-quickstart

3. Create and start a virtual environment:

   uv venv && source .venv/bin/activate

4. Install the modular Python package:
   Nightly

   uv pip install modular \
     --index-url https://dl.modular.com/public/nightly/python/simple/ \
     --prerelease allow

   Stable

   uv pip install modular \
     --extra-index-url https://modular.gateway.scarf.sh/simple/

pip

1. Create a project folder:

   mkdir vision-quickstart && cd vision-quickstart

2. Create and activate a virtual environment:

   python3 -m venv .venv/vision-quickstart \
     && source .venv/vision-quickstart/bin/activate

3. Install the modular Python package:
   Nightly

   pip install --pre modular \
     --index-url https://dl.modular.com/public/nightly/python/simple/

   Stable

   pip install modular \
     --extra-index-url https://modular.gateway.scarf.sh/simple/

conda

1. If you don't have it, install conda. A common choice is with brew:

   brew install miniconda

2. Initialize conda for shell interaction:

   conda init

   If you're on a Mac, instead use:

   conda init zsh

   Then restart your terminal for the changes to take effect.

3. Create a project:

   conda create -n vision-quickstart

4. Start the virtual environment:

   conda activate vision-quickstart

5. Install the modular conda package:
   Nightly

   conda install -c conda-forge -c https://conda.modular.com/max-nightly/ modular

   Stable

   conda install -c conda-forge -c https://conda.modular.com/max/ modular

Serve your model

To get Llama 3.2 11B Vision Instruct, you must have a Hugging Face user access token and approved access to the Llama 3.2 11B Vision Instruct Hugging Face repo.

To create a Hugging Face user access token, see Access Tokens. Within your local environment, save your access token as an environment variable:

export HF_TOKEN="hf_..."

Use the max serve command to start a local model server with the Llama 3.2 Vision model:

max serve \
  --model meta-llama/Llama-3.2-11B-Vision-Instruct \
  --max-length 108172 \
  --max-batch-size 1

You may need to alter the --max-length and --max-batch-size parameters depending on the amount of memory you have access to. The above command is optimized for a p4d.24xlarge instance with one NVIDIA A100 GPU and 96 vCPUs.

This will create a server running the Llama-3.2-11B-Vision-Instruct multimodal model on http://localhost:8000/v1/chat/completions, an OpenAI compatible endpoint.

While this example uses the Llama 3.2 Vision model, you can replace it with any of the models listed in the MAX Builds site.

The endpoint is ready when you see this message printed in your terminal:

Server ready on http://0.0.0.0:8000 (Press CTRL+C to quit)

For a complete list of max CLI commands and options, refer to the MAX CLI reference.

Interact with your model

MAX supports OpenAI's REST APIs and you can interact with the model using either the OpenAI Python SDK or curl:

Python

You can use OpenAI's Python client to interact with the vision model. First, install the OpenAI API:

pixi

pixi add openai

uv

uv add openai

pip

pip install openai

conda

conda install openai

Then, create a client and make a request to the model:

generate-image-description.py

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="meta-llama/Llama-3.2-11B-Vision-Instruct",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What is in this image?"
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg"
                    }
                }
            ]
        }
    ],
    max_tokens=300
)

print(response.choices[0].message.content)

In this example, you're using the OpenAI Python client to interact with the MAX endpoint running on local host 8000. The client object is initialized with the base URL http://0.0.0.0:8000/v1 and the API key is ignored.

When you run this code, the model should respond with information about the image:

python generate-image-description.py

A rabbit is sitting in a field. It has long ears and a white belly. It is looking at the camera.

curl

You can send requests to the local endpoint using curl. The following request includes an image URL and a question to answer about the provided image:

curl -N http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
  "model": "meta-llama/Llama-3.2-11B-Vision-Instruct",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "What is in this image?"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg"
          }
        }
      ]
    }
  ],
  "max_tokens": 300
}' | grep -o '"content":"[^"]*"' | sed 's/"content":"//g' | sed 's/"//g' | tr -d '\n' | sed 's/\\n/\n/g'

This sends an object location to an image along with a text prompt to the model, and you should receive a response similar to this:

A rabbit is sitting in a field. It has long ears and a white belly. It is looking at the camera.

When making requests with max serve, you do not need to include model-specific image tags within your prompt.

For complete details on all available API endpoints and options, see the MAX Serve API documentation.

Next steps

Now that you have successfully set up MAX with an OpenAI-compatible endpoint, checkout out these other tutorials:

Was this page helpful?

Thank you! We'll create more content like this.

Thank you for helping us improve!

😔 What went wrong?

Some code doesn’t work

It includes inaccurate information

It's missing information I need

It was difficult to understand

Other

Submit