Offline inference
Offline inference with MAX allows you to run large language models directly in Python without relying on external API endpoints. This is in contrast to online inference, where you would send requests to a remote service.
When to use offline inference
You'll want to use offline inference in scenarios where you want to perform model inference without the need for a separate model inference server. Typically this includes where you have to process a batch of inputs concurrently.
This approach is beneficial for tasks that require high throughput and can be executed in a controlled environment, such as data preprocessing, model evaluation, or when working with large datasets that need to be processed in batches.
How offline inference works
The core of offline inference revolves around the the
LLM
class which provides
a Python interface to load and run language models.
Specify the model from a Hugging Face repository or a local path and MAX handles
the process of downloading the model. The
PipelineConfig
class allows you to specify parameters related to the inference pipeline, such as
max_length
and
max_num_steps
.
The generate()
function is used to generate text from the model.
Quickstart
This quickstart demonstrates how to use offline inference using a Hugging Face model with MAX in Python.
-
Set up your project:
- pip
- uv
- magic
-
Create a project folder:
mkdir quickstart && cd quickstart
mkdir quickstart && cd quickstart
-
Create and activate a virtual environment:
python3 -m venv .venv \
&& source .venv/bin/activatepython3 -m venv .venv \
&& source .venv/bin/activate -
Install the
modular
Python package:- Nightly
- Stable
pip install modular \
--index-url https://download.pytorch.org/whl/cpu \
--extra-index-url https://dl.modular.com/public/max-nightly/python/simple/pip install modular \
--index-url https://download.pytorch.org/whl/cpu \
--extra-index-url https://dl.modular.com/public/max-nightly/python/simple/pip install modular \
--index-url https://download.pytorch.org/whl/cpupip install modular \
--index-url https://download.pytorch.org/whl/cpu
-
Install
uv
:curl -LsSf https://astral.sh/uv/install.sh | sh
curl -LsSf https://astral.sh/uv/install.sh | sh
Then restart your terminal to make
uv
accessible. -
Create a project:
uv init quickstart && cd quickstart
uv init quickstart && cd quickstart
-
Create and start a virtual environment:
uv venv && source .venv/bin/activate
uv venv && source .venv/bin/activate
-
Install the
modular
Python package:- Nightly
- Stable
uv pip install modular \
--index-url https://download.pytorch.org/whl/cpu \
--extra-index-url https://dl.modular.com/public/max-nightly/python/simple/uv pip install modular \
--index-url https://download.pytorch.org/whl/cpu \
--extra-index-url https://dl.modular.com/public/max-nightly/python/simple/uv pip install modular \
--index-url https://download.pytorch.org/whl/cpuuv pip install modular \
--index-url https://download.pytorch.org/whl/cpu
-
Install
magic
:curl -ssL https://magic.modular.com/ | bash
curl -ssL https://magic.modular.com/ | bash
Then run the
source
command that's printed in your terminal. -
Create a project:
magic init quickstart --format pyproject && cd quickstart
magic init quickstart --format pyproject && cd quickstart
-
Install the
max-pipelines
conda package:- Nightly
- Stable
magic add max-pipelines
magic add max-pipelines
magic add "max-pipelines==25.3"
magic add "max-pipelines==25.3"
-
Start the virtual environment:
magic shell
magic shell
-
Create a file named
main.py
with the following code:from max.entrypoints.llm import LLM
from max.pipelines import PipelineConfig
from max.serve.config import Settings
def main():
model_path = "modularai/Llama-3.1-8B-Instruct-GGUF"
print(f"Loading model: {model_path}")
# Configure the inference pipeline
pipeline_config = PipelineConfig(model_path=model_path)
settings = Settings()
# Initialize the LLM engine
llm = LLM(settings, pipeline_config)
prompts = [
"In the beginning, there was",
"I believe the meaning of life is",
"The fastest way to learn python is",
]
print("Generating responses...")
responses = llm.generate(prompts, max_new_tokens=50)
for i, (prompt, response) in enumerate(zip(prompts, responses)):
print(f"========== Response {i} ==========")
print(prompt + response)
print()
if __name__ == "__main__":
main()from max.entrypoints.llm import LLM
from max.pipelines import PipelineConfig
from max.serve.config import Settings
def main():
model_path = "modularai/Llama-3.1-8B-Instruct-GGUF"
print(f"Loading model: {model_path}")
# Configure the inference pipeline
pipeline_config = PipelineConfig(model_path=model_path)
settings = Settings()
# Initialize the LLM engine
llm = LLM(settings, pipeline_config)
prompts = [
"In the beginning, there was",
"I believe the meaning of life is",
"The fastest way to learn python is",
]
print("Generating responses...")
responses = llm.generate(prompts, max_new_tokens=50)
for i, (prompt, response) in enumerate(zip(prompts, responses)):
print(f"========== Response {i} ==========")
print(prompt + response)
print()
if __name__ == "__main__":
main()This script downloads the
modularai/Llama-3.1-8B-Instruct-GGUF
model (if not already downloaded) and then run inference locally. While the initial model download requires internet access, the actual inference process is self-contained and does not send requests to a remote service for generating text.You can update the script to use a different model or modify the prompts to generate different responses. For a list of available models, see our Model repository. We chose the Llama-3.1-8B-Instruct-GGUF model for this example because it's not gated, meaning it's freely available without requiring special access permissions or authentication.
For offline inference, MAX supports models in GGUF format. This includes most generative LLMs with "Chat" modality, but the specific configuration parameters might vary between models. Always refer to the model's documentation for compatibility details and optimal configuration settings.
-
Run the script:
python main.py
python main.py
This command will download the model and generate responses for the prompts.
You should see output like the following:
Generating responses...
========== Response 0 ==========
In the beginning, there was Andromeda. The Andromeda galaxy, that is. It's the
closest major galaxy to our own Milky Way, and it's been a source of fascination
for astronomers and space enthusiasts for centuries. But what if I told you that
there's
========== Response 1 ==========
I believe the meaning of life is to find your gift. The purpose of life is to give it away to others.
I believe that the meaning of life is to find your gift. The purpose of life is to give it away to others.
I believe that the meaning of life is
========== Response 2 ==========
The fastest way to learn python is to practice with real-world projects. Here are
some ideas for projects that you can use to learn Python:
1. **Command Line Calculator**: Create a command line calculator that can perform
basic arithmetic operations like addition, subtraction, multiplication, and
division.Generating responses...
========== Response 0 ==========
In the beginning, there was Andromeda. The Andromeda galaxy, that is. It's the
closest major galaxy to our own Milky Way, and it's been a source of fascination
for astronomers and space enthusiasts for centuries. But what if I told you that
there's
========== Response 1 ==========
I believe the meaning of life is to find your gift. The purpose of life is to give it away to others.
I believe that the meaning of life is to find your gift. The purpose of life is to give it away to others.
I believe that the meaning of life is
========== Response 2 ==========
The fastest way to learn python is to practice with real-world projects. Here are
some ideas for projects that you can use to learn Python:
1. **Command Line Calculator**: Create a command line calculator that can perform
basic arithmetic operations like addition, subtraction, multiplication, and
division.
Next steps
For more information on offline inference, see the following:
Was this page helpful?
Thank you! We'll create more content like this.
Thank you for helping us improve!