Deploy and chat with your model

Once a training run completes, you can serve the model on the machine that trained it, or on any machine you've connected, and talk to it from the app or from your own code on your local network. The model is served on your hardware only.

Before you start

The run is completed. Running, failed or stopped runs can't be deployed.
The machine is Online in Settings, meaning it has checked in within the last 10 minutes.
One of its GPUs is in the inference pool. Open Settings, find the Connect infrastructure card, open the machine's GPU map, and set a card to Inference pool. A newly connected machine has no inference card until you do this. Training keeps using the other cards; a card is never shared between the two.
There's enough disk space. The first time any model is served on a machine, the serving software installs there and needs about 12 GB free. A model that isn't already on the machine (one trained in the cloud, or on another machine) downloads first and needs about 10 GB free.
Quantization was chosen at training time. The training form's Serving quantization field defaults to Auto, which keeps full precision when the model fits the card with room to spare and otherwise picks the best fit. If you chose FP8, the inference card has to be an RTX 40-series or newer.

Deploy a model

Either:

On the run's page, once it's completed, click Deploy to followed by the machine's name.
Or on Models, under Trained Checkpoints, click Deploy on the model's row. A model trained in the Hugging Face cloud can be served on your machine too; it downloads there the first time. If you have more than one machine, the Models page asks Where should it run? so you can pick.

You'll see Deploy Requested, and the machine picks the request up on its next check-in, about 20 seconds later.

A machine serves one model at a time. Deploying a different run to a machine that's already serving replaces the current model, with no extra confirmation. Deploying the same run again changes nothing.

While it starts up

The Connect infrastructure card shows a serving row for the machine, and once the model is up the Models page shows Serving on followed by the machine's name in its Status column. The row moves through:

Deploy requested
Downloading model, only when the files aren't already on the machine
Starting vLLM
Serving

Starting usually takes a minute or two. The first time a machine serves anything, the serving software installs first, which can take 10 to 15 minutes. If the model hasn't come up after 15 minutes the row reads Serving failed with the reason, and the machine keeps retrying on its own.

Chat in the app

Open Chat. In the Fine-Tuned Model dropdown, the On your hardware group lists the models your machines are serving right now. From a run's page, Test in Chat opens the chat with that model already selected.

Your messages travel to the machine through Azul and the answer streams back the same way. The model itself only ever runs on your hardware.

A few limits apply: an answer can take up to 4 minutes before the chat gives up; a machine handles a few chats at a time and asks you to retry when it's busy; the app asks for up to 1024 tokens per reply; and knowledge collections can't be attached to these chats yet. If the machine stops serving, the chat shows Workstation not serving and won't send until it's back.

Use the model from your own code

The machine runs a standard OpenAI-compatible server, with no API key.

By default it listens only on the machine itself, on port 8000. To reach it from other computers on your network, set AZUL_SERVE_BIND=0.0.0.0 in ~/azul-workstation/.env on the machine and run the connect command again. Settings then shows the address with a copy button while the model is healthy. It looks like http://MACHINE-ADDRESS:8000/v1.

The model name to send is the Name shown on Models under Trained Checkpoints.

curl http://MACHINE-ADDRESS:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "MODEL-NAME", "messages": [{"role": "user", "content": "Hello"}]}'

GET /v1/models lists what's being served. This is plain HTTP on your local network, meant for your own machines. Don't expose it to the internet.

Stop serving

In Settings, the machine's serving row has a Stop button. Anything using the address loses access, and the machine stops within about 20 seconds.

The card stays in the inference pool, so training still won't use it. To train on it again, move it back to Training pool in the GPU map, or choose Use every GPU for training in the training form, which also stops serving. Training never takes a card that's serving a model; if every card is serving, training runs wait.

If the machine goes offline

The deployment stays in place. After 10 minutes without contact the machine shows Offline; after 15 minutes the serving row reads Serving (stale report) and the chat refuses to send. When the machine comes back, serving restarts on its own. Agent updates restart the served model briefly too.

Not yet supported

Comparing a model served on your machine against its base model side by side.
Scoring a model served on your machine with the evaluation judge.
Knowledge collections in chats with these models.