FAQ
Each answer is short and links to the page that explains it. If your question is not here, write to infra@acsresearch.org, as Get help says.
Why does the model not answer my question?
Every model is a Base model, which continues your text and does not answer it, and the API applies no chat template. What a Base model does explains how to write a prompt for it.
Why does my output differ from the output in the docs?
A Base model samples each token, so each run of a request writes other text, and each output in the docs is one sample. Why repeats differ explains it.
How do I get the same text twice?
Set temperature to 0, or send the same seed to the same model. vLLM's batching can still change a token now and then, as Get the same tokens again explains.
Can I use the openai SDK?
Yes, with client.completions.create(...), since the API follows OpenAI's completions API. The API serves no chat, as No chat template explains.
How do I send vLLM's fields from the openai SDK?
Put them in extra_body, because the SDK raises TypeError for an argument that it does not know. Send vLLM's fields shows an example.
How do I get the whole distribution over the vocabulary?
Set vLLM's prompt_logprobs to -1, within the size caps for one prompt. Get the whole vocabulary shows it, and how to get it for generated tokens too.
Why does a misspelled field get an error?
The API refuses a field that it does not know with 400 invalid_request, so that a typo cannot change a result without a warning. Why the API is strict explains it.
Why did my Completion stop before max_tokens?
With finish_reason "stop", the model wrote its end-of-text token or a stop text. With "length", POST /v1/completions may have lowered max_tokens to fit your tokens, as Near the end of your tokens explains.
Why did I get a 429 error?
The code tells you why:
budget_exceeded: your tokens cannot hold even one token for each choice.rate_limited: too many requests.spend_cap: new GPU work is paused (Spend limit).
Handle the 429 errors tells them apart in code.
How do I get more tokens?
Ask for them on the App's Request more tokens page, under Usage. An admin approves the request with a Grant, or rejects it with a note. Ask for more tokens says how.
Why was my first request slow?
The model was not Warm, so the request waited while Inference started a container for it. Warm and cold models explains it.
What happens to my Job when my script stops?
The Job keeps running, because it runs in the API and not in your script. What runs in the background explains it.
How do I revoke an API key?
On the App's API Keys page, open the menu at the end of the key's row and select Revoke. Scripts that send the key then get 401 at once. Revoke a key has the steps.
Can I share my API key?
No, because your access is personal. Your access is personal says where to point others.