Sample continuations

Use POST /v1/completions to sample continuations of your prompts: set the sampling fields, ask for many choices at once, stream the tokens, and repeat a sample. Each example reads ACS_API_BASE and ACS_API_KEY, which you export in the Quickstart.

Set the sampling fieldsCopy Anchor Link

1. Choose temperature, from 0 to 100. 0 takes the likeliest token each time, and the default, 1, samples from the model's own probabilities. A higher value samples rarer tokens more often.
2. To sample only from the likeliest tokens, set top_p, or vLLM's top_k or min_p.
3. Set max_tokens to the most tokens for each choice, and stop to the texts that end a choice early.
4. Send the request.

curl "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 16, "temperature": 0.7, "top_p": 0.9, "stop": ["\n"]}'

A field that you leave out takes OpenAI's default. The model's own settings play no part, so a request samples in the same way on every model. How to sample lists each sampling field and its default.

Send several promptsCopy Anchor Link

1. Put the prompts in a list in prompt: texts, or lists of token ids.
2. Set n to the number of choices for each prompt.
3. Set max_tokens. A request with more than one prompt, or with n above 1, must set it.
4. Read choice i of prompt p at the index p * n + i.

Here, jq prints the index and the text of each choice:

curl -s "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-8b", "prompt": ["The sky is", "The grass is"], "max_tokens": 2, "n": 2}' \
  | jq -c '.choices[] | {index, text}'

{ "index": 0, "text": " blue." }
{ "index": 1, "text": " blue because" }
{ "index": 2, "text": " always gre" }
{ "index": 3, "text": " always gre" }

0 ' so dark'
1 ' the limit'
2 ' never gre'
3 ' always belonging'

Choices 0 and 1 continue "The sky is", and choices 2 and 3 continue "The grass is". The curl command and the Python program sampled apart, so their texts differ.

The usage adds up all the prompts and all the choices. One request takes one Hold for all its choices, and asks for at most 1,024 choices in all (Size caps). Sample many rollouts also sends many requests at once.

Send vLLM's fields from the openai SDKCopy Anchor Link

vLLM, the engine that runs the Base models in Inference, takes fields that OpenAI's API does not have, such as top_k, min_p, and repetition_penalty. Request fields marks each of them. The openai SDK has no argument for these fields, and raises TypeError if you pass one.

1. Put the vLLM fields in a dictionary.
2. Pass it as extra_body. The SDK adds its fields to the top level of the request body.

import os

from openai import OpenAI

client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
completion = client.completions.create(
    model="llama-8b",
    prompt="The sky is",
    max_tokens=2,
    temperature=0.7,
    extra_body={"top_k": 20, "min_p": 0.05, "repetition_penalty": 1.1},
)
print(completion.choices[0].text)

 the limit

With curl, put the same fields at the top level of the body. The API refuses a field named extra_body with 400 invalid_request.

curl "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "temperature": 0.7, "top_k": 20, "min_p": 0.05, "repetition_penalty": 1.1}'

Stream a CompletionCopy Anchor Link

1. Set stream: true. With curl, add -N, so that curl prints each event when it arrives.
2. To get the token counts, add "stream_options": {"include_usage": true}.
3. Read the data: events until data: [DONE].

curl -N "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "stream": true, "stream_options": {"include_usage": true}}'

The curl command prints the events:

data: {"id":"cmpl-a18dc61150cade3a","object":"text_completion","created":1791780733,"model":"meta-llama/Llama-3.1-8B","choices":[{"index":0,"text":" deep","logprobs":null,"finish_reason":null}]}

data: {"id":"cmpl-a18dc61150cade3a","object":"text_completion","created":1791780733,"model":"meta-llama/Llama-3.1-8B","choices":[{"index":0,"text":" and","logprobs":null,"finish_reason":"length"}]}

data: {"id":"cmpl-a18dc61150cade3a","object":"text_completion","created":1791780733,"model":"meta-llama/Llama-3.1-8B","choices":[],"usage":{"prompt_tokens":4,"completion_tokens":2,"total_tokens":6}}

data: [DONE]

The Python program sampled apart from the curl command. It prints the text as the tokens arrive, and then the token count:

 teeming
[tokens: 6]

Each chunk has the next text of one choice, and its index. With several prompts or choices, the chunks of the choices come mixed. finish_reason is null until the last chunk of a choice.

The last chunk of include_usage has "choices": [], so chunk.choices[0] would fail on it. The check if chunk.choices: skips that chunk, and the loop works with or without include_usage. Without it, the stream has no token counts, and your Hold still settles to the tokens that Inference wrote.

A stream that ends without data: [DONE] did not finish. If Inference fails after the stream starts, the last event before [DONE] is an error object, and the openai SDK raises openai.APIError (Inference fails during a stream). logprobs works in a stream. The API refuses prompt_logprobs in a stream with 400, and a Job cannot stream.

Stop a stream earlyCopy Anchor Link

1. Open the stream in a with block.
2. Leave the loop when you have the text that you need. The with block then closes the connection.

import os

from openai import OpenAI

client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
with client.completions.create(
    model="llama-8b", prompt="The sky is", max_tokens=200, stream=True
) as stream:
    for chunk in stream:
        if chunk.choices:
            print(chunk.choices[0].text, end="", flush=True)
            if "." in chunk.choices[0].text:
                break
print()

This program stops at the first token that has a full stop. When the connection closes, the API stops its request to Inference. You pay for the tokens that the stream sent until then, and the Hold gives back the rest (How a Hold ends). With curl, stop the command with Ctrl+C.

Get the same tokens againCopy Anchor Link

1. Set temperature to 0, or set a seed of 0 or more.
2. Send the same request again, with every other field the same, to the same model.

import os

from openai import OpenAI

client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
for _ in range(2):
    completion = client.completions.create(
        model="llama-8b", prompt="The sky is", max_tokens=8, seed=7
    )
    print(repr(completion.choices[0].text))

The two requests get the same tokens, except where vLLM's batching changes them. vLLM batches your request with other requests. The batch can change a log probability by a little, and now and then a token where two tokens are almost equally likely. To compare log probabilities between requests, send each request more than once, and read a small difference as no difference. Why repeats differ explains it.