Quickstart

In this tutorial, you send one prompt to a Base model with curl and with Python. By the end, you will have read the tokens that it wrote, with the log probability of each, and streamed them as they arrive.

Set upCopy Anchor Link

1. Apply with the Apply form. When an admin approves your Application, the approval email lets you set your password. Get access has each step.

2. With your password set, sign in to the App and make an API key, as Make an API key shows.

3. Put the key and the API's address in shell variables. The curl commands and the Python programs in these docs read both:

   Copy Text

   export ACS_API_KEY=sk-acs-...
   export ACS_API_BASE=https://api.v2-staging.infra.acsresearch.org/v1

4. Install the openai Python SDK, which the Python programs use:

   Copy Text

   pip install openai

Send a promptCopy Anchor Link

Send the prompt "The sky is" to llama-8b, Meta's Llama 3.1 8B. Ask for at most 2 tokens, and for the log probability of each token with logprobs. Run the curl command, the Python program, or both:

curl "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "logprobs": 1}'

The curl command prints the whole Completion:

{
  "id": "cmpl-946f88cbcf513663",
  "object": "text_completion",
  "created": 1791780418,
  "model": "meta-llama/Llama-3.1-8B",
  "choices": [
    {
      "index": 0,
      "text": " dark with",
      "logprobs": {
        "text_offset": [0, 5],
        "token_logprobs": [-3.864488124847412, -3.125293254852295],
        "tokens": [" dark", " with"],
        "top_logprobs": [
          { " dark": -3.864488124847412, " the": -1.4269880056381226 },
          { " with": -3.125293254852295, " and": -1.5002931356430054 }
        ]
      },
      "finish_reason": "length",
      "prompt_logprobs": null
    }
  ],
  "usage": { "prompt_tokens": 4, "completion_tokens": 2, "total_tokens": 6 }
}

In the Completion, you will see these fields:

• choices[0].text is the text that the model wrote after your prompt.
• choices[0].logprobs.tokens lists each token, and token_logprobs has the log probability of each.
• top_logprobs has the likeliest tokens at each position. logprobs: 1 asks for 1 of them, and the token that the model wrote is there too.
• finish_reason is "length", because the model stopped at max_tokens.
• usage counts the tokens of the prompt and of the text. The prompt has 4 tokens, because Llama's tokenizer adds <|begin_of_text|> in front of it.

The Python program prints the text and the log probability of each token. The model samples each token, so the Python run is a separate sample, and its tokens differ from those of the curl run:

 a beautiful
[-3.176988124847412, -3.1364614963531494]

The model continued your text. A Base model does not answer a prompt as a question, and the API applies no chat template. Base models explains why.

Stream the tokensCopy Anchor Link

Send the same prompt with stream: true. The API sends each token as a Server-Sent Event when Inference makes it:

curl "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "stream": true}'

The curl command prints one event for each token, then [DONE]. The last token's event has the finish_reason:

data: {"id":"cmpl-9a33ca90561191f8","object":"text_completion","created":1791780425,"model":"meta-llama/Llama-3.1-8B","choices":[{"index":0,"text":" again","logprobs":null,"finish_reason":null}]}

data: {"id":"cmpl-9a33ca90561191f8","object":"text_completion","created":1791780425,"model":"meta-llama/Llama-3.1-8B","choices":[{"index":0,"text":" filled","logprobs":null,"finish_reason":"length"}]}

data: [DONE]

The Python program prints the text as the tokens arrive. It is another separate sample:

 usually the

You have sent a prompt to a Base model, read its tokens with their log probabilities, and streamed them.

Next stepsCopy Anchor Link

Your first steered Completion reads the model's Activations and changes what it writes with a Steering vector. Sample continuations sets the sampling fields, sends several prompts at once, and handles streams. Read token probabilities gets the log probabilities of your prompt and of the whole vocabulary. Jobs runs many requests in the background and keeps the results.