Errors

Every error of the /v1 routes comes as the error object of the old platform's API, with its status codes and its code values. Code written for that API reads these errors unchanged. The openai SDK raises its usual error classes. Retry requests safely says which errors to try again.

The error objectCopy Anchor Link

Every error answer has the error object, and the id of the request next to it:

{
  "error": {
    "message": "The model `gpt-2` does not exist. The API serves llama-8b, llama-405b, trinity-truebase.",
    "type": "invalid_request_error",
    "param": "model",
    "code": "model_not_found"
  },
  "request_id": "req_9f2c4e1a7b3d4c5e8f6a0b1c2d3e4f5a"
}

Every answer, an error too, has the header X-Request-Id, for example X-Request-Id: req_9f2c4e1a7b3d4c5e8f6a0b1c2d3e4f5a. The id is new for each request. An error that comes after the answer started, as the last event of a stream, has the id in request_id and in error.request_id. The API logs each request with its id, so send the id when you report a bug.

Status codesCopy Anchor Link

The API answers most errors itself. The rest come from Inference, and the API names them by what Inference said.

Answers from the APICopy Anchor Link

Answers about InferenceCopy Anchor Link

Each of these has the type upstream_error, except the two that Inference passes on as they are. A stream that has its 200 gets each of them as its last event (Inference fails during a stream).

Not enough tokensCopy Anchor Link

Before POST /v1/completions sends a request to Inference, it takes a Hold of the most tokens that the request can use. Near the end of your tokens, it lowers max_tokens to what fits (Near the end of your tokens). It answers 429 budget_exceeded only when your Budget cannot hold the prompt and one token for each choice.

A Job, a Loom Branch, and the Playground do not lower a max_tokens that you set. A Completion of theirs that your Budget cannot hold fails with budget_exceeded (Holds of background work). A harvest that your tokens cannot hold gets 429 budget_exceeded when you submit it. If your tokens run short before it starts, it fails with the harvest code insufficient_quota.

The message says how many tokens the request can use, and which Budget is short. For example, the prompt "The sky is" with "max_tokens": 2 can use 13 tokens, as the Hold counts them:

• This request can use up to 13 tokens, more than your token budget has left. Ask for more tokens at https://v2-staging.infra.acsresearch.org/usage/request.
• This request can use up to 13 tokens, more than this API key's token budget has left. when the key has its own Budget.
• When your monthly allowance is used up too, the message says so, and gives the day in UTC on which it refills.

The address opens the App's form to ask for more tokens (Ask for more tokens).

Handle the end of your Budget handles the error in code.

Unknown fieldCopy Anchor Link

The API refuses a field that it does not know, so a misspelled field cannot change a result without a warning. An unknown field inside an object, such as a Steering vector, gets 400 invalid_request with param set to the top-level field. Its message names where the field is, for example Unknown parameter: 'apply_steering_vectors[0].scael'.

curl "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-8b", "prompt": "The sky is", "temprature": 0}'

{
  "error": {
    "message": "Unknown parameter: 'temprature'.",
    "type": "invalid_request_error",
    "param": "temprature",
    "code": "invalid_request"
  },
  "request_id": "req_9f2c4e1a7b3d4c5e8f6a0b1c2d3e4f5a"
}

Unknown modelCopy Anchor Link

The API answers a model that it does not serve itself, from its list of served models, before it holds any tokens. A Job or a Loom Branch with such a model gets 400 model_not_found too, and the API makes nothing.

curl "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-2", "prompt": "The sky is"}'

{
  "error": {
    "message": "The model `gpt-2` does not exist. The API serves llama-8b, llama-405b, trinity-truebase.",
    "type": "invalid_request_error",
    "param": "model",
    "code": "model_not_found"
  },
  "request_id": "req_9f2c4e1a7b3d4c5e8f6a0b1c2d3e4f5a"
}

Prompt too longCopy Anchor Link

The prompt and max_tokens together must fit the model's capabilities.max_model_len from GET /v1/models. In a request of several prompts, each prompt must fit with max_tokens on its own. trinity-truebase has a max_model_len of 8,192 tokens. The prompt "The sky is" has 4 tokens with the BOS token, so "max_tokens": 8190 does not fit:

curl "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "trinity-truebase", "prompt": "The sky is", "max_tokens": 8190}'

The API refuses the request before it holds any tokens, and answers 400 with param set to prompt. In a request of several prompts, the message names the prompt that does not fit, such as prompt[1]:

{
  "error": {
    "message": "This model's maximum context length is 8192 tokens, and the prompt has 4 tokens and max_tokens is 8190: 8194 in all. Shorten the prompt or lower max_tokens.",
    "type": "invalid_request_error",
    "param": "prompt",
    "code": "context_length_exceeded"
  },
  "request_id": "req_9f2c4e1a7b3d4c5e8f6a0b1c2d3e4f5a"
}

The API counts a prompt's tokens as the Hold counts them. It sends a text that is too long for its tokenizer to read on to Inference. Inference refuses such a text with the same code, in vLLM's words.

Inference failsCopy Anchor Link

When Inference fails, the API answers with the type upstream_error, a code that names the failure, and retryable, as Answers about Inference lists. The API does not try a failed request again.

After Inference fails 5 requests to a model in a row, the API sends that model no request for 60 seconds. Each request to the model in that time gets 503 circuit_open at once, and its Retry-After header and retry_after_seconds give the seconds left. Then the next requests try the model again. One that succeeds ends the pause, and one that fails starts it again. A Job's request, a Run, or a Branch of that model fails at once with the same message. Other models are not affected.

A Completion without a stream that runs for more than 150 seconds already has its 200 status. A failure after that comes as the body, in the error object. The openai SDK raises no exception for it, and returns a Completion whose error field holds the error object. Requests without a stream explains the wait.

Inference fails during a streamCopy Anchor Link

A stream has its 200 status when Inference starts its answer. If Inference has not answered after 2 seconds, as while a cold model starts, the stream gets its 200 then (Requests with a stream). Each later error, such as an answer about Inference, is the last event before [DONE]:

data: {"error":{"message":"<Inference's message>","type":"upstream_error","param":null,"code":"upstream_server_error","retryable":false,"request_id":"req_9f2c4e1a7b3d4c5e8f6a0b1c2d3e4f5a"},"request_id":"req_9f2c4e1a7b3d4c5e8f6a0b1c2d3e4f5a"}

data: [DONE]

The openai SDK raises openai.APIError at the error event, after the chunks that came before it. The tokens before the error are real, and your Hold settles to the tokens that Inference used.

A stream from Inference is complete only with its last event. If it ends before that, for example when a connection closes, the API sends an error event before [DONE]. Its message is Inference ended the stream before it was complete., its code is upstream_server_error, and retryable is false. A new try runs the whole request again and pays again for each token that it streams. A cut stream therefore never looks complete. A Loom Node fails in the same way, and keeps its text so far.

Failed harvestsCopy Anchor Link

A harvest that fails after you submit it answers no error status. GET /v1/harvest/{job_id} answers the Job with status failed, and its error is a message that says why. Handle a failed harvest lists each cause and what to do.

A Job whose request fails answers its error at GET /v1/completion-jobs/{job_id}/result, with the status that POST /v1/completions would have answered (Fetch the result). Its request_id is the Job's request_id, not the X-Request-Id of that answer.

Errors in the openai SDKCopy Anchor Link

The openai SDK raises the error class that matches the status:

Each error class of a status is a kind of openai.APIStatusError. Its status_code is the status, body holds the error object, and code, param, and type are the fields of the error object. Its request_id is the X-Request-Id header.

import os

import openai
from openai import OpenAI

client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
try:
    client.completions.create(model="gpt-2", prompt="The sky is")
except openai.BadRequestError as error:
    print(error.status_code, error.code, error.param)

400 model_not_found model

The SDK sends some requests again before it raises the error (Retries in the openai SDK).