Work in the background
Jobs, harvests, Loom Nodes, and Playground Runs run inside the API, not in your script or your browser, so they finish with no client connected. A few rules decide when each one runs, and what happens to it when the API restarts.
What runs in the background
When you submit a Job or a harvest, grow a Branch, or start a Run, the API saves the work and answers at once. The API's Runner then runs the work to its end. Your script can stop, your connection can drop, and you can close the tab. The work goes on. You read its result later.
Four kinds of work run this way:
- A Job is one Completion request. The API runs it and keeps its result.
- A harvest reads the residual stream of many prompts. The API starts it on Inference's GPUs, follows it until it ends, and gives you links to its files.
- A Loom Node is one Completion of the text on the path to its parent. It runs as a stream, so its text grows while Inference writes it.
- A Playground Run is one Completion that you start in the App. The API keeps it until you delete it.
A request to POST /v1/completions is not background work. Your client waits for its answer.
Each Job, each harvest Job, each Node, and each Run takes its own Hold when it starts to run. Holds of background work explains how.
The order of the work
The API runs up to 4 Jobs at once. A Job stays queued until it starts.
When more Jobs wait than can run, the API starts the oldest Job of a User who has no Job running. It starts a Job of a User who already has one running only when no other User's Job waits. So one User's long Jobs do not keep another User waiting long. A Job of yours can wait behind newer Jobs of other Users while another of your Jobs runs.
Loom Nodes run apart from Jobs, up to 8 at once, the oldest first. A long Job never delays a Branch. Harvests run apart from both, the oldest first. A harvest waits for no Job, and runs on GPUs of its own. Playground Runs also run apart from both, so a Run starts while your Jobs run. Inference then batches the requests that reach it at the same time.
Work whose API key is paused waits, and the API runs the work behind it. Jobs, Harvests, and Looms list the numbers.
When the API restarts
The API restarts when a new version of it is deployed. It can also stop without warning, for example when its process crashes. Jobs, harvests, and Nodes go on after both. Each request of a Job, and each harvest, spends tokens once.
Before a planned restart, the API gives running work a short time to end. Work that does not end in time is stopped and goes back to queued. The API releases its Hold, so the cut request costs nothing. When the API runs again, the work runs again:
- A Job whose request already has its answer keeps that answer and does not run again. A request that the restart cut off runs again.
- A harvest keeps its GPU run, which goes on while the API restarts. The API follows the same GPU run again, and starts no second one.
- A Node starts again from the beginning, with no text.
- A Run starts again from the beginning.
While new GPU work is paused (Spend limit), the work that went back to queued waits, and runs when the pause ends.
When the API stops without warning, its running work shows running for some minutes. Then the API finds that the process is gone. It releases the Holds of that work, and queues each Job, harvest, and Node again, as after a planned restart. A Playground Run fails instead and keeps the text that it had, so you see what it wrote before the API stopped. It costs nothing.
Paused keys and suspended access
Background work runs with the API key that made it. When you pause that key, its work waits. Work that has not started stays queued, and holds no tokens. A running Job finishes the request that is in Inference, and then goes back to queued. A running harvest runs to its end. When you resume the key, the work runs from where it stopped, and each request still pays once.
A paused key gets 401 for every call, so it cannot read its own Jobs and Looms while it is paused. A revoke stops the work for good. Each of its Completions that has not run fails with 401 invalid_api_key, and spends no tokens. Each of its harvests that has not started fails with error.code unauthenticated. Playground Runs use no API key, so a paused or revoked key does not touch them.
When an admin suspends your access, none of your work runs. Each Completion that waits fails with 401 and the message Your access to ACS Infra is suspended. Write to infra@acsresearch.org if you think this is a mistake., and spends no tokens. Work that failed while your access was suspended does not run again when an admin restores it. If your access is suspended says what else changes.
Next steps
Run a Job submits a request and fetches its result. Grow a Loom with the API branches under a Node and follows the Nodes. Find your Runs again opens the Runs that the API kept. Manage your keys pauses and resumes a key.