Steer a Completion

Add a Steering vector to the residual stream of a Base model at the Layers you choose, to change what it writes. Read the steered Layers with Activations to see what your vector changed. How a Steering vector changes it explains the addition.

Send Steering vectors in the field apply_steering_vectors of POST /v1/completions. The field has the name that vllm-lens uses. The openai SDK does not know it, so the Python programs send it in extra_body. A steered Completion holds and spends tokens as any other Completion does.

The Python programs need Python 3.14 (for compression.zstd), NumPy, and the openai SDK. They decode the residual stream with the helper from Decode the tensor.

Write a Steering vectorCopy Anchor Link

A Steering vector is an object in the list apply_steering_vectors. Its activations are JSON numbers, or the same numbers encoded as in Encode a Steering vector. For example, {"activations": [vector.tolist()], "layer_indices": [1], "scale": 2} adds 2 times vector at Layer 1.

• activations has one row for each Layer in layer_indices. Row i goes to Layer layer_indices[i].
• A row is one vector of the model's hidden size. Inference adds it at every prompt token, and at every generated token that runs through the model.
• A row can also be a list of vectors, one for each position that it steers (Steer chosen positions).

GET /v1/models lists the hidden size as capabilities.hidden_size. meta-llama/Llama-3.1-8B has a hidden size of 4096, so each vector has 4096 numbers. To send a NumPy array, convert it with array.tolist(). The API sends the numbers to Inference as float32, so each number must be within float32's range.

Encode a Steering vectorCopy Anchor Link

Encode a large Steering vector to send it in about a tenth of the bytes of JSON numbers. The API checks an encoded vector by the same rules as numbers, and sends it to Inference as it is. This helper encodes a NumPy array of one row for each Layer, as float32:

import base64
from compression import zstd

import numpy as np

def encode(rows):
    array = np.asarray(rows, dtype=np.float32)
    return {
        "data": base64.b64encode(zstd.compress(array.tobytes())).decode(),
        "dtype": "float32",
        "original_dtype": "torch.float32",
        "shape": list(array.shape),
        "compression": "zstd",
    }

Send {"activations": encode([vector]), "layer_indices": [1], "scale": 2} in place of vector.tolist(). The API also takes bfloat16 numbers as int16 with original_dtype torch.bfloat16, as the old platform's encode_vector writes them. They take half the bytes of float32. shape is [layers, hidden size], or [layers, positions, hidden size] for a list of vectors for each Layer. The data must be one zstd frame that states its size, as zstd.compress writes it.

Steer and read in one requestCopy Anchor Link

This request adds 2 times a vector at Layer 1, and reads Layer 1 to show the change. The vector is [1, 2, 3] and then 4093 zeros:

import base64
import os
from compression import zstd

import numpy as np
from openai import OpenAI

def decode(tensor):
    raw = zstd.decompress(base64.b64decode(tensor["data"]))
    array = np.frombuffer(raw, dtype=tensor["dtype"]).reshape(tensor["shape"])
    if tensor["original_dtype"] == "torch.bfloat16":
        array = (array.view(np.uint16).astype(np.uint32) << 16).view(np.float32)
    return array

vector = np.zeros(4096)
vector[:3] = [1, 2, 3]

client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
completion = client.completions.create(
    model="llama-8b",
    prompt="The sky is",
    max_tokens=2,
    extra_body={
        "output_residual_stream": [1],
        "apply_steering_vectors": [{"activations": [vector.tolist()], "layer_indices": [1], "scale": 2}],
    },
)
residual_stream = decode(completion.activations["residual_stream"])
print(residual_stream.shape)
print(residual_stream[0, :, :4].tolist())

The program prints the shape of Layer 1, and the first 4 numbers of each token:

(1, 5, 4096)
[[2.015625, 4.0, 5.9375, -0.05419921875], [1.9765625, 4.0, 6.0, 0.0030517578125], [1.9921875, 4.0, 6.0, 0.0157470703125], [2.0, 4.0, 6.0, 0.0150146484375], [2.015625, 3.984375, 6.0, -0.01611328125]]

The read of a steered Layer includes the steer. Layer 1's own numbers are close to 0 here, so the first 3 numbers of each token are close to 2, 4, and 6. The fourth number stays small.

A request with Steering vectors can also stream, with or without output_residual_stream.

Compare with a zero-scale baselineCopy Anchor Link

Send the same request at scale: 0 as the baseline for a steer, not the request without Steering vectors. A request with Steering vectors takes another path through Inference, as Why compare with scale 0 explains.

Set temperature: 0 in both requests, so that sampling does not change the text between them. A difference between the two answers then comes from your vector. vLLM's batching can still change a token now and then (How to sample).

This program steers Layer 1 at scale 0 and at scale 2, and reads Layers 0 and 1 each time. It prints both texts, and the first 4 numbers of the difference at each token:

import base64
import os
from compression import zstd

import numpy as np
from openai import OpenAI

def decode(tensor):
    raw = zstd.decompress(base64.b64decode(tensor["data"]))
    array = np.frombuffer(raw, dtype=tensor["dtype"]).reshape(tensor["shape"])
    if tensor["original_dtype"] == "torch.bfloat16":
        array = (array.view(np.uint16).astype(np.uint32) << 16).view(np.float32)
    return array

vector = np.zeros(4096)
vector[:3] = [1, 2, 3]

client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
baseline = client.completions.create(
    model="llama-8b",
    prompt="The sky is",
    max_tokens=1,
    temperature=0,
    extra_body={
        "output_residual_stream": [0, 1],
        "apply_steering_vectors": [{"activations": [vector.tolist()], "layer_indices": [1], "scale": 0}],
    },
)
steered = client.completions.create(
    model="llama-8b",
    prompt="The sky is",
    max_tokens=1,
    temperature=0,
    extra_body={
        "output_residual_stream": [0, 1],
        "apply_steering_vectors": [{"activations": [vector.tolist()], "layer_indices": [1], "scale": 2}],
    },
)
print(repr(baseline.choices[0].text), repr(steered.choices[0].text))
difference = decode(steered.activations["residual_stream"]) - decode(baseline.activations["residual_stream"])
print(difference[:, :, :4].round(2).tolist())

' the' ' a'
[[[0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0]], [[2.0, 3.990000009536743, 6.019999980926514, 0.0], [2.0, 4.0, 6.010000228881836, 0.0], [2.0, 4.0, 6.0, 0.0], [2.0, 4.010000228881836, 6.010000228881836, 0.0]]]

Layer 0 comes before the steer, so it does not change. Layer 1 moved by 2 * [1, 2, 3] at each token, within bfloat16's precision, and its fourth number did not move. Here, the steer also changed the next token from " the" to " a".

Equal texts do not show that a steer ran. To see the steer, read the steered Layer as this program does.

Check a steer on your own vectorCopy Anchor Link

The API answers a steer at a Layer that you did not mean as it answers any other request. To check your vector v at Layer L, read Layers L - 1 and L, with max_tokens: 1 and temperature: 0. With more generated tokens, the steered text goes another way, and every later row differs for that reason alone.

1. Send scale: 0 as the baseline.
2. Send the same request with a scale other than 0 and norm_match: false. Layer L - 1 must match the baseline. Layer L must move by scale * v at each token.
3. Send the same request with norm_match: true. Subtract the baseline from each row of Layer L. The norm of each difference must be about scale times the norm of the baseline row.

The program in Compare with a zero-scale baseline does steps 1 and 2 for Layer 1.

Steer chosen positionsCopy Anchor Link

To steer some positions only, give each Layer a list of vectors in activations, one for each position, and name the positions in position_indices. A position counts from 0 at the first prompt token. On llama-8b, position 0 of a text prompt is the BOS token. Without position_indices, the vectors go to positions 0, 1, 2, and so on.

This program adds a vector at position 2 of Layer 1, first at scale 0 and then at scale 2. It reads Layers 0 and 1 each time, and prints the first 4 numbers of the difference at each token:

import base64
import os
from compression import zstd

import numpy as np
from openai import OpenAI

def decode(tensor):
    raw = zstd.decompress(base64.b64decode(tensor["data"]))
    array = np.frombuffer(raw, dtype=tensor["dtype"]).reshape(tensor["shape"])
    if tensor["original_dtype"] == "torch.bfloat16":
        array = (array.view(np.uint16).astype(np.uint32) << 16).view(np.float32)
    return array

vector = np.zeros(4096)
vector[:3] = [1, 2, 3]

def at_position_2(scale):
    return {"activations": [[vector.tolist()]], "layer_indices": [1], "position_indices": [2], "scale": scale}

client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
baseline = client.completions.create(
    model="llama-8b",
    prompt="The sky is",
    max_tokens=1,
    temperature=0,
    extra_body={"output_residual_stream": [0, 1], "apply_steering_vectors": [at_position_2(0)]},
)
steered = client.completions.create(
    model="llama-8b",
    prompt="The sky is",
    max_tokens=1,
    temperature=0,
    extra_body={"output_residual_stream": [0, 1], "apply_steering_vectors": [at_position_2(2)]},
)
difference = decode(steered.activations["residual_stream"]) - decode(baseline.activations["residual_stream"])
print(difference[:, :, :4].round(2).tolist())

Layer 0 comes before the steer, so it does not move. On Layer 1, only the row of position 2 moves, by 2 * [1, 2, 3] up to the rounding of bfloat16.

Steer a span of positionsCopy Anchor Link

To add one vector at every position of one or more spans, give one vector for each Layer and position_spans, a list of [start, end] pairs. A span covers start to end - 1. For example, {"activations": [vector.tolist()], "layer_indices": [1], "position_spans": [[2, 5]]} adds vector at positions 2, 3, and 4 of Layer 1.

Spans may come in any order, and may not overlap. They cover at most 4,096 positions in all. The API sends Inference one copy of each vector for each position, so the cap on numbers counts each position: Size caps.

Positions that a request reachesCopy Anchor Link

Inference adds a vector at a position when a forward pass runs that position. The forward passes run each prompt token, and each generated token but the last, which the model samples and never runs. So a request with a prompt of P tokens and max_tokens M reaches positions 0 to P + M - 2. It never reaches the last position of the model's context, max_model_len - 1.

The API answers 400 on apply_steering_vectors for a position after that, before it holds any tokens. It counts a text prompt's tokens as the Hold does, with the model's tokenizer (The Hold). Inference counts the tokens too, and refuses a position past them with the same error.

This program asks for position 5 with a prompt of 4 token ids and max_tokens: 2, so its passes reach position 4 at most:

import json
import os

from openai import BadRequestError, OpenAI

vector = [0.0] * 4096

client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
try:
    client.completions.create(
        model="llama-8b",
        prompt=[128000, 791, 13180, 374],
        max_tokens=2,
        extra_body={"apply_steering_vectors": [{"activations": [[vector]], "layer_indices": [1], "position_indices": [5]}]},
    )
except BadRequestError as error:
    print(json.dumps(error.response.json(), indent=2))

{
  "error": {
    "message": "apply_steering_vectors steers position 5, but the forward passes of this request reach position 4 at most: the prompt has 4 tokens and max_tokens is 2, and the last generated token never runs through the model. Steer a position up to 4, or raise max_tokens.",
    "type": "invalid_request_error",
    "param": "apply_steering_vectors",
    "code": "invalid_request"
  },
  "request_id": "req_9f2c4e1a7b3d4c5e8f6a0b1c2d3e4f5a"
}

A position that the request reaches is steered only when generation gets there. When a stop text or the end-of-text token ends the Completion first, its finish_reason is stop, and the later positions are not steered.

Steering vector fieldsCopy Anchor Link

Requests that the API refusesCopy Anchor Link

The API answers 400 invalid_request_error for these requests before it holds any tokens, and they do not reach Inference. param is apply_steering_vectors. If the request also has output_residual_stream, the errors for n and for several prompts name that field instead.

• An empty list of Steering vectors, n other than 1, or several prompts.
• More than 8 Steering vectors, or more than 2,097,152 numbers in all of them.
• Rows of activations with different lengths, or a number of rows that is not the number of layer_indices.
• A repeated or negative Layer.
• A Layer that the model does not have, or a vector whose length is not the model's hidden size. GET /v1/models lists both as capabilities.n_layers and capabilities.hidden_size.
• A number outside float32's range, from -3.4028235e+38 to 3.4028235e+38.
• An encoded vector whose data is not base64 or zstd, whose numbers are not float32 or bfloat16, or whose bytes do not fill its shape. Its numbers must be finite.
• A field that a Steering vector does not have, such as scael for scale.
• position_indices with one vector for each Layer, or with more or fewer positions than vectors.
• position_spans with position_indices, or with a list of vectors for each Layer.
• A span that does not end after it starts, spans that overlap, or spans of more than 4,096 positions in all.
• A position that no forward pass of the request reaches, as Positions that a request reaches says.

curl "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "apply_steering_vectors": [{"activations": [[1, 2, 3]], "layer_indices": [1, 2]}]}'

{
  "error": {
    "message": "activations needs one row for each layer in layer_indices.",
    "type": "invalid_request_error",
    "param": "apply_steering_vectors",
    "code": "invalid_request"
  },
  "request_id": "req_9f2c4e1a7b3d4c5e8f6a0b1c2d3e4f5a"
}

The whole request body is also capped, and a body over the cap gets 413 before the API reads the vectors. Size caps lists both caps.

Inference answers only for a served model that GET /v1/models does not list. For such a model, Inference checks the Layers and the hidden size itself, and the API passes its 400 on.

Known limitsCopy Anchor Link

• Vectors that steer the same Layer add up.
• If steering fails inside Inference, every request with Activations or Steering vectors in the same batch fails with 500.

Next stepsCopy Anchor Link

Steer with a contrast vector builds a Steering vector from the Activations of two sets of prompts, and steers with it at a few scales. Ask for Activations in a Job sends many steered requests in the background.