Contrastive Language Models (CLM)

1. The shortest useful mental model

CLM is “CLIP for decisions.” Instead of generating the next action token-by-token, it embeds the current state and each candidate action into a shared vector space, then chooses the action whose vector best matches the state.

2. Why CLM exists

Many agent loops contain tiny decisions that do not warrant a full deliberative language-model call: choose a tool, route a ticket, rank several candidate patches, click one link, classify urgency, or decide whether a condition is true.

A conventional generative LLM typically reads the state and all options together, runs the transformer, and may still decode an answer token-by-token. CLM asks a more specialized question:

This is especially attractive when the candidate set is stable and reusable, such as an agent’s tool catalog. Actions can be embedded once and cached; only the changing state needs a fresh forward pass.

3. Architecture: frozen language model + two learned heads

The released CLM-8B uses a frozen Qwen3-8B encoder with last-token pooling. The backbone is not updated during CLM training. Two relatively small learned projection heads transform backbone embeddings into a state space and an action space that are trained to align.

The official repository describes each encoder as a frozen LLM backbone plus a roughly 20M-parameter trainable projection head. Because the backbone stays fixed, large embedding datasets can be precomputed and head training becomes much cheaper than full-model fine-tuning.

4. Mathematics: bidirectional InfoNCE

Suppose a minibatch contains \(B\) matched state-action pairs:

Construct the full \(B\times B\) similarity matrix:

where \(\tau\) is a learned or configured temperature controlling softmax sharpness.

The CLM objective performs retrieval in both directions: state → action and action → state:

How to read the bidirectional objective

Start with one training example \(i\). Its state embedding \(s_i\) and action embedding \(a_i\) are a matched pair: \(a_i\) is the action that belongs with \(s_i\). The dot product \(s_i^\top a_j\) is the model's compatibility score between state \(i\) and candidate action \(j\). Dividing by the temperature \(\tau\) gives the logit used by the softmax:

A smaller \(\tau\) makes score differences matter more, producing a sharper softmax; a larger \(\tau\) makes the distribution flatter.

First term: retrieve the correct action from a state

Hold the state \(s_i\) fixed and compare it with every action in the minibatch. The numerator is the exponentiated score of the correct action \(a_i\). The denominator contains the scores of all candidate actions \(a_1,\ldots,a_B\). This is simply a softmax probability:

If the correct action is already much closer to the state than all other actions, \(P(a_i\mid s_i)\approx 1\), so \(-\log P(a_i\mid s_i)\approx 0\). If a wrong action scores as highly as—or higher than—the correct action, the correct-action probability falls and the loss rises. Minimizing this term therefore pushes \(s_i\) toward \(a_i\) and away from competing actions.

Second term: retrieve the correct state from an action

This is the same retrieval problem with the direction reversed. Hold action \(a_i\) fixed and ask: which state in the minibatch belongs with this action?

The first term makes each row of the similarity matrix identify the correct action; the second makes each column identify the correct state. Training both directions prevents the geometry from being optimized only for one-way retrieval.

Why the logarithm, minus sign, and averaging appear

Each bracket contains two log-probabilities. Because probabilities lie between 0 and 1, their logarithms are non-positive. The leading minus sign converts “make the correct probability large” into a loss that we minimize:

So the loss penalizes the model increasingly strongly when it assigns little probability to the correct match. The factor \(1/(2B)\) simply averages over the \(B\) examples and the two retrieval directions.

Why in-batch negatives are powerful

For pair \(i\), \(a_i\) is the positive action. Every other \(a_j\) in the batch is automatically treated as a negative for \(s_i\). So one batch of \(B\) positive pairs implicitly provides \(B(B-1)\) mismatched comparisons without manually labeling each wrong answer.

For example, with \(B=1024\), each state sees one positive action and 1,023 in-batch negative actions. The same batch simultaneously supplies the reverse action-to-state comparisons. Larger batches therefore give the contrastive objective a richer set of alternatives to discriminate among.

Hard negatives

Ordinary in-batch negatives can be too easy: many may have little semantic relationship to the state. The more useful failure cases are candidates that are plausible and nearby in meaning, but still wrong. CLM therefore introduces explicit hard negatives \(h_{ik}^{(a)}\), where \(k\) indexes hard negative actions constructed for state \(s_i\):

How to read the hard-negative objective

Again, focus on a single state \(s_i\). The numerator is the score of its correct action \(a_i\). The denominator contains that correct action plus a deliberately selected set of confusing alternatives:

Each term has a direct interpretation:

• \(s_i^\top a_i\): compatibility of the state with the correct action.
• \(h_{ik}^{(a)}\): the \(k\)-th hard negative action for that state.
• \(s_i^\top h_{ik}^{(a)}\): how attractive that wrong-but-plausible action currently looks to the model.
• \(\sum_k \exp(s_i^\top h_{ik}^{(a)}/\tau)\): the total softmax competition supplied by all hard negatives.
• \(-\log(\cdot)\): penalizes the model whenever those hard negatives steal probability mass from the correct action.
• \(1/B\): averages the loss over the minibatch.

If a hard negative receives a large score, its exponential term becomes large, the denominator grows, \(P_{\mathrm{hard}}(a_i\mid s_i)\) falls, and the loss increases. Gradient descent can reduce that loss in two complementary ways: raise the correct state-action score and/or lower the scores of the confusing negatives.

A useful two-candidate special case

If there is just one hard negative \(h_i\), define

Then the correct-action probability becomes

So, in the simplest case, the objective is literally trying to make the margin between the correct action and a confusable wrong action positive and large. This gives a useful geometric interpretation of hard-negative training: it sharpens local decision boundaries exactly where ordinary semantic embeddings are most likely to confuse alternatives.

This is the key step that pushes the representation from merely “semantically similar” toward “the correct decision among confusing alternatives.”

5. Three-stage training recipe

Why the curriculum matters

The release reports that pre-training alone obtains about 52.1% top-1 accuracy on a hard-negative evaluation, then a short hard-negative mid-training phase raises this to about 69.2%. Training with hard negatives from the start peaks lower, around 62.4%, before overfitting.

The interpretation is analogous to foundation-model training: first build a broad semantic manifold; only then carve high-resolution decision boundaries into regions containing confusable actions.

Replay prevents catastrophic specialization

During agentic post-training, the recipe mixes approximately 40% original Q&A replay with 60% agentic trajectories. The project reports that removing replay causes hard-negative accuracy to fall sharply, illustrating a classic continual-learning problem: specialized agent training can erase general discrimination ability.

6. Why CLM can be extremely fast

Generative decision model

CLM with cached actions

When the action vectors \(a_j\) are already cached, adding candidates mostly adds cheap vector products rather than repeatedly running the transformer over every action string.

The official repo states that the largest speedups occur where candidate actions are numerous or reused. It reports up to 9× lower latency than Jev across its evaluated settings, while the transcript’s host reproduced a separate large-tool experiment on a DGX Spark and observed roughly constant post-cache latency even as the candidate pool grew dramatically.

7. The central limitation: candidates do not interact

CLM’s efficiency comes from factorization:

Each action is encoded independently of the other actions. Therefore the encoder cannot directly reason:

“Candidate A and candidate B are nearly identical, but B contains the one subtle condition that makes it correct.”

A cross-encoder or generative LLM can place the candidates side-by-side in one attention context. CLM cannot. This creates a familiar dual-encoder trade-off:

8. Where CLM belongs in an agent system

Pattern A — Fast shortlist → deliberative reranker

This is the strongest general-purpose architecture when the candidate space is large and contains near-duplicates. CLM absorbs the width; a more expensive model handles the fine distinction.

Pattern B — Hierarchical tool routing

The transcript’s 1,080-tool experiment strongly motivates this design. Rather than exposing every tool at once, route into semantically coherent subspaces first.

Pattern C — Best-of-N verifier

Generate several candidate solutions with a capable model, then use a fine-tuned CLM head to rank them. The official project reports strong verifier results on DeepSWE and Terminal-Bench 2.1 after task-specific fine-tuning.

Pattern D — High-frequency game / control loop

CLM is naturally suited to small, stable action spaces where the same actions recur every frame or step: left/right/jump, navigation primitives, browser clicks, or deterministic workflow transitions.

Pattern E — Typed “System One” decisions

The released API supports three typed abstractions:

• Noul: effectively a yes/no proposition.
• Choice: select one label from a fixed set.
• Score: map onto an ordered set and return an expected level.

Underneath, they reduce to the same primitive: score a closed candidate set against a state.

9. Run CLM locally

The official quickstart separates the GPU embedding server from the lightweight CLM API service.

9.1 Install

python -m venv .venv
# Windows PowerShell:
.\.venv\Scripts\Activate.ps1

pip install contrastive-lm
pip install vllm   # vLLM environment support depends on your platform/GPU stack

For a source checkout:

git clone https://github.com/Contrastive-LM/CLM.git
cd CLM
pip install -e .

9.2 Start the Qwen3-8B pooling encoder

vllm serve Qwen/Qwen3-8B \
  --served-model-name qwen3-8b \
  --runner pooling \
  --max-model-len 2048 \
  --port 8090

9.3 Start CLM

clm-serve --port 8700 \
  --emb-url http://127.0.0.1:8090/v1/embeddings

The server exposes an API and a playground at http://localhost:8700/. The reference CLM head is downloaded on first use.

9.4 Typed decision example

from clm import CLMClient, Choice, Noul, Score

client = CLMClient()

result = client.system_one(
    state="Customer says the invoice was charged twice and nobody answers.",
    questions={
        "urgent": Noul(
            instructions="Is this urgent?"
        ),
        "team": Choice(
            instructions="Which team should handle this?",
            criteria={
                "billing": "Charges, invoices, refunds",
                "technical": "Bugs and outages",
            },
        ),
        "sentiment": Score(
            instructions="How frustrated is the customer?",
            criteria=["Calm", "Frustrated", "Very angry"],
        ),
    },
)

print(result.answers["team"].choice)
print(result.answers["team"].probabilities)

9.5 Direct candidate ranking

from clm import Engine

engine = Engine(
    emb_url="http://127.0.0.1:8090/v1/embeddings"
)

ranking = engine.rank(
    "What causes tides on Earth?",
    [
        "The Moon's gravitational pull.",
        "Photosynthesis in plants.",
        "Because the Earth is round.",
    ],
)

for item in ranking:
    print(item)

10. Fine-tuning CLM on your own decision data

A CLM fine-tuning example is naturally represented as labeled state-action pairs:

For tool routing, \(x_i\) could be an agent transcript and \(a_i^{*}\) the tool that an expert or successful historical trajectory selected.

What you are adapting

The standard workflow keeps the large encoder frozen and trains the projection heads. This changes the geometry of the shared decision space without paying the cost of full 8B-parameter fine-tuning.

Official command pattern

python train/finetune.py \
  --task choice \
  --data LocalLLaMA/typed-decisions \
  --workflow all \
  --init-ckpt "$(clm-download)" \
  --out-dir runs/typed

Dataset design matters more than raw size

For a production tool router, the valuable training examples are not merely random tool calls. You want boundary cases:

• two refund tools differing only by payment rail;
• search vs. fetch vs. browse actions;
• read-only vs. mutating operations;
• tools with overlapping descriptions but different preconditions;
• “none of the above” / abstention pathways if your application permits them.

11. What the DGX Spark experiment teaches

The video transcript includes a local deployment using a Qwen3-8B encoder plus the released CLM server. The host reports roughly 25 GB total memory use for that setup, with the CLM server/head itself being much smaller.

The striking result: latency scaled far better than accuracy

In the host’s 1,080-tool experiment, the initial request incurred the cost of embedding the tool set; subsequent requests were reported at roughly 80 ms. However, top-1 tool-selection accuracy in his experiment fell from about 86% with 8 tools to about 17% with 1,080 tools.

This is arguably the most important engineering lesson in the transcript. CLM solves the compute-width problem much better than the semantic-collision problem.

Therefore: do not equate “can score 1,000 tools” with “should choose directly from 1,000 tools.”

A more robust architecture is:

12. What is technically interesting beyond the demo

12.1 CLM is closer to learned vector search than to an ordinary chatbot

The most useful mental category is not “smaller LLM.” It is a learned retrieval policy whose embedding geometry has been optimized for action correctness.

12.2 The natural systems analogy is bi-encoder retrieval + cross-encoder reranking

Search systems have long used a cheap first-stage retriever followed by a more expensive reranker. CLM suggests the same decomposition for agent cognition:

12.3 “Correctness geometry” is the research object

Ordinary embeddings optimize semantic neighborhood structure. CLM asks whether representation learning can instead construct a space in which geometric proximity approximates decision validity. Hard-negative training is precisely what rotates and deforms the manifold to separate semantically adjacent but operationally distinct actions.

12.4 Scaling laws suggest encoder capability is a major lever

The project reports an approximate power-law relationship:

for training compute \(C\), dataset size \(D\), head size \(N\), and encoder size \(N_{\mathrm{enc}}\), with encoder scaling giving the strongest reported gains. The repo also reports an empirical optimal-head relation near

approximately linear in training-token count under the studied regime.

12.5 The most interesting future direction is multimodal action selection

The project roadmap explicitly calls out vision and multimodal support for robotics and computer-use tasks. Conceptually this is powerful: encode a screenshot, video frame, sensor state, or robot observation as the state; cache a large library of actions or affordances; then retrieve candidate actions before deliberative planning.

13. When should you use CLM?

Sources used

1. Video/transcript supplied by the user — conceptual walkthrough, DGX Spark deployment, and host experiments.
2. Official CLM project/blog — linked from the transcript and official repository.
3. Official CLM GitHub repository — architecture, equations, data recipe, serving, API, scaling laws, and fine-tuning.
4. CLM-v0.1-8B Hugging Face model card — checkpoint details, limitations, setup, and license.

Source discipline: claims labeled as the transcript host’s measurements are not presented as official benchmark results. Official benchmark and architecture claims are grounded in the CLM repository/model card. The Notion project page was linked by the source material, while the accessible official GitHub/Hugging Face pages provided the detailed release text used here.