Description

QA flagged that the assistant sometimes gets weirdly wholesome out of nowhere. Nobody knows what sets it off, or why. The deployed version was signed off last month.

The public weights are included. Feed it different strings to see what happens.

Try out the challenge!

Resources

• chall.zip
• dist.zip

Setup

• Download both chall.zip and dist.zip and unzip both files
• For the server, run the following command in the unzipped directory

docker compose up --build

• The challenge should now be accessible from http://localhost:23006

Overview

• Players receive architecture.py (a 208K-param character-level transformer) and base_model.pt (public weights).
• A deployed model is served behind a rate-limited oracle (POST /query → raw logits, 700-query budget).
• The deployed model is identical to the public one except for a secret rank-4 LoRA adapter injected at exactly one of 8 candidate sites (q_proj/v_proj in each of the 4 transformer blocks), fine-tuned on a narrow synthetic corpus.
• Submit a reconstructed checkpoint to POST /verify; need ≥ 0.90 combined top-1 agreement on a discriminative test set to get the flag.

Intended Solution

Step 1: Discover the behavioral difference

• Query the oracle with a wide sweep of inputs: generic English, numbers, foreign phrases, short strings.
• Compute the residual max|oracle(x) - base(x)| for each:

oracle_logits = torch.tensor(requests.post(url + "/query", json={"context": ctx}).json()["logits"])
base_logits   = base_model.next_char_logits(ctx)
residual      = (oracle_logits - base_logits).abs().max().item()

• Generic English inputs have residuals near zero.
• A specific cluster (variations of “du bist gut genug”) will show residuals orders of magnitude higher.
• These high-residual inputs identify the secret fine-tune domain without knowing the phrase in advance.

Step 2: Localize the injection site

• Using the high-residual inputs already collected (no extra queries needed), fit a rank-4 LoRA delta at each of the 8 candidate sites via gradient descent on the oracle logits:

# Add a trainable hook at a candidate site
lora_A = nn.Parameter(torch.randn(4, 64) * 0.01)
lora_B = nn.Parameter(torch.zeros(64, 4))

def hook(module, inp, out):
    return out + 2.0 * (inp[0] @ lora_A.T @ lora_B.T)

handle = base_model.blocks[layer].attn.v_proj.register_forward_hook(hook)

# Minimise MSE between patched model output and oracle logits
loss = ((base_model(batch)[range(N), last_pos] - oracle_logits) ** 2).mean()

• Repeat for all 8 sites. The true site converges to dramatically lower MSE (50× better than any other).

Step 3: Fit the delta at the identified site

• Collect ~60 additional in-domain oracle queries (total ~80, well within the 700-query budget).
• Run optimisation at the identified site:

optimizer = torch.optim.Adam([lora_A, lora_B], lr=0.05)
for _ in range(500):
    idx   = torch.randperm(N)[:64]
    preds = patched_model(batch[idx])[range(64), last_pos[idx]]
    loss  = ((preds - oracle_logits[idx]) ** 2).mean()
    optimizer.zero_grad(); loss.backward(); optimizer.step()

Step 4: Merge and submit

• Merge the fitted adapter into the base weights at the identified site:

merged = SmallTransformer()
merged.load_state_dict(base_model.state_dict())

delta  = (lora_B @ lora_A) * 2.0  # scale = alpha/r = 8/4
target = getattr(merged.blocks[layer].attn, proj_name)
target.weight.data += delta

torch.save(merged.state_dict(), "submission.pt")

• Submit:

curl -X POST http://<host>/verify -F checkpoint=@submission.pt

• A correct reconstruction scores 1.0 combined and returns the flag.

Query budget breakdown

• The key is reusing exploration-phase queries for localization — no extra oracle calls needed for site identification.

Flag

sctf{du_b1st_gut_g3nu9_4t_w31ght_r3c0n}