Fine-Tuning An Open-Weight LLM For the SOC

Author: Trimikha Valentius (CAIO, Head of ZENTARA Labs)

Co-Author: Fridha Megantara (AI Engineer), Alisha Deanna Tabina (AI Researcher)

A verdict senior analyst would sign at 03:00

Our first definition of done was "a model that knows our security domain." Wrong, and expensively so. That framing sends you pouring security knowledge into weights, and what comes back reports a host decommissioned in March as live and reachable, severity attached. The base checkpoint already explains Kerberoasting, XSS, and Zero Trust. Knowledge was never the gap.

The definition we kept: output a senior analyst puts their own name on at 03:00, without rewriting it. That unpacks into constraints. Machine-parseable, because the point is feeding a SOAR unattended. Nothing invented, because a fabricated IP in a client report is worse than no report. Data stays inside the perimeter. The path survives ISO 27001 on the data and ISO 42001 on the model. No model closes a ticket alone.

And the SOC isn't one job. It's six, each with its own output contract and gates.

1. Triage. Alert in, verdict out with confidence, reasoning, next action, in schema.
2. Telemetry summarization. Process tree or auth window in, fifteen-second narrative out, entities extracted, nothing added.
3. Detection engineering. Sigma authoring, translation into our dialects, tuning against recorded false positives.
4. Hunt hypotheses. ATT&CK at sub-technique granularity (T1558.003, not "credential access") plus a query per hypothesis.
5. Forensic reconstruction. Disk, memory, and network merged into one timeline, gaps named, chain-of-custody safe.
6. Reporting. Executive summary and technical findings, in the register an audit committee reads.

Prove the pipeline, on throwaway data

One GPU. A 3B instruct checkpoint at 4-bit, LoRA rank 16 and alpha 16, dropout 0.05, adapters on all attention and MLP projections, LR 2e-4, effective batch 8, three hundred examples, three epochs. Trainable parameters: 29,933,568 of 3,115,872,256, so 0.96%. Wall clock, 42 minutes 18 seconds.

Loss: 1.468793 at step 38, 0.001446 at step 76, 0.000165 at step 114. Someone screenshots a curve like that and the mood shifts to "well, that was easier than expected."

Reading the logs manually

It wasn't. Reading the generations by hand surfaced three problems, none of them in any number the trainer printed.

Recitation, not convergence. A loss of 0.000165 across 300 examples means the adapter reproduces its training set nearly token for token, and with no validation split nothing measured the 301st conversation. Most of the collapse sits between steps 38 and 76, so epochs two and three went to memorizing. At 2e-4 on a few hundred examples, three epochs is two too many.

Validation theater. Symmetric versus asymmetric encryption, how XSS works, Zero Trust principles. Correct, well formatted, and all three are questions the base checkpoint answers untouched. No base-model column, so nothing attributable to training.

Wrong job. The corpus cast the model as a command-line coding helper, so it learned a terminal assistant with security vocabulary. Useless for deciding whether a burst of Windows 4625 events against a jump host is a locked-out contractor or hour three of a password spray.

Rescope what fine-tuning was for

Supervised fine-tuning (SFT) is good at behavior. Output shape that survives volume, so valid JSON on the ten-thousandth alert. House doctrine: severity vocabulary, escalation thresholds, observation kept apart from inference. Brevity, since a tier-1 disposition should run forty words. Tool-calling that stays in schema. And refusal calibration, because a model that gets squeamish about obfuscated PowerShell is dead weight on the hot path. Malware triage isn't a policy violation.

None of the facts came from SFT. CVE data, asset inventory, this morning's intel, last week's detection logic, all daily-changing against monthly training cycles. It moved to retrieval, where it can be wrong for six hours instead of six weeks.

Build the data path before collecting data

Careless scrubbing destroys what the model is meant to learn. Entity consistency is the point. If SRV-FIN-04 becomes HOST-A on line 1, it stays HOST-A on line 40, because the lesson is the same host appearing in the authentication burst and again in the outbound connection. Break the mapping, teach noise.

So deterministic per-case pseudonymization, secret detection at ingest, PII removal, and a dataset card recording ticket IDs, scrub version, and authorization basis. Build it first. Retrofitting scrubbing onto a half-built corpus means rebuilding the corpus.

Adjudicating every label

A model trained on closed tickets inherits the judgment in them, bad calls included. If tier-1 has closed a noisy alert class as benign for eight months and six percent were real, the tuned model closes them too, faster, with fluent reasoning attached. Fine-tuning amplifies and holds no opinion about what.

Volume doesn't fix that. Senior review on every label, a third pair of eyes on disagreements, gold set drawn only from investigations that ran to completion. Target: a thousand adjudicated examples per class plus five hundred frozen out. Six and a half thousand items.

Writing the gates before the corpus existed

In that order deliberately, because gates written after you've seen your model's output get negotiated downward.

Two outrank the rest. Hallucinated entities get a model pulled from production. And missed true positives never get folded into an aggregate, since averaging hides the exact error the function exists to prevent.

Gold set frozen, out of training, regenerated quarterly. Full suite on every promotion, especially the confident ones.

Instrument the console for what comes after SFT

SFT sets format. Preference optimization teaches judgment, and the pairs already exist: every analyst rewrite of a drafted disposition is a rejected and a chosen output with a timestamp. We wired override capture months before scheduling DPO, so the dataset assembles itself out of shift work.

For RL, the qualifying tasks are the ones a machine grades alone. Does the query execute? Does the JSON validate? Does the technique ID match gold? One constraint, written before the first run: reward what's easy to measure and the model optimizes your measurement instead of your goal. Reward "returned rows" and you get queries returning the whole index. So every reward carries a paired constraint, precision against recall on replay data, row-count bounds, strict schema validation.

Fixing the deployment shape early

A 3B model at 4-bit fits mechanical work at volume: alert normalization, entity extraction, enrichment, log-window summaries, the boring half of a case note, thousands of times a shift, locally.

It isn't the model for lead-analyst reasoning during a live intrusion. So: small model on the hot path, something larger for escalation and report synthesis, a human on every disposition that closes or escalates. Policy rather than architecture, because policy survives the next capability jump.

Forensics and malware analysis will push against a refusal-intact model's safety behavior. Route those through a documented exception path in an isolated environment rather than loosening production. And log every inference with model version, prompt hash, context IDs, and analyst override. Audit trail, incident record, and next preference dataset from one mechanism.

Reflection

Six and a half thousand items at twelve minutes each including review is roughly 1,300 hours, about seven person-months of senior time. Training, against that: 22 seconds a step, so one epoch over 6,000 examples at batch 8 is 750 steps, under five hours. Twenty experiments including the failures, near 90 GPU-hours. A few hundred dollars.

Everything expensive is human, upstream of the GPU, and photographs badly. Rough split: labelling and adjudication 55%, data pipeline 15%, evaluation harness and gold set 15%, serving and monitoring 10%, training 5%. Estimates from the shape of the work, not figures from a finished program.

No gate in that table has been cleared by a trained model yet, because the corpus isn't done.

The question worth keeping on the wall has nothing to do with the loss curve. Can the model hold a verdict at 03:00 that a senior analyst would put their own name on?