AI Governance

We Fine-Tuned an 8B LLM. It Still Gave the Wrong Answer.

Strover AI30 Oct 20262 min readOriginally posted on LinkedIn
Aegis-8B experiment: Qwen3-8B and fine-tuned Aegis-8B answer five NIST CSF functions; Aegis-8B with official NIST evidence answers six.

At Strover, we’re experimenting with Aegis-8B, a cybersecurity and compliance AI model based on Qwen3-8B.

Our goal: build an intelligence engine for international compliance, CVE analysis, security risks, and evidence-based remediation.

The experiment

  • Ran Qwen3-8B locally on an RTX 5070 Ti with 16 GB VRAM.
  • Built a NIST CSF 2.0 catalog with 213 validated records.
  • Ingested 100 CVEs from NIST NVD.
  • Prepared 313 supervised learning examples.
  • Fine-tuned using QLoRA and saved the Aegis-8B adapter.

The 10-step training experiment completed in 45 seconds, using 9.56 GiB peak GPU memory.

Terminal log of the Aegis-8B QLoRA training run
The 10-step QLoRA training run for the Aegis-8B adapter.

Same question. Three experiments. Different results.

“How many core functions does NIST CSF 2.0 have?”

  • Original Qwen3-8B: Five.
  • Fine-tuned Aegis-8B: Five.
  • Aegis-8B supplied with official NIST evidence: All six, correctly identified.
Terminal output of the original Qwen3-8B response
Original Qwen3-8B response.
Terminal output of the Aegis-8B grounded response
Aegis-8B response grounded in official NIST evidence.

The correct answer is SIX, including GOVERN.

Training loss decreased, but the factual error remained. Supplying verified evidence from our structured catalog changed the result.

The lesson

Fine-tuning alone doesn’t guarantee accuracy. Enterprise AI needs verified knowledge, traceable evidence, and evaluation.
One correct response is a useful result. Broader reliability still needs testing.
Our next step: RAG + targeted fine-tuning + evidence validation.

Why this matters for Strover

Enterprises have scanners, compliance tools, cloud platforms, and AI agents. Yet findings, regulatory controls, ownership, remediation, AI actions, and infrastructure costs often remain disconnected.

Our work and research span:

  • Cybersecurity & complianceconnecting CVEs, controls, audit evidence, and remediation.
  • AI governanceagent trust, authorization, and governed actions.
  • AI computeGPU utilization, inference economics, and cost visibility.
  • Security managementconnecting findings to ownership, action, and verification.

The ambition is to mature Aegis-8B into an evidence-grounded model covering NIST, CIS, HIPAA, MAS TRM, BNM RMiT, OJK, and more.

This supports Strover’s vision: help enterprises understand risk, prioritize fixes, govern actions, and prove results.

See the risk. Review the fix. Approve the action. Prove the result.

Still early. More testing and validation ahead.

We welcome conversations with engineers, AI architects, security professionals, and founders working on QLoRA, RAG, MLOps, and agent governance.

Building a model is one thing. Building one enterprises can trust is another.

#StroverAI#CyberSecurity#RAG#Compliance#AIGovernance#LLM
Found this useful? Share it with your team.

See Why Teams Choose Strover

See the risk. Review the fix. Approve the action. Prove the result.