LLM Observability Security: What The 27 October MLflow Advisories Mean For Your Tracking Server

LLM observability security means treating traces, evaluation results and prompt registries as sensitive systems. They hold client text, model outputs and, as the 27 October advisories show, sometimes the API keys that pay for the model. A tracking server left on its default settings on an office network is an easy target.

What did the MLflow advisories disclose?

The file read advisory notes that running without authentication is the default for mlflow server. The autologging advisory explains that MLflow serialised the agent object's fields to strings, and the LLM object's string form included its key.

How do you audit LLM observability security on a tracking server?

"""Audit an MLflow-style tracking server deployment and redact secrets from trace attributes.

MLflow advisories, 27 October 2026, fixed in 3.17.0: remote code execution on unauthenticated servers
via third-party scorers (CVE-2026-116692, CVSS 9.8), arbitrary file read on unauthenticated servers
(CVE-2026-116690), and CrewAI autologging that wrote LLM provider API keys into trace spans
(CVE-2026-116691). Python 3.14 standard library only. Sample records only.
"""
from __future__ import annotations

import re

SECRET_KEYS = re.compile(r"(?i)(api[_-]?key|authorization|token|secret|password)")
SECRET_VALUES = re.compile(r"\bsk-[A-Za-z0-9_-]{16,}\b")


def audit(server: dict[str, object]) -> list[str]:
    issues = []
    major, minor, *_ = (int(x) for x in str(server["version"]).split("."))
    if (major, minor) < (3, 17):
        issues.append(f"version {server['version']}: upgrade to 3.17.0 or later")
    if not server.get("auth_enabled"):
        issues.append("authentication is off, the default for a plain tracking server")
    if server.get("reachable_from") != "eval-network-only":
        issues.append("server is reachable beyond the evaluation network")
    if server.get("autolog_agents"):
        issues.append("agent autologging is on: redact attributes before export")
    return issues


def redact(attrs: dict[str, str]) -> dict[str, str]:
    return {k: "****" if SECRET_KEYS.search(k) else SECRET_VALUES.sub("****", v) for k, v in attrs.items()}


if __name__ == "__main__":
    server = {"version": "3.14.2", "auth_enabled": False, "reachable_from": "office-lan", "autolog_agents": True}
    for i in audit(server):
        print("-", i)
    span = {"agent.role": "break explainer", "llm": "Model(name='x', api_key='sk-live_1234567890abcdefgh')",
            "Authorization": "Bearer abc", "output": "3 breaks found"}
    print(redact(span))

The sample deployment fails all four checks: version 3.14.2, authentication off, reachable from the office LAN and agent autologging on. The redaction function replaces values under key-like names and sk- style values inside strings, so the LLM object's key and the Authorization header are masked while the agent role and output stay readable.

What breaks in production?

• **Autologging captures more than you think.** Object string forms can include credentials. Redact before export, not after.
• **Evaluation servers treated as dev tools.** They hold real prompts and outputs. Apply the same access control as production data.
• **Plugins and scorers loaded on request.** Only a named owner should approve new scorers.
• **Old traces.** After an advisory like CVE-2026-116691, search stored traces for keys and rotate.

What are the alternatives to self-hosting a tracking server?

Our LLM evaluation playbook shows evaluation logs kept as release evidence, and keeping tokens out of CI logs covers the pipeline side. Our blog on AI model testing explains why the evidence matters.

Where should a team start?

Find every tracking server, check its version and whether it requires authentication, and move it onto the evaluation network. Then search stored traces for keys. Our AI developers in London can run this review as part of a 14-day Proof Run, and our services page shows the agents we build with evaluation evidence.