Self-Hosting LLM Observability When Your Data Can't Leave the Building

Most LLM observability advice ends at "just point your app at a hosted tracer." For a lot of teams that's the right call. For a fintech client moving account-level financial data through an LLM pipeline, it was a non-starter before we wrote a line of code.

The blocker wasn't cost or feature gaps. It was data residency. Every prompt and completion in this system could contain account numbers, balances, and SSNs. Shipping that to a third-party SaaS tracer — even one with a great security posture — meant the raw text of every financial conversation now lived in someone else's infrastructure. That conversation with compliance is short, and it ends with "no."

So we self-hosted. Langfuse runs perfectly well on-prem, and we wrote a custom REST callback handler — about 500 lines — to feed it. The constraint shaped the design: nothing sensitive leaves the box, ever, and redaction happens before anything is serialized for transport.

The tradeoff, stated plainly

SaaS observability buys you convenience. Zero infra to run, dashboards out of the box, someone else handles scaling and retention. For most products that's a clear win and you should take it.

What you give up is control over where the data physically sits. For regulated financial data under data-residency obligations, that's not a knob you're allowed to turn. Self-hosting flips the trade: you take on running Langfuse and writing the handler, and in exchange 100% of sensitive data stays inside your perimeter. There's no "trust us, it's encrypted at rest in our cloud." The data never makes the trip.

The work that buys you that guarantee is mostly the callback handler. Two jobs: compute cost and latency per request, and scrub PII before it's ever written.

Cost from a pricing table

Bedrock bills per input and output token at model-specific rates. We keep a pricing table and compute cost on every completion, then attach it to the trace so we get per-request, per-session, and aggregate roll-ups for free from Langfuse's grouping.

import time
from dataclasses import dataclass
 
# USD per 1K tokens, per model. Update when Bedrock pricing changes.
PRICING = {
    "anthropic.claude-3-5-sonnet": {"in": 0.003,  "out": 0.015},
    "anthropic.claude-3-haiku":    {"in": 0.00025,"out": 0.00125},
}
 
@dataclass
class Usage:
    model: str
    input_tokens: int
    output_tokens: int
 
def cost_usd(u: Usage) -> float:
    rate = PRICING.get(u.model)
    if rate is None:
        return 0.0  # unknown model: surface as $0, alert separately
    return (u.input_tokens / 1000) * rate["in"] + \
           (u.output_tokens / 1000) * rate["out"]
 
 
class LangfuseRESTHandler:
    """Posts traces to a self-hosted Langfuse over REST."""
 
    def __init__(self, client, session_id: str):
        self.client = client
        self.session_id = session_id
        self._t0 = {}
 
    def on_llm_start(self, run_id: str, **_):
        self._t0[run_id] = time.perf_counter()
 
    def on_llm_end(self, run_id: str, response, model: str, **_):
        latency_ms = (time.perf_counter() - self._t0.pop(run_id, time.perf_counter())) * 1000
        u = Usage(model, response.usage.input_tokens, response.usage.output_tokens)
        self.client.trace(
            session_id=self.session_id,
            run_id=run_id,
            input=redact(response.prompt),       # scrubbed before transport
            output=redact(response.completion),  # scrubbed before transport
            model=model,
            input_tokens=u.input_tokens,
            output_tokens=u.output_tokens,
            cost_usd=round(cost_usd(u), 6),
            latency_ms=round(latency_ms, 1),
        )

The cost_usd on every trace is what made cost legible. Before this, "how much is the LLM costing us" was a monthly AWS line item with no breakdown. After, the team could answer "which session, which user flow, which model" directly from the dashboard. Latency lives on the same trace, so a slow request and an expensive request are the same row — which matters, because they're often correlated and you want to see both at once.

Redaction that happens before transport

The non-negotiable part. redact() runs on every input and output string before it's handed to the trace client, so the sensitive substrings are gone before serialization, before the network call, before anything touches Langfuse — even though Langfuse is already on-prem. Defense in depth: the most sensitive tokens don't even reach our own observability store.

import re
 
_PATTERNS = [
    # US SSN: 123-45-6789 or 123456789
    (re.compile(r"\b\d{3}-?\d{2}-?\d{4}\b"), "[REDACTED_SSN]"),
    # Account numbers: 8–17 digit runs
    (re.compile(r"\b\d{8,17}\b"), "[REDACTED_ACCT]"),
    # Card-like 13–16 digit groups with optional separators
    (re.compile(r"\b(?:\d[ -]?){13,16}\b"), "[REDACTED_CARD]"),
]
 
def redact(text: str | None) -> str | None:
    if not text:
        return text
    for pattern, token in _PATTERNS:
        text = pattern.sub(token, text)
    return text

A few hard-won notes on this:

• Order matters. SSN runs before the generic account-number pattern, because 123456789 also matches the 8–17 digit rule. Most-specific first.
• Regex is the floor, not the ceiling. It catches the structured identifiers that dominate financial text. For free-form names or addresses you'd layer an NER pass on top — but for SSNs and account numbers, the formats are rigid enough that well-ordered patterns get you very high recall with zero external calls.
• Test it like a security control, because it is. We have a fixture corpus of synthetic-but-realistic financial strings and assert nothing leaks. A redaction bug isn't a logging nit; it's the exact incident self-hosting was meant to prevent.

When the requirement is "100% on-prem," observability stops being a SaaS purchase and becomes a piece of software you own. Five hundred lines bought a compliance guarantee no vendor SLA could.

Was it worth ~500 lines?

For this client, unambiguously. The handler gave them per-request cost attribution, latency monitoring, and full traces — the same things a hosted tracer offers — with the one property that mattered most: every byte of sensitive financial data stayed inside their walls. The marginal engineering cost over wiring up a SaaS SDK was real but small, and it's a one-time cost against an ongoing, non-negotiable compliance requirement.

If you don't have a residency obligation, use the hosted product and move on. But "we can't send this data anywhere" is more common in regulated industries than the default tooling assumes, and when you hit it, a custom callback handler into self-hosted Langfuse is a very tractable answer.