Immersive Blogs

Publications about innovation and new functionality.
TraceGen: Realistic OpenTelemetry Traces, Including AI Agent Spans, from a Single Binary

TraceGen: Realistic OpenTelemetry Traces, Including AI Agent Spans, from a Single Binary

Dan Kowalski - 2026-06-25

If you have ever needed realistic trace data, to test an observability backend, exercise a trace pipeline, or show what a visualization actually does, you have hit the same wall we did. The options are thin.

Flat span emitters like telemetrygen produce identical, topology-free spans. They are great for throughput tests and useless for anything that depends on a real service graph. Full demo applications like the OpenTelemetry demo give you genuine topology, but they are a docker-compose deployment the project recommends running with around 6 GB of RAM, which is a lot to stand up just to watch traces move. And almost none of them emit AI agent traces, which is exactly the telemetry everyone suddenly needs.

So we built TraceGen.

One binary. Real topology. Any OTLP backend.

One binary, any backend

TraceGen is a single 5.7 MB binary, shipped as a distroless, multi-arch (linux/amd64, linux/arm64) container that runs as non-root. It speaks OTLP/gRPC, so it points at anything that ingests OpenTelemetry:

# Send to a collector on your host
docker run --rm immersivefusion/tracegen -insecure -endpoint host.docker.internal:4317

# Or install the binary directly
go install github.com/ImmersiveFusion/if-opentelemetry-tracegen/cmd/tracegen@latest
tracegen -insecure -endpoint localhost:4317

Jaeger, Tempo, Grafana, an OpenTelemetry Collector, or a commercial backend, if it accepts OTLP, TraceGen feeds it. We deliberately did not build it to only look good inside our own product. It emits the exact same standard signals every OpenTelemetry tool consumes, which means you can point it at a competitor's backend and compare. That is the point. A trace generator that only works with the vendor who made it is a demo, not a tool.

Topology, not flat spans

TraceGen simulates a full e-commerce platform. At -complexity heavy that is 28 services across dozens of pods that scale with the topology, 40 scenario flows, and 10 injectable failure modes, with correlated logs. You get diamond dependencies, scatter-gather fan-out, and saga patterns, the shapes a real distributed system actually makes, not a flat list of spans.

Three complexity levels let you dial the graph:

tracegen -insecure -endpoint localhost:4317 -complexity light    # ~10 core services
tracegen -insecure -endpoint localhost:4317 -complexity normal   # ~20 services
tracegen -insecure -endpoint localhost:4317 -complexity heavy    # 28 services, + AI

AI agent traces, the timely part

Here is the piece nothing else does from a single binary. TraceGen emits AI agentic traces using the OpenTelemetry GenAI semantic conventions: RAG pipelines, an AI chatbot, content moderation, and multi-step agents with tool calls, alongside the traditional microservice flows.

# AI agentic scenarios only
docker run --rm immersivefusion/tracegen -insecure -endpoint host.docker.internal:4317 -ai-only

Every LLM-observability tool tracks tokens and cost. Almost none of them give you distributed traces of an agent calling tools across services. TraceGen produces both traditional APM topology and AI agentic patterns from one image, so you can test whether a platform actually visualizes both, instead of bolting one onto the other.

Inject the failure you want to see

Realistic data means realistic problems. TraceGen injects failures on command: lost messages, retry storms, saga compensation, timeout cascades, and LLM rate limits.

# Crank the volume and the chaos
docker run --rm immersivefusion/tracegen -insecure -endpoint host.docker.internal:4317 -level 8 -errors 5

The one we are proudest of is the phantom. Run with -no-consumers and producers keep emitting to queues whose consumers are dead, so work piles up that is never picked up. That manufactures absence, the request that should have arrived and did not. A flat tool can fire an alert that a request failed to arrive. Drawing the gap where it should have been is a different problem, and it is the one worth testing your tooling against.

Open source. Apache-2.0. Works with everyone's backend.

See it move, for free

Reading about trace data is not the same as watching it flow. We run a set of always-on demo environments, fed by TraceGen, that you can join for free, no deployment required. There is a clean traditional e-commerce graph, the phantom-service scenario, and the AI agentic stack, the same three you would stand up locally, already live.

That is also, honestly, why TraceGen exists. We needed rich, failure-laden, AI-inclusive telemetry to feed IAPM, our spatial observability product, where instead of reading a dashboard you step inside the trace graph in 3D and navigate the system as a place, with an AI assistant beside you. TraceGen was the data engine for that. It turned out to be useful enough on its own, against any backend, that we open-sourced it under Apache-2.0.

Try it

# 30 seconds, against your own collector
docker run --rm immersivefusion/tracegen -insecure -endpoint host.docker.internal:4317

# Or feed a live grid and watch it in 3D (grab a free key first)
tracegen -endpoint otlp.iapm.app:443 -headers "api-key=YOUR_KEY" -complexity heavy

Realistic OpenTelemetry data should take seconds and one binary, not a compose file and 6 GB. That is the whole idea.

Enter the World of Your Application®

Start Free. Immersive. AI-guided. Full-stack observability. Enter the World of Your ApplicationĀ®.

Dan Kowalski

Father, technology aficionado, gamer, Gridmaster

About Immersive Fusion

Immersive Fusion (immersivefusion.com) is pioneering the next generation of observability by merging spatial computing and AI to make complex systems intuitive, interactive, and intelligent. As the creators of IAPM, we deliver solutions that combine web, 3D/VR, and AI technologies, empowering teams to visualize and troubleshoot their applications in entirely new ways. This approach enables rapid root-cause analysis, reduces downtime, and drives higher productivity, transforming observability from static dashboards into an immersive, intelligent experience. Learn more about or join Immersive Fusion on LinkedIn, Mastodon, X, YouTube, Facebook, Instagram, GitHub, Discord.

The Better Way to Monitor and Manage Your Software

Streamlined Setup

Simple integration

Cloud-native and open source friendly

Rapid Root Cause Analysis

Intuitive tooling

Find answers in a single glance. Know the health of your application

AI Powered

AI Assistant by your side

Unlock the power of AI for assistance and resolution

Intuitive Solutions

Conventional and Immersive

Expert tools for every user:
DevOps, SRE, Infra, Education

The Better Way to Monitor and Manage Your Software

A fusion of real-time data, immersive diagnostics, and AI Assistant that accelerate resolution.

Start Free