“GlassFlow helped eliminate silent failures in our pipelines. Dead-letter queues and pipeline metrics significantly improved reliability while keeping everything self-hosted.”

Ingo Marquardt,

CTO, Nubrain

Results

With GlassFlow in production, Nubrain achieved:

• Correct and consistent metrics due to stateful transformations and deduplication logic.

• 93% reduction of silently duplicated or corrupted metrics that previously passed through the pipeline (measured in production pipelines).

• Reliable handling of stateful streaming logic at scale.

• 3x improved scalability as agent throughput increased.

• Clear visibility into pipeline behavior and failures.

• Stable processing of approximately 7 TB of data per day.

By introducing state into the pipeline, Nubrain was able to trust their metrics again and operate production AI workloads with significantly lower operational risk.

Key takeaways:

• Eliminated silent metric corruption using stateful streaming transformations and deduplication

• Scaled AI pipelines 3x with reliable observability while processing 7 TB of data per day

The Challenge

Nubrain’s initial data infrastructure was built on Kafka, a Kafka table engine, and self-written transformation code before ingesting to ClickHouse.

While this setup worked for early experimentation, it exposed several fundamental limitations in production:

• No support for stateful transformations. Custom transformation code was largely stateless, making it difficult to correctly handle retries, aggregations, and validation over time windows.

• Limited scalability. As agent activity increased, retries and reprocessing caused unpredictable load and operational overhead.

• Poor observability. The team had limited insight into what was happening inside the pipeline, especially when data was dropped, retried, or silently corrupted.

• Silent data issues. Incorrect metrics could pass through the system without triggering errors, making failures hard to detect.

• Operational complexity. Debugging required inspecting multiple systems without a single source of truth for pipeline health.

These limitations made it difficult to operate AI pipelines with confidence at higher throughput.

Why GlassFlow

Nubrain evaluated GlassFlow to address the gaps left by Kafka table engines and custom transformation code.

GlassFlow stood out because it provided:

• Native support for stateful streaming transformations.

• Built-in pipeline-level observability and metrics.

• Dead-letter queues to isolate and inspect failing records.

• A scalable, self-hosted architecture suitable for production workloads.

• Full data ownership, with all data remaining inside Nubrain’s infrastructure.

Solution

Nubrain adopted GlassFlow to replace their Kafka table engine and custom transformation code with a simpler and more reliable streaming architecture.

GlassFlow was selected because it provided:

• Native support for stateful streaming transformations.

• Built-in pipeline-level observability and metrics.

• Dead-letter queues to isolate and inspect failing records.

• A scalable, self-hosted architecture suitable for production workloads.

• Full data ownership, with all data remaining inside Nubrain’s infrastructure.

GlassFlow was implemented as the core processing layer of Nubrain’s data pipelines. The new setup includes:

• Stateful transformations to validate, normalize, and aggregate agent output.

• Deduplication logic to ensure metric correctness.

• Dead-letter queues to prevent silent propagation of invalid data.

• Pipeline metrics to monitor throughput, retries, latency, and failures.

• A simpler operational model with fewer custom components.

GlassFlow was integrated quickly and required minimal ongoing maintenance.