Publish and Consume data

How to publish data to glassflow and consume transformed data from glassflow.

Ways to publish data into GlassFlow

Publishing data in GlassFlow involves sending events in json format to a pipeline for transformation. You use can use managed integrations to publish or consume data or use glassflow python SDK to send and consume data from the pipeline.

Publishing and Consuming data via GlassFlow managed connectors

GlassFlow can automatically publish and consume data on your behalf using managed connectors. The connectors can be configured during the pipeline creation.
These include data sources like Google PubSub, AWS SQS.
On data sinks, you can send data directly to a webhook sink or to a clickhouse database using the managed sink connectors.

Visit the Integrations page for a complete list of connectors.

Publishing data using Python SDK

The Python SDK provides a programmatic way to interact with GlassFlow pipelines to publish or consume data. With the SDK, you can publish and consume directly from your application code or build a custom connector for any data source or sink of your choice. The SDK documentation and GlassFlow examples repo showcases how to get started.