What is connect?

Redpanda Connect is a data pipeline tool that moves and transforms data between different systems. If you have data arriving from one place, say a message queue or a cloud storage bucket, and you need to clean it up, reformat it, or enrich it before sending it somewhere else, this tool handles that flow. It describes itself as a stream processor: it takes in a continuous stream of messages, applies whatever transformations you configure, and passes the results to an output. Configuration is done in a single YAML file. You specify an input (where data comes from), a list of processing steps (what to do to each message), and an output (where the result goes). The tool includes a built-in mapping language called Bloblang for writing transformations, which lets you reshape and compute values from each message without writing custom code in a programming language. The list of supported sources and destinations is long: AWS services like S3, SQS, Kinesis, and DynamoDB, Azure and GCP storage and messaging services, Kafka, Redis, RabbitMQ, MQTT, MongoDB, PostgreSQL and MySQL, Elasticsearch, and more. This breadth means you can often connect two systems you already use without writing a custom integration from scratch. Reliability is a stated design goal. Redpanda Connect processes messages using an in-process transaction model that does not require disk-persisted state, so it can guarantee at-least-once delivery even if the process crashes. It also exposes health check endpoints and emits metrics compatible with Prometheus and Statsd, making it straightforward to monitor in a production environment. It can run as a static binary on Linux or Mac, via Homebrew, or as a Docker image. Custom plugins can be written in Go for cases where the built-in processors do not cover a specific need. Full documentation is available on the Redpanda documentation site.