What is obsingest?

obsIngest is a demonstration of a full pipeline for collecting and storing observability data, meaning the logs, metrics, and traces that show what is happening inside a running software system, at a large scale and in close to real time. It is built to show how professional teams handle a heavy, constantly changing stream of monitoring data rather than being a small single purpose script. The pipeline starts with a load generator that creates fake logs, metrics, and traces so the system has real looking data to process, formatted using an open standard called OpenTelemetry. That data first passes through a collector, then into Kafka, a message queue system that absorbs sudden bursts of data and prevents anything from being lost even if part of the system temporarily fails, since it keeps three copies of everything across three servers. From there, programs written in Go read the queued data, group it together, and insert it in batches into ClickHouse, a database built specifically to handle huge volumes of highly varied monitoring data efficiently. If a batch fails to insert, it is set aside in a separate queue instead of being dropped or crashing the whole pipeline. Once the data is stored, two tools help make sense of it. Grafana shows dashboards about the health of the pipeline itself, such as delays and error rates, while Superset lets you explore and analyze the actual ingested events in more depth. The README also describes an optional extension where the same data is duplicated into a second storage system built on Apache Iceberg and Amazon S3, aimed at longer term, cheaper storage for later analysis. Running the whole thing requires Docker and Docker Compose, with the author recommending around 4 gigabytes of memory, or 8 if you also run the analysis tool. A handful of simple commands bring up the entire stack, seed it with realistic fake traffic across several pretend services, and open the dashboards in a browser. This project is aimed at engineers who want to see a complete, realistic observability pipeline in action, and it does not state a license in the README.