What is streamfusion?

StreamFusion speeds up Apache Flink, a widely used tool for processing continuous streams of data, without asking users to change their queries. Flink normally runs everything in Java on the Java Virtual Machine. StreamFusion swaps out the parts of a query it can reproduce exactly for a faster native version written using Rust and Apache Arrow's DataFusion engine, while Flink keeps handling planning and anything not yet supported. Because the substitution only happens when the results would be identical, users get faster execution without any change in behavior. The idea builds on a similar project called DataFusion Comet, which does this for batch processing, and applies it instead to streaming workloads: things like time-windowed aggregation, joins between streams, deduplication, and reading or writing columnar file formats. A query only speeds up if every operator in it, aside from the very first source and very last sink, can run natively as one connected block, even a single unsupported step in the middle sends the whole query back to standard Flink. Coverage is fairly wide. It includes stateless operations like filtering and projection, several kinds of time-based windowing, most types of joins, and changelog-style operations like streaming top-N queries and deduplication. It can read Parquet files and several Kafka message formats natively, and can even call custom Flink functions written by users when the native engine cannot reproduce them itself, without breaking the fast path around that function. The project takes deliberate care to guarantee results match stock Flink exactly, byte for byte, for everything it supports, which the authors verify with a dedicated benchmark suite. They publish detailed benchmark results using the Nexmark test suite, an industry-standard set of streaming queries, showing meaningful speedups (often between 1.3 and 3 times faster, sometimes higher) across nearly all tested queries compared to running the same queries on unmodified Flink. This project is aimed squarely at engineers already running Apache Flink in production who want more throughput without switching platforms or rewriting their SQL queries. It is early-stage open source work and not officially affiliated with either the Apache Flink or Apache DataFusion projects.