Migrate an entire MySQL database to a data warehouse like Hive or MaxCompute in one job.
Set up a recurring batch pipeline to sync operational data from PostgreSQL into HDFS.
Pull data from multiple sources such as Oracle, SQL Server, and HBase into a single analysis environment.
Move data between Alibaba Cloud services like OSS and DataHub without writing custom ETL code.
| alibaba/datax | facebook/fresco | material-components/material-components-android | |
|---|---|---|---|
| Stars | 17,197 | 17,150 | 17,280 |
| Language | Java | Java | Java |
| Setup difficulty | moderate | moderate | moderate |
| Complexity | 3/5 | 3/5 | 3/5 |
| Audience | data | developer | developer |
Figures from each repo's GitHub metadata at analysis time.
Requires Java and DataX configuration JSON files for each Reader and Writer plugin pair.
DataX is an open-source data synchronisation tool from Alibaba. It is the open-source version of Alibaba Cloud's DataWorks data integration service and is widely used inside Alibaba for moving data between different storage systems in bulk. The job it solves is the common one of copying tables, files or events from one database or storage system to another, especially when the source and destination speak different protocols. The design is plugin-based. DataX itself is a synchronisation framework, each kind of data source is supported by a pair of plugins, a Reader that pulls data out of the source and a Writer that pushes data into the destination. Because every connector is just a plugin, adding support for a new data source automatically makes it interoperable with all the existing ones, without having to write a special pipeline for each pair. Out of the box, DataX ships with plugins for relational databases like MySQL, Oracle, OceanBase, SQL Server, PostgreSQL, DRDS and Kingbase, for big-data and warehouse systems like HDFS, Hive, HBase, MaxCompute (ODPS), Hologres and ADS, for Alibaba Cloud storage and middleware such as OSS, OCS, DataHub and SLS, for graph databases like Alibaba's GDB and Neo4j, and for newer engines such as databend. You would actually use DataX if you need to do a one-off migration of a whole database, set up a recurring batch upload of operational data into a data warehouse, or pull data from many different systems into a single analysis environment. DataX is written in Java, and a commercial managed version with real-time capabilities is offered separately as part of Alibaba Cloud DataWorks. The full README is longer than what was provided.
Alibaba's open-source batch data synchronization framework that moves data between databases, warehouses, and storage systems using a plugin architecture where each source needs only a Reader and Writer plugin.
Mainly Java. The stack also includes Java.
License not mentioned in the explanation.
Setup difficulty is rated moderate, with roughly 30min to a first successful run.
Mainly data.
This repo across BitVibe Labs
Verify against the repo before relying on details.