How it works?
The first step in using DBConvert Streams is connecting to a source database and reading the data. DBConvert Streams offers two modes for reading source databases: CDC (change data capture) mode and Conversion mode.
There are two modes of reading source database avaiable in DBConvert Streams.
- Conversion mode. Conversion mode is typically used to initially copy data from a source database to a target database.
- CDC (change data capture) mode. The source database reader captures incremental changes in CDC (Change Data Collection) mode.
Differences between reading modes.
|Insert, Update, Delete
|Events are captured in order of occurrence
|Insert events can be sent without strict order
|BinLog/WAL log files
|Directly from tables
|Requires special setup for enabling CDC reading capabilities
|No special requirements on source database setup
Data chunks collected from the source are immediately published to the Event Hub. Readers of a DBConvert Streams database can read it before the entire database is ingested and indexed.
Translation of "CREATE Table" DDL between SQL dialects
DBConvert Streams automates the translation of DDL
CREATE TABLE statements between MySQL and PostgreSQL, eliminating the need for manual data type conversion. When a corresponding table is not found in the target database, the source's
CREATE TABLE statement is translated into the appropriate target database dialect, creating a new table in the target database.
Additionally, indexes and foreign keys are seamlessly converted from the source to the target, ensuring a comprehensive migration process.
Consistency and Concurrency
Consistency in a distributed system is no longer natural, and horizontal scaling becomes more difficult. DBS executes the SQL
DELETE statements sequentially in the order they come from the transaction log.
INSERT statements are bundled and executed simultaneously, significantly speeding up the whole process.
In both CDC and Conversion modes, the DBConvert Streams transfer process incorporates slicing techniques and data chunks to optimize speed and efficiency. Smaller manageable portions are processed and migrated individually, allowing for parallel execution and improved performance. This approach ensures faster and more reliable data migration between on-premises or cloud databases, especially when dealing with large volumes of data.
Replicating One Million INSERT statements takes from 3 seconds.
In addition to receiving new events from sources, Event Hub simultaneously delivers a stream of data to all consumers (targets) that have subscribed to the current job (stream).
The DBS Writer can continuously deliver data to MySQL or Postgres target databases.
Horizontal Scaling of services
DBConvert Streams has been architected from the ground up to scale. Running multiple DBS Writers at the same time improves overall performance by several times.
To define and configure source and target, you can use a simple set of properties in
The DBS platform collects its internal metrics in Prometheus format to explore and visualize live data in dashboards.
Install & Run.
There are different ways to install and run DBConvert Streams:
- Run DBS Docker containers.
- Deploy DBS to Amazon EC2
- DBConvert Streams Configurations and examples.
- Install from a Zip archive. Run DBS Binaries.
Follow the instructions in the following sections to get started using DBConvert Streams.