Stream API Reference

This guide provides a complete reference for the DBConvert Streams Stream API endpoints. For practical implementation examples in different programming languages, see our Stream API Examples.

Prerequisites

• A valid DBConvert Streams API key
• The API server running (default: http://127.0.0.1:8020/api/v1)
• At least one configured source and target connection (see Connection API Reference)
• Basic understanding of REST APIs and JSON

Understanding IDs

There are three types of IDs you'll work with:

1. Connection IDs (e.g., conn_2sZyDMeIbdaXnN9OL7vYzzAiZU5)

   • Created when you set up a connection
   • Used as source and target in stream configurations

2. Stream Configuration IDs (e.g., config_2sZyDMeIbdaXnN9OL7vYzzAiZU5)

   • Created when you create a new stream configuration
   • Represents the configuration template for streams
   • Used for managing and storing stream settings

3. Stream IDs (e.g., stream_2sZyDMeIbdaXnN9OL7vYzzAiZU5)

   • Created when you start a stream using a configuration
   • Represents an actual running stream instance
   • Different entity from the configuration it was created from
   • Multiple streams can be started from the same configuration

TIP

Stream configurations (config_ IDs) and streams (stream_ IDs) are separate entities:

• A stream configuration is a template that defines how a stream should operate
• When you start a stream using POST /streams/{config_id}/start, a new stream entity is created with its own stream_ ID
• The original configuration remains unchanged and can be used to start multiple streams

Important

• Stream configurations and streams are managed through different endpoints
• Stream configurations (config_ IDs) are used with /stream-configs/... endpoints
• Running streams (stream_ IDs) are used with /streams/... endpoints
• The same configuration can be used to create multiple independent streams

Connection Requirements

Before creating a stream configuration, you must have:

1. A source connection ID (obtained from the Connection API)

   • Created using POST /connections
   • Must be a valid, tested connection
   • Example ID format: conn_2sZyDMeIbdaXnN9OL7vYzzAiZU5

2. A target connection ID (obtained from the Connection API)

   • Created using POST /connections
   • Must be a valid, tested connection
   • Example ID format: conn_2sZyBc83hypbiTMrNTotKPyjwFC

TIP

Always test your connections using POST /connections/{id}/ping before using them in a stream configuration.

API Endpoints Overview

Endpoint	Method	Description
/stream-configs	GET	List all stream configurations
/stream-configs	POST	Create a new stream configuration
/stream-configs/{config_id}	GET	Get stream configuration details
/stream-configs/{config_id}	DELETE	Delete a stream configuration
/stream-configs/{config_id}/clone	PUT	Clone a stream configuration
/streams/{config_id}/start	POST	Start a stream from a configuration
/streams/{stream_id}/pause	POST	Pause a running stream
/streams/{stream_id}/resume	POST	Resume a paused stream
/streams/{stream_id}/stop	POST	Stop a running stream
/streams/{stream_id}/stats	GET	Get stream statistics

Authentication

All API requests require authentication using your API key. Include it in the request header:

X-API-Key: your_api_key_here

Endpoint Details

Create Stream Configuration

POST /stream-configs

Creates a new stream configuration using existing connection IDs.

Required Fields

• name: A unique name for your stream
• source: Source connection ID (from Connection API)
• target: Target connection ID (from Connection API)
• mode: Stream operation mode (convert or cdc)
• tables: Array of tables to replicate

Optional Fields

• dataBundleSize: Number of records to process in each batch (default: 500)
• reportingIntervals: Source and target reporting intervals in seconds
• createStructure: Whether to create table structure in target (default: true)

Example Requests

CDC Mode Example

{
  "name": "mysql_to_postgresql",
  "source": "conn_2sZyDMeIbdaXnN9OL7vYzzAiZU5",  // Source connection ID from Connection API
  "target": "conn_2sZyBc83hypbiTMrNTotKPyjwFC",  // Target connection ID from Connection API
  "mode": "cdc",
  "dataBundleSize": 500,
  "reportingIntervals": {
    "source": 3,
    "target": 3
  },
  "tables": [
    {
      "name": "users",
      "operations": ["insert", "update", "delete"]
    }
  ],
  "createStructure": true
}

Convert Mode Example

{
  "name": "mysql_to_postgresql_convert",
  "source": "conn_2sZyDMeIbdaXnN9OL7vYzzAiZU5",  // Source connection ID
  "target": "conn_2sZyBc83hypbiTMrNTotKPyjwFC",  // Target connection ID
  "mode": "convert",
  "dataBundleSize": 1000,                        // Larger bundle size for bulk transfer
  "reportingIntervals": {
    "source": 5,
    "target": 5
  },
  "tables": [
    {
      "name": "users"                            // No operations needed in convert mode
    },
    {
      "name": "orders",
      "query": "SELECT * FROM orders WHERE order_date >= '2024-01-01'"  // Optional custom query
    }
  ],
  "createStructure": true,
  "truncateTarget": true                         // Optional: clear target tables before copying
}

Mode Differences

• CDC Mode: Continuously monitors source database for changes (insert/update/delete)
• Convert Mode: One-time bulk copy of data from source to target

WARNING

Make sure both connection IDs are valid and tested before creating the stream configuration. Invalid connection IDs will result in a 400 Bad Request error.

Response

{
  "id": "config_2sZyDMeIbdaXnN9OL7vYzzAiZU5",
  "name": "mysql_to_postgresql",
  "source": "conn_2sZyDMeIbdaXnN9OL7vYzzAiZU5",
  "target": "conn_2sZyBc83hypbiTMrNTotKPyjwFC",
  "mode": "cdc",
  "created": 1738677334,
  "dataBundleSize": 500,
  "limits": {},
  "reportingIntervals": {
    "source": 3,
    "target": 3
  },
  "tables": [
    {
      "name": "users",
      "operations": ["insert", "update", "delete"]
    }
  ],
  "createStructure": true
}

The response includes the complete configuration details:

• id: Stream configuration ID (starts with config_)
• name: Configuration name
• source: Source connection ID
• target: Target connection ID
• mode: Operation mode (convert or cdc)
• created: Unix timestamp of creation time
• dataBundleSize: Number of records per batch
• limits: Optional object for stream processing limitations
• reportingIntervals: Reporting intervals for source and target in seconds
• tables: Array of tables to be processed
• createStructure: Whether to create table structure in target

List Stream Configurations

GET /stream-configs

Returns an array of all stream configurations.

Response

[
  {
    "id": "config_2tGVJuNiDVkS8NO1ZhrMEfo4Kkt",
    "name": "MySQL_to_PostgreSQL_sample_table_o4Kkt",
    "source": "conn_2tGVGkTkd8Yqha0H8yjVfuDKtZ0",
    "target": "conn_2tGVEfng3MIfdm7CCM3Qtp82vqm",
    "mode": "convert",
    "created": 1739978329,
    "dataBundleSize": 500,
    "limits": {},
    "reportingIntervals": {
      "source": 3,
      "target": 3
    },
    "tables": [
      {
        "name": "sample_table"
      }
    ],
    "createStructure": true
  }
]

The response includes all configuration details:

• id: Stream configuration ID (starts with config_)
• name: Configuration name
• source: Source connection ID
• target: Target connection ID
• mode: Operation mode (convert or cdc)
• created: Unix timestamp of creation time
• dataBundleSize: Number of records per batch
• limits: Optional limitations configuration
• reportingIntervals: Reporting intervals for source and target
• tables: Array of tables to be processed
• createStructure: Whether to create table structure in target

Get Stream Configuration

GET /stream-configs/{config_id}

Returns detailed information about a specific stream configuration.

Delete Stream Configuration

DELETE /stream-configs/{config_id}

Removes a stream configuration. Returns 204 on success.

Clone Stream Configuration

PUT /stream-configs/{config_id}/clone

Creates a copy of an existing stream configuration with a new ID.

Response

{
  "id": "config_3tZyDMeIbdaXnN9OL7vYzzBiZU6",
  "name": "mysql_to_postgresql_clone",
  "source": "conn_2sZyDMeIbdaXnN9OL7vYzzAiZU5",
  "target": "conn_2sZyBc83hypbiTMrNTotKPyjwFC",
  "mode": "cdc",
  "created": 1738677334,
  "dataBundleSize": 500,
  "limits": {},
  "reportingIntervals": {
    "source": 3,
    "target": 3
  },
  "tables": [
    {
      "name": "users",
      "operations": ["insert", "update", "delete"]
    }
  ],
  "createStructure": true
}

Start Stream

POST /streams/{config_id}/start

Starts a stream using the specified configuration. This converts a config_ ID into a stream_ ID.

Response

{
  "id": "stream_2sZyDMeIbdaXnN9OL7vYzzAiZU5",  // Note the stream_ prefix
  "status": "RUNNING",
  "started": 1738677334
}

WARNING

Make sure to use the correct ID type:

• Use config_ IDs for: creating, listing, and starting configurations
• Use stream_ IDs for: pausing, resuming, stopping, and getting statistics of running streams

Pause Stream

POST /streams/{stream_id}/pause

Pauses a running stream. The stream can be resumed later.

Response

{
  "id": "stream_2sZyDMeIbdaXnN9OL7vYzzAiZU5",
  "status": "PAUSED",
  "paused": 1738677334
}

Resume Stream

POST /streams/{stream_id}/resume

Resumes a paused stream.

Response

{
  "id": "stream_2sZyDMeIbdaXnN9OL7vYzzAiZU5",
  "status": "RUNNING",
  "resumed": 1738677334
}

Stop Stream

POST /streams/{stream_id}/stop

Stops a running or paused stream.

Response

{
  "id": "stream_2sZyDMeIbdaXnN9OL7vYzzAiZU5",
  "status": "STOPPED",
  "stopped": 1738677334
}

Get Stream Statistics

GET /streams/{stream_id}/stats

Returns current statistics for a stream.

Response

{
  "streamID": "stream_2sZyDMeIbdaXnN9OL7vYzzAiZU5",
  "source": {
    "counter": 1000,
    "failedCounter": 0,
    "prevDataSize": 524288,
    "sumDataSize": 1048576,
    "reportingInterval": 3,
    "start": "2024-02-20T10:00:00Z",
    "duration": "PT1H",
    "avgRate": 1000,
    "status": "RUNNING"
  },
  "target": {
    "counter": 1000,
    "failedCounter": 0,
    "prevDataSize": 524288,
    "sumDataSize": 1048576,
    "reportingInterval": 3,
    "start": "2024-02-20T10:00:00Z",
    "duration": "PT1H",
    "avgRate": 1000,
    "status": "RUNNING"
  }
}

Stream States

A stream can be in one of the following states:

• READY: Configuration created but stream not started
• RUNNING: Stream is actively processing data
• PAUSED: Stream is temporarily paused
• STOPPED: Stream has been stopped
• FINISHED: Stream completed (for convert mode)
• FAILED: Stream encountered an error

Error Handling

The API uses standard HTTP status codes and returns error details in the response body:

{
  "error": "Error message description"
}

Common status codes:

• 400: Invalid request
• 401: Invalid or missing API key
• 404: Stream not found
• 409: Cannot start stream (e.g., already running)
• 503: Service unavailable

See Also

• Stream API Examples - Programming examples in various languages
• Connection API Reference - API reference for connection management
• Understanding Streams - Conceptual guide to streams
• Stream Configuration Guide - Detailed configuration options