Continuous Downloader (dccd.continuous_dl)

Module to download continuously data.

Module to download continuously data (orderbook, trades, etc.) and update automatically the database. Supports Binance, Bitfinex, Bitmex, Bybit, Kraken, and OKX exchanges via WebSocket.

Contents:

Streams real-time data (trades, order book, OHLCV) via WebSocket with automatic reconnection. Each exchange provides a downloader class and convenience functions. The time_step parameter controls how often data is snapshotted to disk (in seconds); until sets the total run duration (in seconds, or as an absolute timestamp).

Convenience functions vs. class API

Convenience functions

Downloader class

Configuration

Fixed parameters

Custom callbacks via set_process_data()

Save format

Default CSV or Parquet

Any IODataBase saver

Typical use

Quick one-shot script

Embedded in a long-running service

Convenience function (simplest):

from dccd.continuous_dl import get_data_binance

# Stream BTC/USDT trades + book, save every 60 s for 1 h
get_data_binance('/data/crypto/', pair='BTCUSDT',
                 time_step=60, until=3600, form='parquet')

Class-based API (full control):

from dccd.continuous_dl import DownloadBinanceData
from dccd.tools.io import IODataBase

dl = DownloadBinanceData(pair='BTCUSDT', time_step=60, until=3600)
dl.set_trades_saver(IODataBase('/data/crypto/trades', method='parquet'))
dl.set_book_saver(IODataBase('/data/crypto/book', method='parquet'))
dl.run()

For exchange-specific details (WebSocket URL, available channels, pair format) see the exchange pages in the sidebar.

Downloader classes

binance.DownloadBinanceData([pair, ...])

Download data continuously from Binance via combined WebSocket streams.

bitfinex.DownloadBitfinexData([time_step, ...])

Basis object to download data from a stream websocket client API.

bitmex.DownloadBitmexData([time_step, until])

Basis object to download data from a stream websocket client API.

bybit.DownloadBybitData([pair, time_step, ...])

Download data continuously from Bybit via WebSocket v5.

kraken.DownloadKrakenData([pair, time_step, ...])

Download data continuously from Kraken via WebSocket v2.

okx.DownloadOKXData([pair, time_step, ...])

Download data continuously from OKX via WebSocket v5.

Convenience functions

binance.get_data_binance(path[, pair, ...])

Download order book and trades data from Binance via WebSocket.

binance.get_orderbook_binance(path[, pair, ...])

Download order book data from Binance via WebSocket.

binance.get_trades_binance(path[, pair, ...])

Download trades data from Binance via WebSocket.

bitfinex.get_data_bitfinex(channel, process_func)

Download data from Bitfinex exchange and update the database.

bitfinex.get_orderbook_bitfinex(symbol[, ...])

Download reconstructed order book from Bitfinex exchange.

bitfinex.get_trades_bitfinex(symbol[, ...])

Download trades tick by tick from Bitfinex exchange.

bitmex.get_data_bitmex(process_func, *args)

Download data from Bitmex exchange and update the database.

bitmex.get_orderbook_bitmex(*args[, ...])

Download reconstructed order book from Bitmex exchange.

bitmex.get_trades_bitmex(*args[, time_step, ...])

Download trades tick by tick from Bitmex exchange.

bybit.get_data_bybit(path[, pair, ...])

Download order book and trades data from Bybit.

bybit.get_orderbook_bybit(path[, pair, ...])

Download order book data from Bybit.

bybit.get_trades_bybit(path[, pair, ...])

Download trades data from Bybit.

kraken.get_data_kraken(path[, pair, ...])

Download order book and trades data from Kraken via WebSocket.

kraken.get_orderbook_kraken(path[, pair, ...])

Download order book data from Kraken via WebSocket.

kraken.get_trades_kraken(path[, pair, ...])

Download trades data from Kraken via WebSocket.

okx.get_data_okx(path[, pair, time_step, ...])

Download order book and trades data from OKX via WebSocket.

okx.get_orderbook_okx(path[, pair, ...])

Download order book data from OKX via WebSocket.

okx.get_trades_okx(path[, pair, time_step, ...])

Download trades data from OKX via WebSocket.