Bitfinex Continuous Downloader (dccd.continuous_dl.bitfinex)

Objects and functions to download data from Bitfinex exchange.

These functions and objects allow you to continuously download data and update your database.

High level API

get_data_bitfinex(channel, process_func, process_params={}, save_method='dataframe', io_params={}, time_step=60, until=None, path=None, **kwargs)

Download data from Bitfinex exchange and update the database.

Parameters:

channelstr, {‘book’, ‘book_raw’, ‘trades’, ‘trades_raw’}

Websocket channel to get data, by default data will be aggregated (OHLC for ‘trades’ and reconstructed orderbook for ‘book’), add ‘_raw’ to the channel to get raw data (trade tick by tick or each orders).

process_funccallable

Function to process and clean data before to be saved. Must take data in arguments and can take any optional keywords arguments, cf function exemples in dccd.process_data.

process_paramsdict, optional

Dictionary of the keyword arguments available to process_func, cf documentation into dccd.process_data.

save_method{‘DataFrame’, ‘SQLite’, ‘CSV’, ‘Excel’, ‘PostgreSQL’, ‘Oracle’, ‘MSSQL’, ‘MySQL’},

It will create an IODataBase object to save/update the database in the specified format save_method, default is ‘DataFrame’ it save as binary pl.DataFrame object. More informations are available into dccd.tools.io.

io_paramsdict, optional

Dictionary of the keyword arguments available to the dccd.tools.io.IODataBase callable method. Note: With SQL format some parameters are compulsory, see details into dccd.tools.io.

time_stepint, optional

Number of second between two snapshots of data, default 60 (1 minute).

untilint, optional

Number of seconds before stoping to download and update, default is None. If until equal 0 or None it means it never stop.

pathstr, optional

Path to save/update the database, default is None. If path is None, database is saved at the relative path ‘./database/bitfinex/channel’.

**kwargs

Any revelevant keyword arguments will be passed to the websocket connector, see Bitfinex API documentation [2] for more details.

Warning

‘book_raw’ and ‘trades_raw’ can be very memory expensive.

See also

process_data

function to process/clean data (set_marketdepth, set_ohlc, set_orders, set_marketdepth).

tools.io.IODataBase

object to save/update the database with respect to specified format.

References

[2]

https://docs.bitfinex.com/v2/docs/ws-public

get_orderbook_bitfinex(symbol, precision='P0', frequency='F0', lenght='25', time_step=60, until=None, path=None, save_method='dataframe', io_params={})

Download reconstructed order book from Bitfinex exchange.

get_trades_bitfinex(symbol, time_step=60, until=None, path=None, save_method='dataframe', io_params={})

Download trades tick by tick from Bitfinex exchange.

Low level API

class DownloadBitfinexData(time_step=60, until=3600, checkpoint_dir=None)

Bases: ContinuousDownloader

Basis object to download data from a stream websocket client API.

Parameters:

time_stepint or None, optional

Number of seconds between two snapshots of data, minimum is 1, default is 60 (one minute). Each time_step seconds data will be processed and pushed to the database. Pass None to receive data tick-by-tick without periodic aggregation.

untilint, optional

Number of seconds before stopping, or a future Unix timestamp at which to stop. Default is 3600 (one hour).

Attributes:

hoststr

Address of host to connect.

conn_pardict

Parameters of websocket connection.

wswebsockets.client.WebSocketClientProtocol

Connection with the websocket client.

is_connectbool

True if is connected, False otherwise.

tsint

Number of second between two snapshots of data.

tint

Current timestamp but rounded by ts.

untilint

Timestamp to stop to download data.

Methods

set_process_data(func, **kwargs)	Set processing function.
set_saver(call, **kwargs)	Set saver object to save data or update a database.
__call__(channel, **kwargs)	Open a websocket connection and save/update the database.

__call__(channel, **kwargs)

Open a websocket connection and save/update the database.

Run asynchronously two loops to get data from bitfinex websocket and save/update the database.

Parameters:

channel{‘book’, ‘book_raw’, ‘trades’, ‘trades_raw’}

Channel to get data, by default data will be aggregated (OHLC for ‘trades’ and reconstructed orderbook for ‘book’), add ‘_raw’ to the channel to get raw data (trade tick by tick or each orders).

**kwargs

Any revelevant keyword arguments will be passed to the websocket connector, see API documentation [1] for more details.

Warning

‘book_raw’ and ‘trades_raw’ can be very memory expensive.

References

[1]

https://docs.bitfinex.com/v2/docs/ws-public

set_process_data(func, **kwargs)

Set processing function.

Parameters:

funccallable

Function to process and clean data before to be saved. Must take data in arguments and can take any optional keywords arguments, cf exemples in dccd.process_data.

**kwargs

Any keyword arguments to be passed to func.

set_saver(call, **kwargs)

Set saver object to save data or update a database.

Parameters:

callcallable

Callable object to save data or update a database. Must take data in arguments and can take any optional keywords arguments, cf exemples in dccd.io_tools.

**kwargs

Any keyword arguments to be passed to call.

DownloadBitfinexData streams real-time trades and order book data from Bitfinex via WebSocket. No authentication is required.

Note

Bitfinex does not provide a native OHLCV WebSocket channel. OHLCV bars are instead computed from the trade stream by get_ohlc_bitfinex, which aggregates trades into candles at the requested time_step.

Exchange specifics

WebSocket URL	wss://api-pub.bitfinex.com/ws/2
Available channels	Trades (trades) and order book (book).
OHLCV	Derived from the trades stream via aggregation (not a native channel).
Authentication	Not required for public streams.

Quick example

from dccd.continuous_dl import DownloadBitfinexData, get_data_bitfinex
from dccd.tools.io import IODataBase

# Convenience function
get_data_bitfinex('/data/crypto/', pair='tBTCUSD',
                  time_step=60, until=3600, form='parquet')

# Class-based API
dl = DownloadBitfinexData(pair='tBTCUSD', time_step=60, until=3600)
dl.set_trades_saver(IODataBase('/data/crypto/trades', method='parquet'))
dl.set_book_saver(IODataBase('/data/crypto/book', method='parquet'))
dl.run()