Input/Output tools (dccd.tools.io)

Tools and object to load, append and save different kind of database.

class IODataBase(path='./', method='csv')

Object to save a pl.DataFrame into different kind/format of database.

Parameters:

pathstr, optional

Path of the database, default is ‘./’ (current directory).

method{‘DataFrame’, ‘SQLite’, ‘CSV’, ‘Excel’, ‘parquet’, ‘PostgreSQL’, ‘Oracle’, ‘MSSQL’, ‘MySQL’}

Format of database, default is CSV.

Attributes:

pathstr

Path of the database.

methodstr

Kind/format of the database.

Methods

save_as_dataframe(new_data[, name, ext])	Append and save new_data as a pickle binary file.
save_as_sql(new_data[, table, name, ext, ...])	Append new_data into a SQL database via ADBC (PostgreSQL, MySQL, …).
save_as_sqlite(new_data[, table, name, ext])	Append new_data into a SQLite table (stdlib sqlite3, no extra deps).
save_as_csv(new_data[, name, ext])	Append new_data to a CSV file (header written once).
save_as_excel(new_data[, name, sheet_name, ext])	Append new_data to an Excel file (requires openpyxl).
save_as_parquet(new_data[, name, ext, ...])	Append new_data to a Parquet file.
__call__(new_data, **kwargs)	Append and save new_data in the configured format.

get_from_dataframe(name, ext='.dat')

Load data from pickle binary file.

get_from_sqlite(name, table='main_table', ext='.db')

Load data from a SQLite table.

save_as_csv(new_data, name=None, ext='.csv')

Append new_data to a CSV file (header written once).

save_as_dataframe(new_data, name=None, ext='.dat')

Append and save new_data as a pickle binary file.

save_as_excel(new_data, name=None, sheet_name='Sheet1', ext='.xlsx')

Append new_data to an Excel file (requires openpyxl).

Warning

Slow method, not recommended for large datasets.

save_as_parquet(new_data, name=None, ext='.parquet', compression='snappy')

Append new_data to a Parquet file.

save_as_sql(new_data, table='main_table', name=None, ext='', driver=None, username=None, password=None, host=None, port=None, **kwargs)

Append new_data into a SQL database via ADBC (PostgreSQL, MySQL, …).

Requires the appropriate ADBC driver, e.g. adbc_driver_postgresql.

save_as_sqlite(new_data, table='main_table', name=None, ext='.db')

Append new_data into a SQLite table (stdlib sqlite3, no extra deps).

get_df(path, name, ext='')

Load a DataFrame from a pickle binary file.

Parameters:

path, name, extstr

Directory path, file name, and optional extension.

Returns:

pl.DataFrame

Loaded DataFrame, or an empty DataFrame if the file is not found.

save_df(df, path, name, ext='')

Save a DataFrame as a pickle binary file.

Parameters:

dfpl.DataFrame

DataFrame to persist.

path, name, extstr

Directory path, file name, and optional extension.