Skip to content

Compute derived datasets

Jump to bottom
You Xie edited this page Apr 29, 2024 · 3 revisions

In this tutorial, you will learn how to compute derived datasets (i.e. further transformation), based on persisted raw vendor data. If your datasets haven't been mapped in yet, first read about How to Onboard a data vendor and catalog.

1. Create a derived dataset class under dbmaster/derived

There's no fundament difference between a catalog class vs a derived class, except there is one more function called def compute() which allows you to define the transformation logic.

Take PmomBinance as an example, which computes Price Momentum (i.e. Percentage Change) across a variety of frequencies.

import sqlalchemy as sa
from dbmaster.derived import DerivedBase


engine = sa.create_engine(f"sqlite:///{config.derived.pmom.path}")
metadata = sa.MetaData()

class PmomBinance(DerivedBase):  # subclass from DerivedBase
    ...  # similar to catalog class, i.e. define table, set(), get()

    @classmethod
    def compute(cls, df: pd.DataFrame, **kwargs) -> pd.DataFrame
        ...  # your transformation/business logic
        return result

Explanation:

  • def compute() function takes in a pd.DataFrame sourced from raw vendor data, further transforms on it, and returns a pd.DataFrame to be called by def set() function to be saved into database.

2. Run compute command to transform and save derived dataset

(py3.12) PS C:\code\dbmaster> python -m dbmaster compute pmom --vendor=binance --period="['-30d', '-14d', '-7d', '-3d', '-1d', '-12h', '-8h', '-4h', '-1h', '-30m', '-15m', '-5m', '+1d']" --step='5m' --datefrom='2024-04-25 21:00:00' --symbol="['BTCUSDT', 'ETHUSDT', 'BNBUSDT', 'SOLUSDT', 'XRPUSDT', 'DOGEUSDT', 'ADAUSDT', 'SHIBUSDT', 'AVAXUSDT', 'DOTUSDT']"
2024-04-28 18:56:02 [INFO] dbmaster.command <command.py:79>: Computing Price Momentum for symbol=['BTCUSDT', 'ETHUSDT', 'BNBUSDT', 'SOLUSDT', 'XRPUSDT', 'DOGEUSDT', 'ADAUSDT', 'SHIBUSDT', 'AVAXUSDT', 'DOTUSDT'], from vendor='binance', with: period=['-30d', '-14d', '-7d', '-3d', '-1d', '-12h', '-8h', '-4h', '-1h', '-30m', '-15m', '-5m', '+1d'], step='5m', datefrom=Timestamp('2024-04-25 21:00:00'), dateto=None, kwargs={}
2024-04-28 18:56:06 [INFO] dbmaster.derived.pmom <pmom.py:169>: Inserted 0 rows to pmom_binance. Timestamp=2024-04-25 21:20:00
2024-04-28 18:56:06 [INFO] dbmaster.derived.pmom <pmom.py:169>: Inserted 0 rows to pmom_binance. Timestamp=2024-04-25 21:40:00
2024-04-28 18:56:06 [INFO] dbmaster.derived.pmom <pmom.py:169>: Inserted 0 rows to pmom_binance. Timestamp=2024-04-25 21:30:00
2024-04-28 18:56:06 [INFO] dbmaster.derived.pmom <pmom.py:169>: Inserted 0 rows to pmom_binance. Timestamp=2024-04-25 21:25:00
2024-04-28 18:56:06 [INFO] dbmaster.derived.pmom <pmom.py:169>: Inserted 0 rows to pmom_binance. Timestamp=2024-04-25 21:10:00
2024-04-28 18:56:06 [INFO] dbmaster.derived.pmom <pmom.py:169>: Inserted 0 rows to pmom_binance. Timestamp=2024-04-25 21:35:00
...
Done. Returned: [None, None, None, None, ...]

Since I've computed pmom today, DBMaster inserts zero rows (because primary keys are defined, it prohibits duplication). You will see some rows being saved into the database defined in config.toml

Clone this wiki locally