feat(parquet): variables metadata and download API for parquet outputs - #3367
feat(parquet): variables metadata and download API for parquet outputs#3367sylvlecl wants to merge 18 commits into
Conversation
- DB models for variables and areas - functions for extracting them from file outputs to be continued with: - some tests - implement the download API for just areas for now - implement other element types Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| from functools import cached_property | ||
| from pathlib import Path | ||
| from typing import Callable, Generic, Literal, Sequence, TypeAlias, TypeVar | ||
| from typing import Callable, Generic, Iterable, Literal, Sequence, TypeAlias, TypeVar |
Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>
Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>
Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>
Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>
Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>
Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>
Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>
Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>
| raise OutputSubFolderNotFound(output_dir.name, "economy|adequacy") | ||
|
|
||
|
|
||
| class FileOutput: |
There was a problem hiding this comment.
helper class to make output directories analyze easier, could probably be re-used in other places to simplify the code
|
|
||
| __tablename__ = "parquet_output" | ||
|
|
||
| id: Mapped[int] = mapped_column(BigInteger, primary_key=True, autoincrement=True) |
There was a problem hiding this comment.
as for study_data_id, the idea here is to introduce a technical ID for outputs, separate from study_id / output_id
This will be beneficial in the future for:
- more compact foreign keys in other tables
- possibility to rename the output if the user wants it
Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>
| mc_ind_vars: Mapped[list[int]] = mapped_column(Columns) | ||
|
|
||
|
|
||
| # TODO: add tables for other element types: links, thermal clusters, etc |
There was a problem hiding this comment.
Easier to focus only on areas for now, but others will need to follow of course
|
|
||
| simulation_range = _extract_simulation_range(dir_path) | ||
|
|
||
| # TODO: first, extract variables metadata to database |
There was a problem hiding this comment.
here we'll need to call the new functions from variables_parsing, and provide the variables index to the parquet creation function so that it uses the column that have been defined there
| return VariableDescription(db_var.name, db_var.unit, db_var.statistic_type) | ||
|
|
||
|
|
||
| def get_area_variables( |
There was a problem hiding this comment.
those functions may go into the repository impl
|
|
||
|
|
||
| @dataclass(frozen=True) | ||
| class OutputParsingResult: |
There was a problem hiding this comment.
We may not need that class if we dump to database the "partial" result one after the other
| yield parse_output_file(data_file, start_col) | ||
|
|
||
|
|
||
| def _extract_areas( |
There was a problem hiding this comment.
This is completely to be done:
now we will need to take care of writing the columns in the order that has been defined at variables parsing time, so that the parquet file is consistent with the variable definitions we have stored in database.
It may involve to re-write most of the code in this file unfortunately.
Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>
Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>
Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>
Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>
Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>
Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>
| from antarest.study.model import MatrixFrequency | ||
|
|
||
|
|
||
| def _parquet_file_name(element_type: ElementType, frequency: MatrixFrequency) -> str: |
1 participant
first pieces of implementation only for areas (values), to be generalized once it works for them:
to be continued for areas with:
parsing variables metadata
retrieving only the relevant data for each area
Then to be generalized with other element types