Can variables defined in ~/sling/env.yaml file be referenced by other task / replication yaml files

[ShahBinoy]

Hi @flarco,

The docs at https://docs.slingdata.io/sling-cli/run/configuration#object-patterns mentions that there are lot of inbuilt variables that can be used to add some dynamic patterns for naming objects, etc.

Is it possible to use the variables defined in the variables section of env.yaml file be accessed by all other task/replication/cli definitions via the {variable_name} convention ?

my env.yaml file

connections:
  S3_LAKE_ZONE_A:
    type: s3

variables:
  path_prefix: /dev-binoy/elt/data_landing

my replication.yaml file

source: MY_MYSQL
target: S3_LAKE_ZONE_A

defaults:
  object: {path_prefix}/{stream_schema}/{stream_table}/{YYYY}_{MM}_{DD}.parquet < - Use Path Prefix ?
  target_options:
    format: parquet

streams:

  # all tables in schema, except "forbidden_table"
  my_schema.*:
    object: {stream_schema}/{stream_table}/{YYYY}_{MM}_{DD}/
    target_options:
      file_max_rows: 400000 # will split files into folder

[flarco]

Hi yes, should work, try as below.

# env.yaml

connections:
  S3_LAKE_ZONE_A:
    type: s3

# this sets environment variables in sling process
variables:
  path_prefix: /dev-binoy/elt/data_landing

# replication.yaml

source: MY_MYSQL
target: S3_LAKE_ZONE_A

defaults:
  object: {path_prefix}/{stream_schema}/{stream_table}/{YYYY}_{MM}_{DD}.parquet < - Use Path Prefix ?
  target_options:
    format: parquet

streams:

  # all tables in schema, except "forbidden_table"
  my_schema.*:
    object: {stream_schema}/{stream_table}/{YYYY}_{MM}_{DD}/
    target_options:
      file_max_rows: 400000 # will split files into folder

env:
  # ${path_prefix} pulls from environment variables in sling process or env
  path_prefix: '${path_prefix}'

[flarco]

oh I see, you're defining in the env.yaml...

[flarco]

OK I updated my response

[matanper]

@flarco Hey this answer really helped me and I couldn't find reference for it in the documentation, worth adding it

[flarco]

Thanks, will do

[ShahBinoy]

Thank you, that resolves it