[Feature Request] Cloud Compute Defined Environment Variables

[bestickley]

Description

I deploy my compute workloads to AWS Lambda and ECS Fargate. I use the AWS CDK to define my functions/containers, including environment variables. Example:

const fn = new nodejs.NodejsFunction(this, 'MyFunction', {
  entry: '/path/to/my/file.ts',
  environment: {
    ENVIRONMENT: "prod",
    FEATURE_FLAG_1: "1",
});
// external param
const apiUrlParam = ssm.StringParameter.fromStringParameterAttributes(this, 'ImportedApiUrl', {
      parameterName: '/config/api_url',
    });
fn.addEnvironment("MY_API_KEY", apiUrlParam.stringValue);
apiUrlParam.grantRead(fn);
// external secret
const mySecret = secretsmanager.Secret.fromSecretNameV2(
      this, 
      'ImportedSecret', 
      'my-api-credentials'
    );
fn.addEnvironment("MY_API_CREDS_ARN", mySecret.secretArn);
mySecret.grantRead(fn);

Now, I have to duplicate all of this in my local .env.schema for varlock to read env vars. BUT, what if varlock could resolve all of this for me?

Motivation

I don't want to have to duplicate env var configuration across my infrastructure-as-code + varlock.

Proposed Solution

.env.schema

# @plugin(@varlock/aws-lambda-env-plugin, "MyLambdaFunction")
# @initAws(region=$AWS_REGION, profile=$AWS_PROFILE)
# ---

# Environment from my AWS Lambda function are automatically resolved. Yay!

Notice the plugin needs to set multiple env vars. The plugin would look up the lambda function, then find the configuration that defines the environment variables, resolve them (sometimes reaching out to SSM or SecretsManager in the process), and set them locally.

Alternatives

Duplicate environment configuration.

Additional Information

Would be great to also have this for ECS (Containers Service). You'd want to resolve the ECS Task Definition which contains environment configuration.

[philmillman]

Yeah this should be doable. We already have the notion of platform defined env vars (https://github.com/dmno-dev/varlock/blob/main/packages/ci-env-info/src/platforms.ts) and there's an active PR that adds plugin schemas (#561). I'll chat with @theoephraim and address any gaps to make this happen.

[bestickley]

thanks for sharing those resources, Phil!

re: this example

# @import(public-schemas:platforms/vercel, VERCEL_ENV, VERCEL_URL)

How does user specify the particular vercel function/compute environment configuration that they want?

Also, for the feature request I'm hoping for, I think it's not true to say that varlock would be importing a schema (.env.schema), varlock would be importing environment configuration straight from AWS Lambda API so that does leave some ambiguity as to how varlock treats those variables. Are they sensitive? Any validation on them? Caching? You'd lose that unless you enumerated all of the variables you want to import in .env.schema, but then you lose some of the benefit I acknowledge above about double defining env var configuration.

[philmillman]

thanks for sharing those resources, Phil!

re: this example

# @import(public-schemas:platforms/vercel, VERCEL_ENV, VERCEL_URL)

How does user specify the particular vercel function/compute environment configuration that they want?

Also, for the feature request I'm hoping for, I think it's not true to say that varlock would be importing a schema (.env.schema), varlock would be importing environment configuration straight from AWS Lambda API so that does leave some ambiguity as to how varlock treats those variables. Are they sensitive? Any validation on them? Caching? You'd lose that unless you enumerated all of the variables you want to import in .env.schema, but then you lose some of the benefit I acknowledge above about double defining env var configuration.

The platform env vars are more so a static list of built-in env vars per platform, not dynamic like what you're proposing. I was just surfacing that as some semi-related prior art. But some of your lambda env might include static stuff like this right?

I know what you're proposing is less so about importing a schema and more about deriving it from the config of a particular lambda but ultimately it would need to map to something similar to a schema IIUC.

[bestickley]

But some of your lambda env might include static stuff like this right?

Yes, see defined runtime environment variables. Those would be helpful for varlock to include, but not as important IMO. Also would be tough, not sure how varlock would resolve those values.

but ultimately it would need to map to something similar to a schema IIUC.

Concur. Glad we're on same page :)

[theoephraim]

How do you set those values locally? Also pulling from aws?
If we are pulling from the deployed config, how do you change things while developing new features?

What about going the other way - having varlock be the source of truth, and then providing helpers to inject into / set things up in cdk?

[bestickley]

How do you set those values locally? Also pulling from aws?

Yes, with SSM Parameter Store and Secrets Manager Plugins. I try to keep all env vars in those places so I can references in my .env.schema.

If we are pulling from the deployed config, how do you change things while developing new features?

Change the value in the cdk ts file, and re-deploy. Or directly update in AWS.

What about going the other way - having varlock be the source of truth, and then providing helpers to inject into / set things up in cdk?

For most of my projects, we want the cloud provider to be the source of truth for env vars. So varlock + cdk will need to read/reference those values. So providing helpers for varlock to read from SSM Param Store / Secrets Manager, and then reference those in cdk seems a little unintuitive since CDK has native ways to import from SSM Param Store and Secrets Manager. Notice in my above example I also need to grant permissions.

Also, sometimes I have separate CDK apps / processes that generate SSM Parameter Store values or Secrets Manager Secrets.

[theoephraim]

Lets say you're working on a new feature and you add a new secret. You create it on SSM and add the wiring to your cdk ts file, but it's not yet deployed. You want to run locally first before deploying it. If our plugin is pulling secret config from the deployed lambda, we can't see it until you deploy it. The other option would be reading it from the cdk ts file somehow, but from what I can tell that would be the most tricky of our options here. Also not sure if you are using type generation but theres potentially a chicken-and-egg problem of generating the TS types for the env vars, which are fed into the code that will be deployed?

Maybe this is all moot - as you'd always be deploying to a dev environment, and your varlock config would point to the right env?

Sorry if my understanding here is a bit off as I don't use these tools regularly, just trying to understand the whole workflow.

[bestickley]

we can't see it until you deploy it.

I'll accept this inconvenience, for the convenience of not having to duplicate env var configuration.

The other option would be reading it from the cdk ts file somehow

Agreed, tricky and not worth it.

Also not sure if you are using type generation

I'm using varlock type generation for cdk code (not runtime, b/c I don't use varlock in runtime code), but I'm ok with temporary lint errors until I deploy.

Maybe this is all moot - as you'd always be deploying to a dev environment, and your varlock config would point to the right env?

I don't think it makes above challenges moot, but I mostly will be deploying to dev env and varlock config would point to right env.

---

My main pain point is I want to run my lambda function locally, but I don't want to have to copy and paste all of the env vars from the aws console into my local setup. I can already achieve this by defining .env.schema that's similar to environment key in cdk ts file, but it's duplicative env var definition. Which has been fine, but I'd love for it to be even better.