From 16d4496857a19f2beb9e2516a76b7af2c37a162a Mon Sep 17 00:00:00 2001 From: derek ziehl Date: Fri, 17 Jul 2026 16:58:22 -0700 Subject: [PATCH] Add CloudFront operational review skill Read-only Amazon CloudFront operational review skill modeled on rds-operation-review and eks-operation-review. Reviews distributions against the AWS Well-Architected Framework across security (WAF, TLS, OAC), reliability (origin failover), performance, cost, and operational excellence, and generates a per-distribution report. Native read-only AWS APIs only; adds a least-privilege per-skill IAM block. Passes Agent Skill Eval (audit + functional + trigger). --- .../devops-agent-skill-policies.yaml | 53 +++ llms.txt | 1 + .../.skilleval.yaml | 3 + .../CHANGELOG.md | 5 + .../cloudfront-operational-review/README.md | 189 ++++++++ skills/cloudfront-operational-review/SKILL.md | 418 ++++++++++++++++++ .../evals/eval_queries.json | 11 + .../evals/evals.json | 73 +++ .../evals/files/distribution-context.json | 5 + .../evals/manual-test-results.md | 65 +++ .../references/findings-severity-catalog.md | 119 +++++ .../references/metrics-thresholds.md | 77 ++++ 12 files changed, 1019 insertions(+) create mode 100644 skills/cloudfront-operational-review/.skilleval.yaml create mode 100644 skills/cloudfront-operational-review/CHANGELOG.md create mode 100644 skills/cloudfront-operational-review/README.md create mode 100644 skills/cloudfront-operational-review/SKILL.md create mode 100644 skills/cloudfront-operational-review/evals/eval_queries.json create mode 100644 skills/cloudfront-operational-review/evals/evals.json create mode 100644 skills/cloudfront-operational-review/evals/files/distribution-context.json create mode 100644 skills/cloudfront-operational-review/evals/manual-test-results.md create mode 100644 skills/cloudfront-operational-review/references/findings-severity-catalog.md create mode 100644 skills/cloudfront-operational-review/references/metrics-thresholds.md diff --git a/cloudformation/devops-agent-skill-policies.yaml b/cloudformation/devops-agent-skill-policies.yaml index 6593576..2892864 100644 --- a/cloudformation/devops-agent-skill-policies.yaml +++ b/cloudformation/devops-agent-skill-policies.yaml @@ -20,6 +20,7 @@ Metadata: - EnableSupportCases - EnableRdsOperationReview - EnableEksOperationReview + - EnableCloudFrontOperationalReview - EnableInvestigationCostGuardrail - EnableEnrichWithSecurityAgent - EnableCrmInvestigationGuidelines @@ -67,6 +68,12 @@ Parameters: AllowedValues: ['true', 'false'] Default: 'true' + EnableCloudFrontOperationalReview: + Type: String + Description: CloudFront Operational Review skill. + AllowedValues: ['true', 'false'] + Default: 'true' + EnableInvestigationCostGuardrail: Type: String Description: Investigation Cost Guardrail skill. @@ -102,6 +109,7 @@ Conditions: SkillAwsHealthEvents: !Equals [!Ref EnableAwsHealthEvents, 'true'] SkillSupportCases: !Equals [!Ref EnableSupportCases, 'true'] SkillRdsOperationReview: !Equals [!Ref EnableRdsOperationReview, 'true'] + SkillCloudFrontOperationalReview: !Equals [!Ref EnableCloudFrontOperationalReview, 'true'] SkillInvestigationCostGuardrail: !Equals [!Ref EnableInvestigationCostGuardrail, 'true'] SkillServiceQuotaCheck: !Equals [!Ref EnableServiceQuotaCheck, 'true'] HasRegionRestriction: !Not [!Equals [!Join ['', !Ref AllowedRegions], '']] @@ -204,6 +212,50 @@ Resources: - logs:GetLogEvents Resource: '*' + # cloudfront-operational-review: read-only CloudFront config + WAF/ACM/CloudWatch/logs + # for the review. All actions are read-only; no Create/Update/Delete/Invalidation. + PolicyCloudFrontOperationalReview: + Type: AWS::IAM::Policy + Condition: SkillCloudFrontOperationalReview + Properties: + PolicyName: DevOpsAgentSkill-CloudFrontOperationalReview + Roles: + - !If [CreateNewRole, !Ref DevOpsAgentRole, !Ref ExistingRoleName] + PolicyDocument: + Version: '2012-10-17' + Statement: + - Sid: CloudFrontRead + Effect: Allow + Action: + - cloudfront:ListDistributions + - cloudfront:GetDistribution + - cloudfront:GetDistributionConfig + - cloudfront:ListCachePolicies + - cloudfront:ListOriginRequestPolicies + - cloudfront:ListResponseHeadersPolicies + - cloudfront:ListOriginAccessControls + - cloudfront:ListCloudFrontOriginAccessIdentities + - cloudfront:ListFieldLevelEncryptionConfigs + - cloudfront:ListKeyGroups + - cloudfront:ListFunctions + - cloudfront:ListVpcOrigins + - cloudfront:GetVpcOrigin + - cloudfront:GetMonitoringSubscription + - cloudfront:ListRealtimeLogConfigs + - cloudfront:ListTagsForResource + Resource: '*' + - Sid: CloudFrontWafAcmRead + Effect: Allow + Action: + - wafv2:GetWebACLForResource + - acm:DescribeCertificate + Resource: '*' + - Sid: CloudFrontCloudWatchLogsGetEvents + Effect: Allow + Action: + - logs:GetLogEvents + Resource: '*' + # investigation-cost-guardrail: adds pricing:GetProducts PolicyInvestigationCostGuardrail: Type: AWS::IAM::Policy @@ -295,6 +347,7 @@ Outputs: - aws-health-events: ${EnableAwsHealthEvents} (health:DescribeEventTypes) - support-cases: ${EnableSupportCases} (support:DescribeCommunications) - rds-operation-review: ${EnableRdsOperationReview} (rds:DownloadDBLogFilePortion, logs:GetLogEvents) + - cloudfront-operational-review: ${EnableCloudFrontOperationalReview} (cloudfront:List*/Get* read, wafv2:GetWebACLForResource, acm:DescribeCertificate, logs:GetLogEvents) - investigation-cost-guardrail: ${EnableInvestigationCostGuardrail} (pricing:GetProducts) - service-quota-check: ${EnableServiceQuotaCheck} (servicequotas:*, cloudwatch:GetMetricData/GetMetricStatistics) Skills covered by AIDevOpsAgentAccessPolicy (no extra policy needed): diff --git a/llms.txt b/llms.txt index f106887..bbf30e2 100644 --- a/llms.txt +++ b/llms.txt @@ -17,6 +17,7 @@ Skills can be used with these AWS DevOps Agent types: - [Support Cases Skill](skills/support-cases/SKILL.md): Searches and analyzes AWS Support cases to find historical incidents with similar symptoms, proven remediations, and recurring patterns - [EKS Operation Review Skill](skills/eks-operation-review/SKILL.md): Performs comprehensive Amazon EKS operational reviews aligned with the AWS EKS Best Practices Guide covering security, reliability, networking, and scalability - [RDS Operation Review Skill](skills/rds-operation-review/SKILL.md): Performs comprehensive Amazon RDS and Aurora operational reviews aligned with the AWS Well-Architected Framework covering security, reliability, performance, cost optimization, and backups +- [CloudFront Operational Review Skill](skills/cloudfront-operational-review/SKILL.md): Performs comprehensive Amazon CloudFront operational reviews aligned with the AWS Well-Architected Framework covering security (WAF, TLS, OAC), reliability (origin failover), caching/performance, cost optimization, and operational excellence - [CRM Production Investigation Guidelines Skill](skills/crm-production-investigation-guidelines/SKILL.md): Sample skill demonstrating how to write production investigation guidelines for the Incident Triage agent type, showing application-specific architecture, incident isolation rules, and structured investigation procedures - [Skip Scheduled Maintenance Skill](skills/skip-scheduled-maintenance/SKILL.md): Sample skill demonstrating how to skip low-priority incidents during a scheduled maintenance window, filtering MEDIUM and LOW severity alarms while preserving escalation for HIGH and CRITICAL incidents - [Enrich with AWS Security Agent Skill](skills/enrich-with-aws-security-agent/SKILL.md): Queries AWS Security Agent CloudWatch logs to retrieve code-level security findings (file, line number, vulnerability type) during incident investigations with potential security root causes diff --git a/skills/cloudfront-operational-review/.skilleval.yaml b/skills/cloudfront-operational-review/.skilleval.yaml new file mode 100644 index 0000000..686a9c7 --- /dev/null +++ b/skills/cloudfront-operational-review/.skilleval.yaml @@ -0,0 +1,3 @@ +audit: + ignore: + - STR-016 # README alongside SKILL.md is intentional diff --git a/skills/cloudfront-operational-review/CHANGELOG.md b/skills/cloudfront-operational-review/CHANGELOG.md new file mode 100644 index 0000000..dd213b1 --- /dev/null +++ b/skills/cloudfront-operational-review/CHANGELOG.md @@ -0,0 +1,5 @@ +# Changelog + +## 1.0.0 + +- Initial version diff --git a/skills/cloudfront-operational-review/README.md b/skills/cloudfront-operational-review/README.md new file mode 100644 index 0000000..201dab8 --- /dev/null +++ b/skills/cloudfront-operational-review/README.md @@ -0,0 +1,189 @@ +# CloudFront Operation Review — AWS DevOps Agent Skill + +A comprehensive Amazon CloudFront operational review skill for [AWS DevOps Agent](https://docs.aws.amazon.com/devopsagent/latest/userguide/about-aws-devops-agent.html). Conducts best-practices assessments aligned with the [Amazon CloudFront Developer Guide](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Introduction.html) and the [AWS Well-Architected Framework](https://docs.aws.amazon.com/wellarchitected/latest/framework/welcome.html) across security, reliability, performance, cost optimization, and operational excellence. Generates a shareable report artifact per distribution. + +## ⚠️ Non-Production Disclaimer + +This skill is provided as **sample code**. It is **not intended for production use without +additional review and testing**. Before relying on it: + +- Validate it in a **non-production environment first**. +- Review the IAM permissions below against your organization's security policies. +- Confirm the findings and thresholds match your operational requirements — the severity + levels in `references/findings-severity-catalog.md` are starting points, not mandates. + +The skill is strictly **read-only** — it never modifies, creates, or deletes any CloudFront, +WAF, ACM, or CloudWatch resource — but you remain responsible for validating its behavior in +your environment. + +## What It Does + +When activated via Chat, this skill instructs the DevOps Agent to: + +1. Discover CloudFront distributions in the account (CloudFront is a global service — no per-region loop). +2. Collect distribution config, origins, cache behaviors, policies, OAC/OAI, WAF association, ACM certificate, CloudFront Functions / Lambda@Edge, VPC origins, and tags. +3. Collect 7-day historical CloudWatch metrics (from `us-east-1`), standard access-log signals, and configuration-change notes. +4. Pull 3-month Cost Explorer data and proportionally estimate per-distribution monthly cost. +5. Analyze against the Well-Architected pillars (Security, Reliability, Performance, Cost Optimization, Operational Excellence). +6. Generate a shareable report artifact per distribution, named `cloudfront-review--.md`. + +All data is gathered through native, read-only AWS APIs (`cloudfront`, `cloudwatch`, `logs`, +`wafv2`, `acm`, `ce`). The skill does **not** use internal tooling, custom scripts, or +non-AWS MCP servers. + +## Agent Types + +This skill is intended for the following agent types (selected in the Operator Web App at upload time): + +- **On-demand** — conversational invocation in Chat ("review my CloudFront distribution", "CDN audit"). +- **Evaluation** — proactive operational improvement recommendations. + +Select **Generic** instead if you want the skill available to all agent types. + +## Prerequisites + +### 1. An AWS DevOps Agent Space with the target AWS account + +You need an existing [Agent Space](https://docs.aws.amazon.com/devopsagent/latest/userguide/getting-started-with-aws-devops-agent-creating-an-agent-space.html) with the target AWS account configured as a cloud source. + +### 2. IAM permissions for the DevOps Agent's primary cloud-source role + +The Agent Space's IAM role must have read access to the following. Most are covered by the +AWS managed policy [`AIDevOpsAgentAccessPolicy`](https://docs.aws.amazon.com/devopsagent/latest/userguide/aws-devops-agent-security-devops-agent-iam-permissions.html) — verify in your account before running the review. A per-skill policy block is provided in the repo's `cloudformation/devops-agent-skill-policies.yaml` (`EnableCloudFrontOperationalReview`). + +- `cloudfront:ListDistributions`, `cloudfront:GetDistribution`, `cloudfront:GetDistributionConfig` +- `cloudfront:ListCachePolicies`, `cloudfront:ListOriginRequestPolicies`, `cloudfront:ListResponseHeadersPolicies` +- `cloudfront:ListOriginAccessControls`, `cloudfront:ListCloudFrontOriginAccessIdentities` +- `cloudfront:ListFieldLevelEncryptionConfigs`, `cloudfront:ListKeyGroups`, `cloudfront:ListFunctions` +- `cloudfront:ListVpcOrigins`, `cloudfront:GetVpcOrigin` +- `cloudfront:GetMonitoringSubscription`, `cloudfront:ListRealtimeLogConfigs` +- `cloudfront:ListTagsForResource` +- `wafv2:GetWebACLForResource` (Scope=CLOUDFRONT, called in `us-east-1`) +- `acm:DescribeCertificate` (called in `us-east-1`) +- `cloudwatch:GetMetricData`, `cloudwatch:GetMetricStatistics`, `cloudwatch:DescribeAlarms`, `cloudwatch:DescribeAlarmsForMetric` +- `logs:DescribeLogGroups`, `logs:FilterLogEvents`, `logs:GetLogEvents` (if real-time/CloudWatch logs are used) +- `s3:GetObject`, `s3:ListBucket` (only if you want the agent to read standard access logs from the S3 log bucket) +- `ce:GetCostAndUsage` +- `tag:GetResources` (optional — cross-service tag reporting) + +The skill operates entirely in **read-only** mode: it never calls `Create*`, `Update*`, +`Delete*`, `Associate*`, `CreateInvalidation`, or `Tag*` CloudFront APIs. + +### 3. Additional CloudWatch metrics (recommended) + +For cache hit ratio, origin latency, and per-status error rates, enable a **monitoring +subscription** (additional metrics) on the distributions you want to review: + +- CloudFront console → distribution → **Monitoring** → enable additional metrics, or +- `aws cloudfront create-monitoring-subscription --distribution-id --monitoring-subscription RealtimeMetricsSubscriptionConfig={RealtimeMetricsSubscriptionStatus=Enabled}` + +Additional metrics incur CloudWatch charges. Without them, the skill still produces a +complete report from default metrics, configuration, and access logs — it just derives cache +hit ratio from logs instead of the `CacheHitRate` metric. + +### 4. Standard access logging (recommended) + +For log-pattern analysis (Step 6), enable standard access logging so the skill can analyze +`x-edge-result-type`, `sc-status`, and TLS fields. If the log bucket is not readable by the +agent role, the skill reports "access logs not reachable" and relies on CloudWatch metrics. + +## Uploading to AWS DevOps Agent + +> Reference: [Uploading a skill](https://docs.aws.amazon.com/devopsagent/latest/userguide/about-aws-devops-agent-devops-agent-skills.html#uploading-a-skill) + +### 1. Package the skill + +From the `skills/` directory in this repo: + +```bash +cd skills +zip -r cloudfront-operational-review.zip cloudfront-operational-review/ \ + -x 'cloudfront-operational-review/evals/*' +``` + +The resulting `cloudfront-operational-review.zip` contains: + +``` +cloudfront-operational-review/ +├── SKILL.md # frontmatter + skill instructions (required) +└── references/ + ├── metrics-thresholds.md + └── findings-severity-catalog.md +``` + +`evals/` is excluded from the upload to keep the zip small (it's only used for offline +evaluation). + +Constraints (enforced at upload time): + +- Total zip size ≤ **6 MB**. +- `SKILL.md` is required and must include `name` and `description` frontmatter. +- A `scripts/` directory is **not** allowed — uploads containing scripts are rejected. + +### 2. Upload via the Operator Web App + +1. Navigate to the **Skills** page in your Agent Space Operator Web App. +2. Click **Add skill** → **Upload skill**. +3. Drag and drop `cloudfront-operational-review.zip` (or browse to it). +4. Select agent types: **On-demand** and **Evaluation** (or leave **Generic** for all types). +5. Review the validation results. +6. Click **Upload**. + +## Usage + +In the DevOps Agent Chat, use natural language (don't name the skill — let the agent select it): + +- *"Run a CloudFront operational review for all distributions."* +- *"Review my CloudFront distribution `E1A2B3C4D5E6F7` for best practices."* +- *"Audit my CDN security — WAF, TLS, and origin access."* +- *"Why is my CloudFront cache hit ratio low?"* +- *"CloudFront health check for `d111111abcdef8.cloudfront.net`."* +- *"ORR for our production CloudFront distributions."* + +The agent will: + +- Collect all data automatically (no prompts for confirmation). +- Use only read-only AWS APIs — no mutating calls, no external scripts. +- Generate a report artifact per distribution, named `cloudfront-review--.md`. + +## Skill Contents + +``` +cloudfront-operational-review/ +├── SKILL.md # main skill instructions (with frontmatter) +├── README.md # this file +├── CHANGELOG.md # version history +├── .skilleval.yaml # eval config (ignores README.md in audit) +├── references/ +│ ├── metrics-thresholds.md # CloudWatch metric thresholds & severity rules +│ └── findings-severity-catalog.md # findings catalog mapped to Well-Architected pillars +└── evals/ # evaluation data (not included in upload zip) + ├── evals.json + ├── eval_queries.json + └── files/ + └── distribution-context.json +``` + +## Best-Practices Sections Covered + +| # | Pillar | Reference | +|---|--------|-----------| +| 1 | Security (WAF, TLS, OAC, FLE, signed URLs, geo) | [Security in CloudFront](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/security.html) | +| 2 | Reliability (origin failover, origin health, VPC origins) | [Origin failover](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/high_availability_origin_failover.html) | +| 3 | Performance (cache hit ratio, compression, HTTP/3, Origin Shield) | [Optimizing caching](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ConfiguringCaching.html) | +| 4 | Cost Optimization (price class, cache efficiency, idle distributions) | [Price classes](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/PriceClass.html) | +| 5 | Operational Excellence (alarms, logging, monitoring subscription, tagging) | [Monitoring CloudFront](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/monitoring-using-cloudwatch.html) | + +## Severity Definitions + +| Severity | Definition | SLA | +|----------|------------|-----| +| CRITICAL | Immediate risk to availability, security, or data integrity | 24–48 hours | +| HIGH | Significant gap that could lead to incidents | 1 week | +| MEDIUM | Notable improvement opportunity | 30 days | +| LOW | Minor optimization or hardening | When convenient | +| INFO | Observation, no action required | N/A | + +## License + +Apache-2.0. See the repository [LICENSE](../../LICENSE). diff --git a/skills/cloudfront-operational-review/SKILL.md b/skills/cloudfront-operational-review/SKILL.md new file mode 100644 index 0000000..8b2f73a --- /dev/null +++ b/skills/cloudfront-operational-review/SKILL.md @@ -0,0 +1,418 @@ +--- +name: cloudfront-operational-review +description: Comprehensive Amazon CloudFront operational review aligned with the AWS + Well-Architected Framework and CloudFront best practices. Use this skill when a user + asks to review, audit, or assess CloudFront distributions or a CDN for best-practices + compliance, security posture (WAF, TLS, OAC), reliability (origin failover), caching + and performance, cost optimization, or operational readiness. Triggers on requests + like "CloudFront review", "CDN audit", "review my distribution", "CloudFront best + practices", "CloudFront health check", "audit my CloudFront security", "why is my + CloudFront cache hit ratio low", or "ORR for CloudFront" — even when the user names + a distribution ID or domain without saying "CloudFront" explicitly. +metadata: + author: derekzie + version: "1.0.0" + aws-devops-agent-skills.agent-types: "Chat tasks, Evaluation" + aws-devops-agent-skills.aws-services: "Amazon CloudFront" + aws-devops-agent-skills.technical-domains: "Networking / Content Delivery" +--- + +# CloudFront Operational Review + +Conduct a comprehensive operational review of Amazon CloudFront distributions aligned +with the [AWS Well-Architected Framework](https://docs.aws.amazon.com/wellarchitected/latest/framework/welcome.html) +and the [Amazon CloudFront Developer Guide](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Introduction.html) +best-practices guidance (security, performance, reliability, and cost). + +This skill uses the **AWS CloudFront, CloudWatch, CloudWatch Logs, WAFv2, ACM, and Cost +Explorer APIs only** — no internal tooling, custom scripts, or non-AWS MCP servers. All +data is collected through native, **read-only** AWS APIs available to the DevOps Agent's +primary cloud-source role. The skill never calls a mutating (`Create*`, `Update*`, +`Delete*`, `Associate*`, `Tag*`) API. + +## When to Use + +Activate this skill when the user asks to: +- Review, audit, or assess CloudFront distributions or their CDN configuration +- Check CloudFront best-practices compliance +- Evaluate CloudFront security (WAF, TLS, OAC), reliability, caching/performance, or cost +- Perform a CloudFront operational readiness review (ORR) +- Investigate distribution health, high error rates, or low cache hit ratio +- Assess whether a distribution follows AWS Well-Architected guidance + +## Step 1: Identify Target Resources + +Ask the user which distributions to review. Accept: +- Specific distribution IDs (e.g. `E1A2B3C4D5E6F7`) or their CloudFront domain names + (e.g. `d111111abcdef8.cloudfront.net`) or alternate domain names (CNAMEs) +- "all distributions" in the account +- A tag-based scope (e.g. "all distributions tagged `Environment=prod`") + +If no scope is given, default to **all distributions in the account**. + +> **CloudFront is a global service.** Distributions are not regional — you do **not** +> iterate per region to list them. `cloudfront.*` calls are made against the global +> endpoint (SDKs route these through `us-east-1`). See "Known API Quirks". + +## Step 2: Discover CloudFront Distributions + +``` +cloudfront.ListDistributions # all distributions (paginate with Marker/NextMarker) +cloudfront.GetDistribution # per distribution — status, ARN, DomainName, ETag +cloudfront.GetDistributionConfig # per distribution — full editable config +``` + +Capture per distribution: +- `Id`, `ARN`, `DomainName`, `Status` (Deployed/InProgress), `Enabled`, `LastModifiedTime` +- `Aliases` (CNAMEs) and `PriceClass` (`PriceClass_All` / `PriceClass_200` / `PriceClass_100`) +- `DefaultRootObject`, `HttpVersion` (http1.1 / http2 / http2and3), `IsIPV6Enabled` +- `WebACLId` (WAF association — empty string means none) +- `DefaultCacheBehavior` and each `CacheBehaviors` item: + `ViewerProtocolPolicy`, `AllowedMethods`, `Compress`, `CachePolicyId`, + `OriginRequestPolicyId`, `ResponseHeadersPolicyId`, `FieldLevelEncryptionId`, + `TrustedKeyGroups` / `TrustedSigners`, `FunctionAssociations`, `LambdaFunctionAssociations` +- `Origins` and `OriginGroups`: per origin — `DomainName`, `OriginAccessControlId`, + legacy `S3OriginConfig.OriginAccessIdentity`, `CustomOriginConfig` + (`OriginProtocolPolicy`, `OriginSslProtocols`, timeouts), `VpcOriginConfig`, + `OriginShield`, `ConnectionAttempts`, `ConnectionTimeout` +- `ViewerCertificate`: `CloudFrontDefaultCertificate` (true = `*.cloudfront.net` default + cert), `ACMCertificateArn` / `IAMCertificateId`, `MinimumProtocolVersion`, `SSLSupportMethod` +- `Restrictions.GeoRestriction` (`RestrictionType`, `Items`) +- `Logging` (legacy standard/S3 access logging: `Enabled`, `Bucket`, `Prefix`) + +## Step 3: Discover Configuration Dependencies + +For each distribution, resolve the referenced policies and associated resources: + +``` +cloudfront.ListCachePolicies # resolve CachePolicyId → TTLs, key settings +cloudfront.ListOriginRequestPolicies # resolve OriginRequestPolicyId +cloudfront.ListResponseHeadersPolicies # resolve ResponseHeadersPolicyId (HSTS, CORS) +cloudfront.ListOriginAccessControls # OAC inventory (signing behavior) +cloudfront.ListCloudFrontOriginAccessIdentities # legacy OAI inventory +cloudfront.ListFieldLevelEncryptionConfigs # field-level encryption +cloudfront.ListKeyGroups # signed URL / signed cookie key groups +cloudfront.ListFunctions # CloudFront Functions (viewer request/response) +cloudfront.ListVpcOrigins # VPC origins inventory (regional resource) +cloudfront.GetVpcOrigin # per VPC origin — backing ALB/NLB/EC2 ARN, status +cloudfront.ListTagsForResource # tags — param is `Resource` = distribution ARN +``` + +For WAF and TLS: + +``` +wafv2.GetWebACLForResource # Scope=CLOUDFRONT, ResourceArn=, region us-east-1 +acm.DescribeCertificate # for ACMCertificateArn — MUST be called in us-east-1 for CloudFront +``` + +`LambdaFunctionAssociations` reference Lambda@Edge function versions (their ARNs live in +`us-east-1`). Record the associated function ARNs and event types (`viewer-request`, +`origin-request`, `origin-response`, `viewer-response`). + +## Step 4: Discover Logging and Monitoring Configuration + +``` +cloudfront.GetMonitoringSubscription # per distribution — are additional CloudWatch metrics enabled? +cloudfront.ListRealtimeLogConfigs # real-time log configs (Kinesis) and their fields +``` + +Also record from Step 2 whether **standard access logging** (`Logging.Enabled`) is on and +which S3 bucket receives logs. Newer distributions may log to CloudWatch Logs / S3 / +Firehose via the logging v2 configuration — note the destination if present. + +## Step 5: Collect CloudWatch Metrics (7-Day Historical) + +**One** `cloudwatch.GetMetricData` call per distribution. Namespace `AWS/CloudFront`. +`Period: 21600` (6 hours). `StartTime`: 7 days ago. `EndTime`: now. + +> **Metrics are global and live in `us-east-1`.** Always query CloudWatch in `us-east-1` +> regardless of where origins are. Default metrics use dimensions +> `DistributionId=` **and** `Region=Global`. Additional metrics use +> `DistributionId=` and `Region=Global` and are only populated when a +> **monitoring subscription** is enabled (Step 4). + +### 5.1 Default metrics (always available, dimension `DistributionId` + `Region=Global`) + +| id | metricName | stat | unit | +|----|------------|------|------| +| `requests` | Requests | Sum | count | +| `bytesDown` | BytesDownloaded | Sum | bytes | +| `bytesUp` | BytesUploaded | Sum | bytes | +| `err4xx` | 4xxErrorRate | Average | % | +| `err5xx` | 5xxErrorRate | Average | % | +| `errTotal` | TotalErrorRate | Average | % | + +### 5.2 Additional metrics (only if monitoring subscription enabled) + +| id | metricName | stat | unit | +|----|------------|------|------| +| `cacheHit` | CacheHitRate | Average | % | +| `originLatency` | OriginLatency | Average | ms | +| `err401` | 401ErrorRate | Average | % | +| `err403` | 403ErrorRate | Average | % | +| `err404` | 404ErrorRate | Average | % | +| `err502` | 502ErrorRate | Average | % | +| `err503` | 503ErrorRate | Average | % | +| `err504` | 504ErrorRate | Average | % | + +If additional metrics are not enabled, record that as an Operational Excellence finding and +derive cache hit ratio from standard access logs (Step 6) instead. + +Fetch `cloudwatch.DescribeAlarmsForMetric` (in `us-east-1`) for the key metrics +(`5xxErrorRate`, `TotalErrorRate`, `OriginLatency`, `CacheHitRate`) per distribution so the +report can flag missing alarms. + +Full threshold table: `references/metrics-thresholds.md`. + +## Step 6: Collect Logs (7-Day) + +### 6.1 Standard access logs +If `Logging.Enabled` is true, the logs land in the configured S3 bucket (fields include +`sc-status`, `x-edge-result-type`, `x-edge-response-result-type`, `time-taken`, +`cs-protocol`, `ssl-protocol`). If the agent has read access to the log bucket / a log +table (Athena, CloudWatch Logs), scan the 7-day window for these signals: + +| Pattern / field value | What it indicates | +|-----------------------|-------------------| +| `x-edge-result-type = Error` | request errored at the edge | +| `x-edge-result-type = Miss` (high ratio) | low cache efficiency | +| `x-edge-response-result-type = Error` | error returned to viewer | +| `sc-status` 502 / 503 / 504 | origin connection / gateway failures | +| `sc-status` 504 + `NonS3OriginCommError` | origin timeout / connection failure | +| `x-edge-result-type = OriginShieldHit` | Origin Shield effectiveness | +| `ssl-protocol = TLSv1 / TLSv1.1` | deprecated TLS in use by viewers | + +### 6.2 504 / origin-connection failure analysis +504s and origin errors typically map to one of: **origin timeout** (origin slower than +`OriginReadTimeout`), **connection refused / closed** (origin down or security-group +block), or **TLS negotiation failure** (origin cert / protocol mismatch, `OriginSslProtocols` +too restrictive). Correlate the 504/5xx rate from Step 5 with these log patterns and the +`OriginLatency` metric. + +### 6.3 Real-time logs +If a real-time log config exists (Step 4), note the destination (Kinesis) and fields; +real-time logs give per-request granularity for deeper latency/error analysis. + +## Step 7: Collect Events + +CloudFront configuration changes and WAF actions surface through CloudTrail (management +events) rather than a CloudFront-native event API. If CloudTrail access is available, look +(last 14 days) for `UpdateDistribution`, `CreateInvalidation`, WAF `UpdateWebACL`, and ACM +certificate changes to correlate config drift with metric/log anomalies. If CloudTrail is +not in scope, rely on `LastModifiedTime` from `GetDistribution` and note the limitation. + +## Step 8: Cost Data + +Once per review (not per distribution): + +``` +costexplorer.GetCostAndUsage # 3 months, GroupBy USAGE_TYPE, + # filter Service = "Amazon CloudFront" +``` + +CloudFront charges are driven by **data transfer out to the internet**, **HTTP/HTTPS +request counts**, and add-ons (real-time logs, Origin Shield, invalidations beyond the free +tier, additional CloudWatch metrics). Cost Explorer does not break down per distribution, so +attribute proportionally: + +1. Get the most recent full month total CloudFront spend by `USAGE_TYPE`. +2. Weight per distribution by 7-day `BytesDownloaded` + `Requests` share across all reviewed + distributions. +3. Note the estimate in the report with an "estimated" badge. + +Cross-reference `PriceClass` and `CacheHitRate`: a lower cache hit ratio raises origin fetch +and data-transfer cost; a broader price class raises edge cost. + +## Step 9: Analyze Against Best Practices + +Evaluate ALL collected data across the Well-Architected pillars and assign a severity to +every finding: CRITICAL, HIGH, MEDIUM, LOW, or INFO. The full findings catalog with +severity rationale is in `references/findings-severity-catalog.md`. + +### 9.1 Security +Ref: [Configuring secure access and restricting access to content](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/SecurityAndPrivateContent.html) + +- **WAF**: no `WebACLId` on an internet-facing distribution → HIGH (CRITICAL for auth'd or + sensitive apps). Confirm with `wafv2.GetWebACLForResource`. +- **TLS minimum protocol**: `MinimumProtocolVersion` of `SSLv3` / `TLSv1` / `TLSv1_2016` / + `TLSv1.1_2016` → HIGH. Recommend `TLSv1.2_2021` or higher. +- **Viewer protocol policy**: `allow-all` (permits plain HTTP) → HIGH. Use + `redirect-to-https` or `https-only`. +- **Origin access**: S3 origin reachable publicly / no OAC (or legacy OAI only) → + HIGH. Prefer **Origin Access Control (OAC)** over legacy OAI; direct public S3 origin → CRITICAL. +- **Default certificate + custom domain**: serving a custom CNAME on the default + `*.cloudfront.net` certificate → MEDIUM (should use ACM custom cert). +- **Field-level encryption**: sensitive form fields (PII/PCI) without field-level encryption + → MEDIUM where applicable. +- **Signed URLs / cookies**: private content served without `TrustedKeyGroups` → MEDIUM + where content is meant to be restricted. +- **Geo restriction**: regulatory/geo-locked content with `RestrictionType=none` → LOW/MEDIUM + where applicable. +- **Response headers policy**: no security headers (HSTS, `X-Content-Type-Options`, + `frame-options`) → LOW. + +### 9.2 Reliability +Ref: [Optimizing high availability with CloudFront origin failover](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/high_availability_origin_failover.html) + +- **Origin failover**: single custom origin with no **origin group** for a critical + distribution → MEDIUM (HIGH for tier-1). Origin groups enable automatic failover on 5xx/timeouts. +- **Origin health**: custom-origin `502/503/504` trend rising (Step 5) → HIGH. +- **VPC origin backing health**: VPC origin whose backing ALB/NLB is unhealthy or in a single + AZ → HIGH. Verify via `GetVpcOrigin` status. +- **Connection settings**: very low `ConnectionAttempts` (1) with no failover, or default + timeouts unsuited to a slow origin → MEDIUM. +- **5xx / origin error rate**: `5xxErrorRate` 7-day avg > 1% → HIGH, > 5% → CRITICAL. + +### 9.3 Performance +Ref: [Optimizing caching and availability](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ConfiguringCaching.html) + +- **Cache hit ratio**: `CacheHitRate` 7-day avg < 90% → MEDIUM, < 80% → HIGH. Investigate + cache policy TTLs, cache-key includes (unnecessary headers/cookies/query strings), and + cache-control at origin. +- **Compression**: `Compress=false` on text/JSON/HTML behaviors → MEDIUM (enables Gzip/Brotli). +- **HTTP version**: `HttpVersion` not `http2and3` (no HTTP/3) → LOW; HTTP/1.1 only → MEDIUM. +- **Origin Shield**: high-traffic distribution without Origin Shield → LOW/MEDIUM (reduces + origin load, improves cache consolidation). +- **Origin latency**: `OriginLatency` 7-day avg > 250 ms → MEDIUM, > 1000 ms → HIGH. +- **Price class vs latency**: `PriceClass_100`/`200` while serving a global audience → + LOW/MEDIUM (added latency for excluded regions) — balance against cost (Step 9.4). + +### 9.4 Cost Optimization +Ref: [Reducing the cost of your CloudFront distribution](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/PriceClass.html) + +- **Price class**: `PriceClass_All` where the audience is region-limited → MEDIUM (narrow it). +- **Low cache hit ratio → cost**: `CacheHitRate` < 90% raises origin fetch + data-transfer + cost → MEDIUM (linked to 9.3). +- **Unused / disabled distributions**: `Enabled=false` or `Requests` 7-day sum ≈ 0 → MEDIUM + (delete or consolidate). +- **Additional metrics on idle distributions**: monitoring subscription enabled on a + near-zero-traffic distribution → LOW. +- **Cost-allocation tags**: missing `Environment`, `Owner`, `CostCenter`, `Application` → + LOW. + +### 9.5 Operational Excellence +Ref: [Monitoring CloudFront](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/monitoring-using-cloudwatch.html) + +- **CloudWatch alarms**: missing alarms on `5xxErrorRate` / `TotalErrorRate`, + `OriginLatency`, `CacheHitRate` → MEDIUM each. +- **Standard logging**: `Logging.Enabled=false` (no access logs) → MEDIUM. +- **Additional metrics subscription**: `GetMonitoringSubscription` shows disabled → LOW/MEDIUM + (no `CacheHitRate` / `OriginLatency` / per-status error rates). +- **Real-time logs**: absent for a high-value distribution needing sub-minute visibility → LOW. +- **Tagging**: missing operational tags (`Environment`, `Owner`, `Runbook`, `OnCall`) → LOW. +- **Default root object**: not set on a website distribution → LOW (bare-domain requests 404). + +## Step 10: Generate Report + +Generate a separate shareable report artifact for **each distribution** reviewed. + +Artifact naming: `cloudfront-review--.md` +Example: `cloudfront-review-E1A2B3C4D5E6F7-2026-07-17.md` + +For each distribution, create the artifact as a Markdown document with: + +### Report Header +``` +# CloudFront Operational Review — +Account: | Service: Amazon CloudFront (global) | Date: +Domain: | Aliases: | Status: | Enabled: +Price Class: | HTTP: | WAF: +``` + +### Executive Summary +- Health: ✅ HEALTHY / ⚠️ WARNINGS / ❌ CRITICAL +- Finding counts by severity +- Top 3 critical/high items + +### Configuration Snapshot +| Item | Value | +| Origins | domain(s), type (S3/custom/VPC), OAC/OAI, protocol policy, timeouts | +| Origin groups | failover configured? members | +| Cache behaviors | count, viewer protocol policy, cache policy, compression | +| Security | WAF web ACL, TLS min version, viewer protocol, cert (ACM/default), FLE | +| Access control | signed URLs/cookies (key groups), geo restriction | +| Edge compute | CloudFront Functions, Lambda@Edge associations | +| Delivery | price class, HTTP version, IPv6, Origin Shield | +| Observability | standard logging, monitoring subscription, real-time logs, alarms | + +### Findings by Pillar +For each of Security, Reliability, Performance, Cost, Operational Excellence: + +| # | Finding | Severity | Current State | Recommendation | + +### CloudWatch Metrics (7-Day) +| Metric | Stat | 7-Day Value | Status | Finding | + +### Log Analysis (7-Day) +| Pattern / status | Occurrences or rate | Severity | Finding | +Include the 504 / origin-connection-failure breakdown (timeout vs refused vs TLS). + +### Configuration Change / Event Notes +`LastModifiedTime` and any CloudTrail-visible `UpdateDistribution` / WAF / ACM changes. + +### Cost Summary +- Latest-month estimated cost for this distribution (proportional split, "estimated" badge) +- Top 3 cost-optimization opportunities (linked to findings) + +### Priority Matrix +| # | Finding | Severity | Pillar | Effort | Impact | + +### Next Steps +- Immediate (CRITICAL/HIGH — 7 days) +- Short-term (MEDIUM — 30 days) +- Long-term (LOW — 90 days) + +### Appendix — Reference Links +- [Amazon CloudFront Developer Guide](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Introduction.html) +- [Security in CloudFront](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/security.html) +- [Restricting access to an S3 origin with OAC](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-restricting-access-to-s3.html) +- [Origin failover](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/high_availability_origin_failover.html) +- [Using Origin Shield](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/origin-shield.html) +- [Managed cache policies](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/using-managed-cache-policies.html) +- [Monitoring CloudFront with CloudWatch](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/monitoring-using-cloudwatch.html) +- [CloudFront pricing / price classes](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/PriceClass.html) +- [AWS WAF with CloudFront](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/distribution-web-aws-waf.html) +- [Well-Architected Framework](https://docs.aws.amazon.com/wellarchitected/latest/framework/welcome.html) + +## Severity Definitions + +| Severity | Definition | SLA | +|----------|------------|-----| +| CRITICAL | Immediate risk to availability, security, or data integrity | Fix within 24–48 hours | +| HIGH | Significant gap that could lead to incidents | Fix within 1 week | +| MEDIUM | Notable improvement opportunity | Plan within 30 days | +| LOW | Minor optimization or hardening | Address when convenient | +| INFO | Observation, no action required | N/A | + +## Known API Quirks (recorded so the agent doesn't trip on them) + +- **CloudFront is GLOBAL.** `cloudfront.*` calls use the global endpoint (SDKs sign against + `us-east-1`). Do **not** loop per region to list distributions. +- **Metrics live in `us-east-1`.** Query `AWS/CloudFront` in `us-east-1`. Default metrics use + dimensions `DistributionId` + `Region=Global`; additional metrics also use `Region=Global` + and require a monitoring subscription. +- **WAF for CloudFront is global scope.** Call `wafv2.GetWebACLForResource` with + `Scope=CLOUDFRONT` from `us-east-1`, passing the distribution ARN as `ResourceArn`. +- **ACM certs for CloudFront must be in `us-east-1`.** Call `acm.DescribeCertificate` in + `us-east-1` even if origins are elsewhere. +- **VPC origins are a REGIONAL resource** even though the distribution is global. The backing + ALB/NLB/EC2 lives in a specific region; resolve health there. +- **`cloudfront.ListTagsForResource` parameter is `Resource`** (the distribution ARN) — not + `ResourceName` (which is the RDS convention). Different services differ here. +- **Pagination uses markers.** `ListDistributions`, `ListVpcOrigins`, `ListCachePolicies`, + `ListRealtimeLogConfigs`, etc. paginate with `Marker` / `NextMarker`, not `NextToken`. +- **`WebACLId` empty string means no WAF** — treat `""` as "not attached", not "unknown". +- **`GetDistributionConfig` vs `GetDistribution`.** Config returns the editable body (+ETag); + `GetDistribution` adds status/ARN/domain metadata. Use both as needed; this is read-only. + +## Data Source Boundaries + +This skill explicitly does **not** call: +- Any mutating CloudFront API (`CreateDistribution`, `UpdateDistribution`, + `CreateInvalidation`, `DeleteDistribution`, `TagResource`, etc.). +- Internal Amazon tooling, custom scripts, or non-AWS MCP servers. + +It stays self-contained on the AWS DevOps Agent's primary cloud-source IAM role using +read-only `cloudfront`, `cloudwatch`, `logs`, `wafv2`, `acm`, and `ce` actions. If the agent +lacks access to the standard access-log S3 bucket, note the limitation in the report and +rely on CloudWatch metrics (Step 5) for error and cache analysis. diff --git a/skills/cloudfront-operational-review/evals/eval_queries.json b/skills/cloudfront-operational-review/evals/eval_queries.json new file mode 100644 index 0000000..ff7ba4d --- /dev/null +++ b/skills/cloudfront-operational-review/evals/eval_queries.json @@ -0,0 +1,11 @@ +[ + {"query": "Which skill would help me run a CloudFront operational review? Just name it; do not run it.", "should_trigger": true}, + {"query": "Is there a skill available for auditing my CDN / CloudFront distribution against best practices? Answer yes or no with the skill name; do not execute it.", "should_trigger": true}, + {"query": "Name the skill that covers CloudFront security (WAF, TLS, OAC) and cost optimization reviews. Do not run any audit.", "should_trigger": true}, + {"query": "Set up a new CloudFront distribution with an S3 origin and OAC", "should_trigger": false}, + {"query": "Create a CloudFront invalidation for /images/* on distribution E1A2B3C4D5E6F7", "should_trigger": false}, + {"query": "How much does CloudFront data transfer out cost per GB in the US?", "should_trigger": false}, + {"query": "Review my RDS instance prod-mysql for best practices", "should_trigger": false}, + {"query": "Write a Python script that sorts a list of numbers", "should_trigger": false}, + {"query": "What's the weather forecast for Sydney this weekend?", "should_trigger": false} +] diff --git a/skills/cloudfront-operational-review/evals/evals.json b/skills/cloudfront-operational-review/evals/evals.json new file mode 100644 index 0000000..8c8b1a1 --- /dev/null +++ b/skills/cloudfront-operational-review/evals/evals.json @@ -0,0 +1,73 @@ +[ + { + "id": "cloudfront-review-smoke-test", + "prompt": "Read distribution-context.json. List each distribution's id, domain name, price class, and account. No analysis needed.", + "expected_output": "Lists every distribution from files/distribution-context.json with its id, domainName, priceClass, and account exactly as defined in the file.", + "files": ["files/distribution-context.json"], + "assertions": [ + "matches regex /E[0-9A-Z]{6,}/", + "matches regex /[a-z0-9]+\\.cloudfront\\.net/", + "contains 'PriceClass'", + "matches regex /\\baccount\\b/" + ] + }, + { + "id": "cloudfront-review-first-resource-identification", + "prompt": "Read distribution-context.json. Identify the FIRST distribution listed and state the report artifact filename you would generate for it (use today's date in YYYY-MM-DD form). Do not run the audit.", + "expected_output": "Identifies the first distribution from files/distribution-context.json (its id and domain) and proposes an artifact name of the form cloudfront-review--.md.", + "files": ["files/distribution-context.json"], + "assertions": [ + "matches regex /cloudfront-review-[A-Za-z0-9]+-\\d{4}-\\d{2}-\\d{2}\\.md/", + "matches regex /E[0-9A-Z]{6,}/" + ] + }, + { + "id": "cloudfront-review-data-source-priority", + "prompt": "According to the skill, which AWS APIs does the CloudFront operational review use to collect data? Name at least three specific API calls or namespaces the skill mentions. Confirm whether the skill uses any mutating (write) CloudFront APIs. No AWS access required.", + "expected_output": "States the skill uses CloudFront, CloudWatch, CloudWatch Logs, WAFv2, ACM, and Cost Explorer APIs and is strictly read-only (no Create/Update/Delete/CreateInvalidation). Names specific calls such as cloudfront.ListDistributions, cloudfront.GetDistributionConfig, cloudwatch.GetMetricData, or wafv2.GetWebACLForResource.", + "files": [], + "assertions": [ + "contains 'CloudFront' or contains 'cloudfront'", + "contains 'CloudWatch' or contains 'cloudwatch'", + "contains 'ListDistributions' or contains 'GetDistributionConfig' or contains 'GetMetricData' or contains 'GetWebACLForResource'", + "contains 'read-only' or contains 'read only' or contains 'not' or contains 'no '" + ] + }, + { + "id": "cloudfront-review-severity-definitions", + "prompt": "List the five severity levels the skill uses for findings, in order from most to least urgent. No AWS access required.", + "expected_output": "Lists CRITICAL, HIGH, MEDIUM, LOW, INFO in that order.", + "files": [], + "assertions": [ + "contains 'CRITICAL'", + "contains 'HIGH'", + "contains 'MEDIUM'", + "contains 'LOW'", + "contains 'INFO'" + ] + }, + { + "id": "cloudfront-review-pillars", + "prompt": "List the Well-Architected pillars the skill evaluates during a CloudFront operational review. No AWS access required.", + "expected_output": "Mentions Security, Reliability, Performance, Cost Optimization, and Operational Excellence.", + "files": [], + "assertions": [ + "contains 'Security' or contains 'security'", + "contains 'Reliability' or contains 'reliability'", + "contains 'Performance' or contains 'performance'", + "contains 'Cost' or contains 'cost'", + "contains 'Operational' or contains 'operational'" + ] + }, + { + "id": "cloudfront-review-global-service-quirk", + "prompt": "According to the skill, CloudFront is a global service. In which AWS region do its CloudWatch metrics live, and does the skill loop per region to list distributions? No AWS access required.", + "expected_output": "States that CloudFront metrics live in us-east-1 and that the skill does NOT loop per region to list distributions because CloudFront is global.", + "files": [], + "assertions": [ + "contains 'us-east-1'", + "contains 'global' or contains 'Global'", + "contains 'not' or contains 'no ' or contains \"don't\" or contains 'does not'" + ] + } +] diff --git a/skills/cloudfront-operational-review/evals/files/distribution-context.json b/skills/cloudfront-operational-review/evals/files/distribution-context.json new file mode 100644 index 0000000..b1fb1c2 --- /dev/null +++ b/skills/cloudfront-operational-review/evals/files/distribution-context.json @@ -0,0 +1,5 @@ +{ + "distributions": [ + { "id": "E1A2B3C4D5E6F7", "domainName": "d111111abcdef8.cloudfront.net", "aliases": ["www.example.com"], "priceClass": "PriceClass_All", "enabled": true, "account": "$accountId" } + ] +} diff --git a/skills/cloudfront-operational-review/evals/manual-test-results.md b/skills/cloudfront-operational-review/evals/manual-test-results.md new file mode 100644 index 0000000..69b9074 --- /dev/null +++ b/skills/cloudfront-operational-review/evals/manual-test-results.md @@ -0,0 +1,65 @@ +# Manual DevOps Agent Test Results — cloudfront-operational-review + +Manual validation in an AWS DevOps Agent Space against a live CloudFront distribution, +complementing the automated Agent Skill Eval results (`report.json`, root `"passed": true`). + +- **Date:** 2026-07-17 +- **Environment:** DevOps Agent Space, On-demand (Chat) agent type +- **Target:** one live CloudFront distribution (ID redacted as `E1XXXXXXXXXXXX`) +- **Method:** each prompt run in a fresh chat; skill never named explicitly (auto-invocation only) + +## Positive cases (skill expected to load and run) + +| Prompt | Skill loaded? | Behavior | Result | +|--------|:-------------:|----------|:------:| +| "Review my CloudFront distribution for best practices" | Yes | Ran the review end-to-end as expected | ✅ Pass | +| "Do a health check on our CDN — is it secure and cached well?" | Yes | Loaded the skill and ran the review as expected | ✅ Pass | + +## Negative cases (skill expected NOT to hijack the response) + +| Prompt | Skill loaded? | Behavior | Result | +|--------|:-------------:|----------|:------:| +| "Set up a new CloudFront distribution with an S3 origin and OAC" | Yes | Gave correct setup guidance/instructions, did not force a review, then *offered* to run a COR | ✅ Acceptable | +| "Create a CloudFront invalidation for /images/*" | No | Attempted the tool call and failed as expected (read-only scope); no skill load | ✅ Pass | +| "How much does CloudFront data transfer out cost per GB in the US?" | No | Answered as a pricing question; did not trigger the skill | ✅ Pass | +| "My CloudFront distribution returns 403 on one path — help me fix the bucket policy" | No | Handled as targeted debugging; did not trigger the skill | ✅ Pass | + +## Notes + +- **"Set up a new distribution…"** — the skill loaded, but behavior was correct: the agent + delivered setup guidance for the user's actual request and only *offered* a CloudFront + Operational Review as an opt-in follow-up. It did not force a review or cross the read-only + boundary. This is consistent with the Agent Skills progressive-disclosure model (load when + possibly relevant, then decide how to act). Treated as acceptable, not a hijack. +- **"Create a CloudFront invalidation…"** — the agent attempted the operation and failed as + expected; the skill is strictly read-only and does not perform mutating actions such as + `CreateInvalidation`. No skill load occurred. +- **Environment difference vs. automated eval:** in the Claude-CLI eval harness the + "set up a new distribution" query scored as not-triggered (`signal=none`); the DevOps Agent + loader is more eager to load the skill but still behaved correctly. Both environments + produced correct outcomes. + +## Overall + +Auto-invocation is reliable on genuine review/health-check prompts, and near-miss prompts +(create/provision, invalidation, pricing, single-path debugging) do not cause the skill to +hijack the response. Behavior is consistent with the automated trigger eval (pass rate 1.0). +No changes required; ship as-is. + +## Automated Agent Skill Eval — summary + +Run with the [Agent Skill Eval](https://github.com/aws-samples/sample-agent-skill-eval) +framework (`skill-eval report`, functional + trigger with `--runs-trigger 1`): + +| Section | Result | +|---------|--------| +| Audit | ✅ Passed — 0 critical, 0 warning (INFO only: external AWS doc links; SKILL.md ~5.4k tokens vs 5k guideline) | +| Functional | ✅ Passed — classified PARETO_BETTER (quality improves while cost drops) | +| Trigger | ✅ Passed — pass rate 1.0 across the should-not-trigger near-misses | +| **Root** | ✅ **`"passed": true`** | + +The generated eval artifacts (`benchmark.json`, `report.json`, `trigger_report.json`) are +intentionally not committed: they record raw model transcripts from the eval environment and +local filesystem paths. Only the eval inputs (`evals.json`, `eval_queries.json`, `files/`) are +committed, matching the convention of the other operation-review skills. Maintainers can +reproduce the run from those inputs. diff --git a/skills/cloudfront-operational-review/references/findings-severity-catalog.md b/skills/cloudfront-operational-review/references/findings-severity-catalog.md new file mode 100644 index 0000000..62c8eb8 --- /dev/null +++ b/skills/cloudfront-operational-review/references/findings-severity-catalog.md @@ -0,0 +1,119 @@ +# CloudFront Findings & Severity Catalog + +Organized by AWS Well-Architected pillar. Maps directly to the checks in `SKILL.md` Step 9. +Each row: the check, the condition that raises it, and the assigned severity. Severities are +starting points — escalate for tier-1 / regulated (PCI, HIPAA) workloads. + +## Severity Definitions + +| Severity | Definition | SLA | +|----------|------------|-----| +| CRITICAL | Immediate risk to availability, security, or data integrity | 24–48 hours | +| HIGH | Significant gap that could lead to incidents | 1 week | +| MEDIUM | Notable improvement opportunity | 30 days | +| LOW | Minor optimization or hardening | When convenient | +| INFO | Observation, no action required | N/A | + +## Security + +| Check | Condition | Severity | +|---|---|---| +| WAF web ACL attached | `WebACLId` empty on internet-facing distribution | HIGH (CRITICAL for auth'd/sensitive apps) | +| Public S3 origin | S3 origin with no OAC/OAI and public bucket policy | CRITICAL | +| Origin Access Control | S3 origin using legacy OAI instead of OAC | MEDIUM | +| TLS minimum protocol | `MinimumProtocolVersion` ≤ `TLSv1.1_2016` | HIGH | +| TLS minimum protocol | `MinimumProtocolVersion` = `TLSv1.2_2018/2019` (not 2021) | LOW | +| Viewer protocol policy | `allow-all` (plain HTTP permitted) | HIGH | +| Custom SSL certificate | Custom CNAME served on default `*.cloudfront.net` cert | MEDIUM | +| Field-level encryption | Sensitive form fields (PII/PCI) without FLE | MEDIUM | +| Signed URLs / cookies | Private content without `TrustedKeyGroups` | MEDIUM | +| Geo restriction | Geo/regulatory-locked content with `RestrictionType=none` | LOW–MEDIUM | +| Security response headers | No response headers policy (HSTS, X-Content-Type-Options) | LOW | +| Lambda@Edge / Functions | Untracked edge functions altering security behavior | INFO–LOW | + +## Reliability + +| Check | Condition | Severity | +|---|---|---| +| Origin failover | Critical distribution, single origin, no origin group | MEDIUM (HIGH for tier-1) | +| Origin error trend | Custom-origin 502/503/504 rising over 7 days | HIGH | +| 5xx error rate | `5xxErrorRate` 7-day avg > 1% | HIGH | +| 5xx error rate | `5xxErrorRate` 7-day avg > 5% | CRITICAL | +| VPC origin backing health | Backing ALB/NLB unhealthy or single-AZ | HIGH | +| Connection settings | `ConnectionAttempts=1` with no failover | MEDIUM | +| Origin timeouts | Default timeouts unsuited to a slow origin | MEDIUM | +| Distribution status | Stuck `InProgress` well beyond deploy window | MEDIUM | + +## Performance + +| Check | Condition | Severity | +|---|---|---| +| Cache hit ratio | `CacheHitRate` 7-day avg < 90% | MEDIUM | +| Cache hit ratio | `CacheHitRate` 7-day avg < 80% | HIGH | +| Compression | `Compress=false` on text/JSON/HTML behaviors | MEDIUM | +| HTTP version | `HttpVersion` not `http2and3` (no HTTP/3) | LOW | +| HTTP version | HTTP/1.1 only | MEDIUM | +| Origin Shield | High-traffic distribution without Origin Shield | LOW–MEDIUM | +| Origin latency | `OriginLatency` 7-day avg > 250 ms | MEDIUM | +| Origin latency | `OriginLatency` 7-day avg > 1000 ms | HIGH | +| Cache key hygiene | Cache policy forwards unnecessary headers/cookies/query strings | MEDIUM | + +## Cost Optimization + +| Check | Condition | Severity | +|---|---|---| +| Price class | `PriceClass_All` for region-limited audience | MEDIUM | +| Cache hit ratio → cost | `CacheHitRate` < 90% raising origin/data-transfer cost | MEDIUM | +| Unused distribution | `Enabled=false` or `Requests` 7-day sum ≈ 0 | MEDIUM | +| Additional metrics on idle | Monitoring subscription on near-zero-traffic distribution | LOW | +| Cost-allocation tags | Missing `Environment`, `Owner`, `CostCenter`, `Application` | LOW | + +## Operational Excellence + +| Check | Condition | Severity | +|---|---|---| +| CloudWatch alarms | Missing alarm on `5xxErrorRate` / `TotalErrorRate` | MEDIUM | +| CloudWatch alarms | Missing alarm on `OriginLatency` / `CacheHitRate` | MEDIUM | +| Standard logging | `Logging.Enabled=false` (no access logs) | MEDIUM | +| Additional metrics | `GetMonitoringSubscription` disabled | LOW–MEDIUM | +| Real-time logs | Absent for high-value distribution needing sub-minute visibility | LOW | +| Default root object | Not set on a website distribution | LOW | +| Operational tags | Missing `Environment`, `Owner`, `Runbook`, `OnCall` | LOW | + +## Best-Practices Checklist (quick pass/fail) + +### Security +- [ ] WAF web ACL attached to internet-facing distributions +- [ ] `MinimumProtocolVersion` ≥ `TLSv1.2_2021` +- [ ] `ViewerProtocolPolicy` = `redirect-to-https` or `https-only` +- [ ] S3 origins locked down with **OAC** (no public bucket access; OAI only as legacy) +- [ ] Custom domains use an ACM certificate (not the default cert) +- [ ] Field-level encryption for sensitive fields where applicable +- [ ] Signed URLs/cookies (key groups) for private content +- [ ] Geo restriction configured where required +- [ ] Response headers policy adds HSTS + security headers + +### Reliability +- [ ] Origin groups configured for automatic failover on critical distributions +- [ ] VPC origin backing target healthy and multi-AZ +- [ ] `5xxErrorRate` within error budget (< 1%) +- [ ] Sensible `ConnectionAttempts` / timeouts for the origin type + +### Performance +- [ ] `CacheHitRate` ≥ 95% +- [ ] Compression enabled on compressible content types +- [ ] `HttpVersion` = `http2and3` +- [ ] Origin Shield enabled for high-traffic / cache-consolidation needs +- [ ] Cache key scoped to only necessary headers/cookies/query strings + +### Cost Optimization +- [ ] Price class matches audience geography +- [ ] Cache hit ratio high enough to minimize origin fetch cost +- [ ] No enabled-but-idle distributions +- [ ] Cost-allocation tags applied + +### Operational Excellence +- [ ] Alarms on `5xxErrorRate`, `TotalErrorRate`, `OriginLatency`, `CacheHitRate` +- [ ] Standard access logging enabled +- [ ] Monitoring subscription (additional metrics) enabled +- [ ] Operational tags present (`Environment`, `Owner`, `Runbook`, `OnCall`) diff --git a/skills/cloudfront-operational-review/references/metrics-thresholds.md b/skills/cloudfront-operational-review/references/metrics-thresholds.md new file mode 100644 index 0000000..9c9bf45 --- /dev/null +++ b/skills/cloudfront-operational-review/references/metrics-thresholds.md @@ -0,0 +1,77 @@ +# CloudFront CloudWatch Metrics Thresholds Reference + +All metrics retrieved via a single `cloudwatch.GetMetricData` call per distribution over a +7-day window with `Period=21600` (6h), in **`us-east-1`** (CloudFront metrics are global and +live in us-east-1). Severity reflects sustained values, not single spikes. + +## Default Metrics (Namespace `AWS/CloudFront`, dimensions `DistributionId` + `Region=Global`) + +Always available for every distribution at no additional charge. + +| Metric | Stat | Normal | Warning | Critical | Finding | +|---|---|---|---|---|---| +| Requests | Sum | traffic baseline | — | ≈ 0 on an enabled dist | Idle distribution — candidate to disable/delete | +| BytesDownloaded | Sum | traffic baseline | — | — | Data-transfer cost driver; compare to cache hit ratio | +| BytesUploaded | Sum | traffic baseline | — | — | Upload-heavy patterns; validate methods allowed | +| 4xxErrorRate | Average | < 1% | > 2% | > 10% | Client errors — check auth, signed URLs, 403/404 causes | +| 5xxErrorRate | Average | < 0.5% | > 1% | > 5% | Origin/edge failures — investigate origin health & failover | +| TotalErrorRate | Average | < 1% | > 3% | > 10% | Overall error budget breach | + +## Additional Metrics (require a monitoring subscription; dimensions `DistributionId` + `Region=Global`) + +Only populated when `cloudfront.GetMonitoringSubscription` shows additional metrics enabled +(these incur CloudWatch charges). If disabled, this is an Operational Excellence finding and +cache hit ratio must be derived from standard access logs. + +| Metric | Stat | Normal | Warning | Critical | Finding | +|---|---|---|---|---|---| +| CacheHitRate | Average | > 95% | < 90% | < 80% | Low cache efficiency — review cache policy TTLs & cache key | +| OriginLatency | Average | < 100 ms | > 250 ms | > 1000 ms | Slow origin — check origin scaling, Origin Shield, region | +| 401ErrorRate | Average | ≈ 0 | > 1% | > 5% | Auth failures at viewer | +| 403ErrorRate | Average | < 1% | > 2% | > 10% | Access denied — OAC/OAI, signed URL, WAF blocks, S3 perms | +| 404ErrorRate | Average | < 2% | > 5% | > 15% | Missing objects — default root object, routing, cache key | +| 502ErrorRate | Average | ≈ 0 | > 0.5% | > 2% | Bad gateway — origin TLS/protocol mismatch | +| 503ErrorRate | Average | ≈ 0 | > 0.5% | > 2% | Origin overloaded / capacity exceeded | +| 504ErrorRate | Average | ≈ 0 | > 0.5% | > 2% | Origin timeout / connection failure (NonS3OriginCommError) | + +## 504 / Origin-Connection Failure Triage (map metric spikes to root cause) + +| Symptom | Likely root cause | Where to confirm | +|---|---|---| +| 504 + high OriginLatency | Origin slower than `OriginReadTimeout` | Origin app latency, raise timeout or scale origin | +| 504 + connection refused/closed in logs | Origin down or security group blocks CloudFront | Origin health, SG ingress, VPC origin backing target | +| 502 + TLS/handshake errors | Origin cert expired or `OriginSslProtocols` mismatch | Origin cert, `OriginProtocolPolicy`, SSL protocols | +| 503 sustained | Origin capacity exceeded | Origin autoscaling, Origin Shield to absorb load | + +## Standard Access Log Signals (7-day scan) + +| Field / value | Severity if prevalent | Action | +|---|---|---| +| `x-edge-result-type = Error` | HIGH | Correlate with 5xx metric and origin logs | +| `x-edge-result-type = Miss` (high ratio) | MEDIUM | Improve cacheability (TTL, cache key) | +| `sc-status` 502/503/504 | HIGH | Origin failover / timeout / capacity review | +| `x-edge-result-type = OriginShieldHit` present | INFO | Origin Shield working as intended | +| `ssl-protocol = TLSv1 / TLSv1.1` | MEDIUM | Deprecated viewer TLS — raise `MinimumProtocolVersion` | + +## Alarm Coverage Expectations + +The skill flags any of these as **MEDIUM** when missing +(`cloudwatch.DescribeAlarmsForMetric` in `us-east-1` returns nothing for the distribution + +metric): + +| Metric | Threshold (suggested) | +|---|---| +| 5xxErrorRate | > 1% for 5 minutes | +| TotalErrorRate | > 3% for 5 minutes | +| OriginLatency | > 1000 ms for 5 minutes (requires additional metrics) | +| CacheHitRate | < 80% for 15 minutes (requires additional metrics) | + +## Notes + +- Cache hit ratio, when computed from access logs, ≈ `Hit / (Hit + Miss + RefreshHit)` from + `x-edge-result-type`. `RefreshHit` counts as a partial hit (revalidated at origin but not + a full download). +- Error-rate metrics are percentages of total requests, already averaged by CloudFront — use + `Average` stat, not `Sum`. +- Zero-traffic windows can make error-rate `Average` misleading; always read error rates + alongside `Requests`.