Skip to content

Commit c400835

Browse files
alistairjevansalistairjevans
committed
Initial content and readmes
Split the stack integration role into read+write policies. Fix the PassRole policy. Gitignore Delete DS_Store README corrections Collect properties in addition to tags. Update README
1 parent da864a6 commit c400835

File tree

13 files changed

+2804
-2
lines changed

13 files changed

+2804
-2
lines changed

‎.github/workflows/test.yml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
name: Tests
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
branches: [main]
8+
9+
jobs:
10+
test:
11+
runs-on: ubuntu-latest
12+
strategy:
13+
matrix:
14+
spec:
15+
- spec/lambda/firehose_metrics_tag_enrichment_spec.rb
16+
- spec/lambda/firehose_logs_tag_enrichment_spec.rb
17+
steps:
18+
- uses: actions/checkout@v4
19+
20+
- uses: ruby/setup-ruby@v1
21+
with:
22+
ruby-version: '3.2'
23+
bundler-cache: true
24+
25+
- name: Run tests
26+
run: bundle exec rspec ${{ matrix.spec }} --format documentation

‎.gitignore‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# OS
2+
.DS_Store
3+
Thumbs.db
4+
5+
.claude/

‎Gemfile‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# frozen_string_literal: true
2+
3+
source 'https://rubygems.org'
4+
5+
# Lambda runtime dependencies
6+
gem 'aws-sdk-ec2'
7+
gem 'aws-sdk-lambda'
8+
gem 'aws-sdk-rds'
9+
gem 'aws-sdk-resourcegroupstaggingapi'
10+
gem 'rexml' # Required by AWS SDK for XML parsing
11+
12+
group :test do
13+
gem 'rspec'
14+
end

‎Gemfile.lock‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
GEM
2+
remote: https://rubygems.org/
3+
specs:
4+
aws-eventstream (1.4.0)
5+
aws-partitions (1.1200.0)
6+
aws-sdk-core (3.241.0)
7+
aws-eventstream (~> 1, >= 1.3.0)
8+
aws-partitions (~> 1, >= 1.992.0)
9+
aws-sigv4 (~> 1.9)
10+
base64
11+
bigdecimal
12+
jmespath (~> 1, >= 1.6.1)
13+
logger
14+
aws-sdk-ec2 (1.588.0)
15+
aws-sdk-core (~> 3, >= 3.241.0)
16+
aws-sigv4 (~> 1.5)
17+
aws-sdk-lambda (1.172.0)
18+
aws-sdk-core (~> 3, >= 3.241.0)
19+
aws-sigv4 (~> 1.5)
20+
aws-sdk-rds (1.303.0)
21+
aws-sdk-core (~> 3, >= 3.241.0)
22+
aws-sigv4 (~> 1.5)
23+
aws-sdk-resourcegroupstaggingapi (1.90.0)
24+
aws-sdk-core (~> 3, >= 3.241.0)
25+
aws-sigv4 (~> 1.5)
26+
aws-sigv4 (1.12.1)
27+
aws-eventstream (~> 1, >= 1.0.2)
28+
base64 (0.3.0)
29+
bigdecimal (4.0.1)
30+
diff-lcs (1.6.2)
31+
jmespath (1.6.2)
32+
logger (1.7.0)
33+
rexml (3.4.4)
34+
rspec (3.13.2)
35+
rspec-core (~> 3.13.0)
36+
rspec-expectations (~> 3.13.0)
37+
rspec-mocks (~> 3.13.0)
38+
rspec-core (3.13.6)
39+
rspec-support (~> 3.13.0)
40+
rspec-expectations (3.13.5)
41+
diff-lcs (>= 1.2.0, < 2.0)
42+
rspec-support (~> 3.13.0)
43+
rspec-mocks (3.13.7)
44+
diff-lcs (>= 1.2.0, < 2.0)
45+
rspec-support (~> 3.13.0)
46+
rspec-support (3.13.6)
47+
48+
PLATFORMS
49+
arm64-darwin-24
50+
ruby
51+
52+
DEPENDENCIES
53+
aws-sdk-ec2
54+
aws-sdk-lambda
55+
aws-sdk-rds
56+
aws-sdk-resourcegroupstaggingapi
57+
rexml
58+
rspec
59+
60+
BUNDLED WITH
61+
2.6.9

‎README.md‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,16 @@
1-
# aws
2-
Contains all Better Stack CloudFormation stacks and Lambdas for AWS Integration
1+
# Better Stack AWS Integration
2+
3+
This repository contains all Better Stack CloudFormation stacks and Lambdas for the Better Stack AWS Integration.
4+
5+
## Getting Started
6+
7+
To get started with Better Stack on AWS, [create a new AWS Source in Better Stack Telemetry](https://telemetry.betterstack.com/team/t0/sources/new?platform=aws).
8+
9+
## Ingested Data
10+
11+
When you deploy our CloudFormation stack you get:
12+
13+
- Automatic ingestion of all CloudWatch metrics into Better Stack.
14+
- Support for RDS Enhanced Metrics (RDSOS)
15+
- Detection and per-log-group optional ingestion of all Cloudwatch log groups.
16+
- Automatic integration with AWS X-Ray, ingest traces into Better Stack Telemetry and view your traces.

‎cloudformation/full/README.md‎

Lines changed: 190 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,190 @@
1+
# Better Stack CloudWatch Integration
2+
3+
Stream AWS CloudWatch metrics, logs, and optionally X-Ray traces and CloudTrail audit logs to Better Stack via Kinesis Data Firehose.
4+
5+
## Prerequisites
6+
7+
1. **Better Stack Account** - Sign up at [betterstack.com](https://betterstack.com)
8+
2. **Cluster ID** - Your Better Stack cluster identifier (e.g., `g1`, `s1234`, `acme`)
9+
3. **Source Token** - Authentication token from Better Stack
10+
4. **Source ID** - Source ID from Better Stack for resource matching
11+
5. **AWS CLI** - Configured with appropriate permissions
12+
13+
## Quick Start
14+
15+
### First Region Deployment
16+
17+
```bash
18+
aws cloudformation deploy \
19+
--template-file better-stack-full.yaml \
20+
--stack-name better-stack-full \
21+
--parameter-overrides \
22+
ClusterId=YOUR_CLUSTER_ID \
23+
SourceToken=YOUR_SOURCE_TOKEN \
24+
SourceId=YOUR_SOURCE_ID \
25+
--capabilities CAPABILITY_NAMED_IAM
26+
```
27+
28+
### Additional Regions
29+
30+
When deploying to additional regions, set `CreateGlobalResources=false` to reuse the IAM roles created in the first region:
31+
32+
```bash
33+
aws cloudformation deploy \
34+
--template-file better-stack-full.yaml \
35+
--stack-name better-stack-full \
36+
--region us-west-2 \
37+
--parameter-overrides \
38+
ClusterId=YOUR_CLUSTER_ID \
39+
SourceToken=YOUR_SOURCE_TOKEN \
40+
SourceId=YOUR_SOURCE_ID \
41+
CreateGlobalResources=false \
42+
--capabilities CAPABILITY_NAMED_IAM
43+
```
44+
45+
## Data Flows
46+
47+
| Data Type | Flow |
48+
|-----------|------|
49+
| Metrics | CloudWatch Metric Stream (JSON) -> Firehose -> Lambda (enrichment) -> Better Stack |
50+
| Logs | CloudWatch Logs -> Subscription Filter -> Firehose -> Lambda (enrichment) -> Better Stack |
51+
| Traces | X-Ray -> CloudWatch Logs (`aws/spans`) -> Subscription Filter -> Better Stack |
52+
| Audit | CloudTrail -> CloudWatch Logs -> Subscription Filter -> Better Stack |
53+
54+
## Parameters
55+
56+
### Required Parameters
57+
58+
| Parameter | Description |
59+
|-----------|-------------|
60+
| `ClusterId` | Your Better Stack cluster ID |
61+
| `SourceToken` | Better Stack source token for authentication |
62+
| `SourceId` | Better Stack source ID for resource matching |
63+
64+
### Deployment Options
65+
66+
| Parameter | Default | Description |
67+
|-----------|---------|-------------|
68+
| `CreateGlobalResources` | `true` | Create IAM roles. Set to `false` for secondary regions. |
69+
70+
### Feature Toggles
71+
72+
| Parameter | Default | Description |
73+
|-----------|---------|-------------|
74+
| `EnableTagEnrichment` | `true` | Enable Lambda-based tag enrichment for metrics and logs |
75+
| `EnableCloudTrail` | `true` | Create a new CloudTrail trail and forward to Better Stack |
76+
| `EnableXRayTransactionSearch` | `true` | Enable X-Ray Transaction Search and forward spans |
77+
78+
## CloudTrail Notes
79+
80+
Most AWS accounts already have CloudTrail enabled. Before enabling `EnableCloudTrail=true`, check for existing trails:
81+
82+
```bash
83+
aws cloudtrail describe-trails
84+
```
85+
86+
If a trail already exists, you may want to:
87+
1. Keep `EnableCloudTrail=false` and manually subscribe the existing CloudTrail log group
88+
2. Or enable it to create a dedicated trail for Better Stack
89+
90+
CloudTrail logs have a 1-day retention in CloudWatch (they are forwarded immediately to Better Stack).
91+
92+
## X-Ray Transaction Search
93+
94+
When `EnableXRayTransactionSearch=true`, the template automatically:
95+
1. Creates a CloudWatch Logs resource policy allowing X-Ray to write spans
96+
2. Enables Transaction Search via `AWS::XRay::TransactionSearchConfig`
97+
3. X-Ray sends trace segments to the `aws/spans` CloudWatch Logs group
98+
99+
No manual steps required - fully automated via CloudFormation.
100+
101+
## Better Stack Integration Role
102+
103+
The template creates a `better-stack-integration-role` IAM role that allows Better Stack to:
104+
- Read CloudWatch metrics and logs
105+
- Manage metric streams and log subscriptions
106+
- Discover AWS resources for context enrichment
107+
108+
After deployment, provide these values to Better Stack:
109+
- **Role ARN**: From stack output `IntegrationRoleArn`
110+
- **External ID**: From stack output `ExternalId`
111+
112+
## Tag Enrichment
113+
114+
When `EnableTagEnrichment=true`, Lambda functions enrich metrics and logs with AWS resource tags.
115+
116+
**Metrics enrichment** adds tags from:
117+
- EC2 instances
118+
- RDS databases
119+
- Lambda functions
120+
- DynamoDB tables
121+
- S3 buckets
122+
- ELB/ALB/NLB load balancers
123+
- SQS queues
124+
- SNS topics
125+
126+
**Logs enrichment** extracts resource info from log group/stream names:
127+
- `/aws/lambda/{function-name}` -> Lambda tags
128+
- `/aws/rds/instance/{db-instance}/{type}` -> RDS tags
129+
- `RDSOSMetrics` (Enhanced Monitoring) -> RDS tags from message body
130+
- `/ecs/{cluster}/...` -> ECS tags
131+
- `/aws/api-gateway/{api-id}` -> API Gateway tags
132+
133+
Lambda code is loaded from regional S3 buckets (`better-stack-lambda-{region}`).
134+
135+
## Stack Outputs
136+
137+
When `CreateGlobalResources=true`, the stack outputs:
138+
139+
| Output | Description |
140+
|--------|-------------|
141+
| `IntegrationRoleArn` | STS role ARN to provide to Better Stack |
142+
| `ExternalId` | External ID to provide to Better Stack |
143+
144+
Better Stack will discover all other resources (Firehose streams, Metric Streams, etc.) via the STS integration role.
145+
146+
## Troubleshooting
147+
148+
### Check Firehose Delivery Errors
149+
150+
```bash
151+
aws logs filter-log-events \
152+
--log-group-name /aws/kinesisfirehose/better-stack-metrics \
153+
--start-time $(date -d '1 hour ago' +%s000)
154+
```
155+
156+
### Check Lambda Errors
157+
158+
```bash
159+
aws logs filter-log-events \
160+
--log-group-name /aws/lambda/better-stack-metrics \
161+
--start-time $(date -d '1 hour ago' +%s000)
162+
```
163+
164+
### Verify Metric Stream Status
165+
166+
```bash
167+
aws cloudwatch get-metric-stream --name better-stack-metric-stream
168+
```
169+
170+
## Cleanup
171+
172+
To delete the stack:
173+
174+
```bash
175+
aws cloudformation delete-stack --stack-name better-stack-full
176+
```
177+
178+
**Note**: S3 buckets have `DeletionPolicy: Retain` to preserve any failed delivery data and CloudTrail logs. Delete manually if needed:
179+
180+
```bash
181+
# Delete Firehose backup bucket
182+
aws s3 rb s3://better-stack-firehose-ACCOUNT_ID-REGION --force
183+
184+
# Delete CloudTrail bucket (if CloudTrail was enabled)
185+
aws s3 rb s3://better-stack-cloudtrail-ACCOUNT_ID-REGION --force
186+
```
187+
188+
## Support
189+
190+
For issues or questions, please reach out at hello@betterstack.com

0 commit comments

Comments
 (0)