Logstash Cloudwatch Logs Codec

Parse CloudWatch Logs subscriptions into individual events.

Installation

This plugin can be installed by Logstash's plugin tool.

bin/logstash-plugin install logstash-codec-cloudwatch_logs

Usage

At its simplest:

input {
  kinesis {
    kinesis_stream_name => "stream"
    codec => cloudwatch_logs
  }
}

Event Format

The CloudWatch Logs codec breaks each multi-event subscription record into individual events. It does this by iterating over the logEvents field, and merging each event with all other top-level fields. The codec drops the logEvents field from the final event.

For example, given a subscription record:

{
    "owner": "123456789012",
    "logGroup": "Example",
    "logStream": "Example1",
    "subscriptionFilters": [
        "RootAccess"
    ],
    "messageType": "DATA_MESSAGE",
    "logEvents": [
        {
            "id": "1",
            "timestamp": 1478014822000,
            "message": "event1"
        },
        {
            "id": "2",
            "timestamp": 1478014825000,
            "message": "event2"
        }
    ]
}

...this codec would yield two individual events:

[
  {
      "owner": "123456789012",
      "logGroup": "Example",
      "logStream": "Example1",
      "subscriptionFilters": [
          "RootAccess"
      ],
      "messageType": "DATA_MESSAGE",
      "id": "1",
      "timestamp": 1478014822000,
      "message": "event1"
  },
  {
      "owner": "123456789012",
      "logGroup": "Example",
      "logStream": "Example1",
      "subscriptionFilters": [
          "RootAccess"
      ],
      "messageType": "DATA_MESSAGE",
      "id": "2",
      "timestamp": 1478014825000,
      "message": "event2"
  }
]