Welcome to the tutorial on how to stream CloudWatch logs to lambda function with subscription filter. Streaming logs to a lambda function can come handy when you want to perform real-time analysis of logs. Since we are streaming the logs to a lambda function, we need to keep in mind the limitation of AWS Lambda.

For the purpose of this tutorial, we will stream the logs of the lambda function. Ideally, we will create two lambda function. And we will stream the logs of first lambda function to the second lambda function. We will get started by creating an IAM role. Assuming you have logged into AWS console and navigate to IAM management console and create the role. I have created the role named lambda_cloudwatch_log_stream with AWSLambdaExecute permission. You can refer to this tutorial on how to create an IAM role.

We will use the same role for both lambda function. Now, let’s navigate to the Lambda Management Console and create two functions. I have created lambdaInvocation and lambdaStreamTarget respectively with Python 3.7 as runtime along with lambda_cloudwatch_log_stream as an existing role.

Within lambda function one, that is lambdaInvocation we will add print(“event”) statement in the code and execute that lambda once to generate the log group in CloudWatch.

We are going to write some code in the second lambda function (i.e. lambdaStreamTarget). But before we move on with it, we will create the subscription filter in CloudWatch. Here, we will select the log group of first lambda function (i.e. lambdaInvocation) and click on Actions -> Stream to AWS Lambda.

stream cloudwatch logs

In the next screen select the second lambda function (i.e. lambdaStreamTarget). I encourage everyone to read the warning message and that’s where I have mentioned earlier that we need to consider the lambda limitations. Click on Next.

stream cloudwatch logs

The coming screen asks us to configure Log Format and Filters. Hence, from the Log Format dropdown, we will select JSON and you can play around with other options too. You can even define custom filter pattern under Subscription Filter Pattern and test the outcome by clicking on Test Pattern. Click on Next.

export cloudwatch logs

The file step which is to Review the configuration and if everything seems good then click on Start Streaming. Hence, we have successfully created the Subscription Filter.

If you execute the first lambda function (i.e. lambdaInvocation) then you will be able to see the log data being populated in the event of second lambda function (i.e. lambdaStreamTarget). Yet, the logs we are looking at in the second lambda function (i.e. lambdaStreamTarget) is in base64 encoding and compressed with gzip. Therefore, we will write some code to decompress and decode the logs to make it readable. Let’s jump to the lambda function and paste the below snippet in the lambda_function file and save it.

import json
import gzip
import base64

def lambda_handler(event, context):
    cloudwatch_event = event["awslogs"]["data"]
    decode_base64 = base64.b64decode(cloudwatch_event)
    decompress_data = gzip.decompress(decode_base64)
    log_data = json.loads(decompress_data)
    return "Thanks"

We will go ahead and execute the first lambda function (i.e. lambdaInvocation) and check the logs of second lambda function (i.e. lambdaStreamTarget). If you remember than we had add print(event) in the same. So the outcome of that print is something we should be able to see. It could look something like this.

export cloudwatch logs

Ideally, it triggers by each line of logs in the source lambda function. The target or the second lambda function (i.e. lambdaStreamTarget) can be invoked by a maximum number of lines present in the logs of source function (i.e. lambdaInvocation). Please refer to the video tutorial mentioned below for end-to-end implementation.

Well, that’s it for now. I have announced the new tutorial series on AWS Audio Analysis, please have a look. In a mean time refer my YouTube channel for more tutorials. Keep sharing and stay tuned for more. Follow me on Twitter