Multi-Region and Multi-Account Monitoring of the CloudTrail Logs via Amazon EventBridge

Dear DevOps community, I hope you’re having a good day!

To add it to it, let’s talk about how to use the EventBridge service and Lambda functions to implement almost free monitoring of CloudTrail logs.

Definitions

AWS CloudTrail is a service that monitors and records account activity across your AWS infrastructure, giving you control over storage, analysis, and remediation actions. As DevOps professionals, you will know that you can scroll through, search for, or examine CloudTrail logs to pinpoint each specific activity that happened in the account, when it occurred, who or what started it, whose resources were affected, etc. This level of insight is particularly helpful for reviewing security breaches, proactively monitoring AWS account vulnerability, or for meeting compliance requirements. Moreover, it can be used to trigger specific events.

Amazon EventBridge is a serverless event bus that helps you receive, filter, transform, route, and deliver events, in our case to a Lambda function. It enables you to design event patterns based on event content to determine which targets (subscribers) get each event going through the bus. Amazon EventBridge offers interesting possibilities for multi-service and even multi-departmental decoupled communications.

AWS Lambda is a serverless, event-driven computing service that allows you to run code for a virtually unlimited type of application or service without provisioning or maintaining servers. AWS Lambda is flexible and has low running costs. Building a Lambda function to examine log files from CloudTrail is simple. When certain events or log entries occur, Lambda scans the logs and sends out messages via SNS. By contacting their API endpoints from within Lambda, you can also quickly construct customized notification hooks for Slack, Zendesk, or other platforms.

CloudTrail configuration

As a first step, you’ll need to configure CloudTrail according to the instructions for collecting logs from multi-regions.

You will also need to follow this instruction to deploy the Lambda function with layers and download a YAML package and put it into the layer to convert messages from JSON to YAML format.

Now, let’s look at our Lambda code:

Lines 1-3: Import the python modules.

Line 8: This is our main function.

Line 9: The URL variable for the Slack channel.

Line 10: Parse the [“detail”][“requestParameters”] fields to determine which account the event came from.

Line 17: The text variable is formed. This variable will be sent to Slack.

Line 32: Encode text into JSON object.

Line 33: Send a message to Slack.

As you can see from the code, we parse the “detail” field and tailor our message so that it contains only the essential information – everything superfluous is discarded.

Amazon EventBridge configuration

Now let’s consider how to create the EventBridge rule:

Open the EventBridge service in the same region in which you deployed your Lambda function.
Go to the <Rules> section.
Create a new rule by clicking on <Create Rule>.
Enter a random name in the <Name> section. The Event bus section remains the default. Click <Next>.
In the Event source section, ignore AWS events, EventBridge partner events and Sample event section – optional. Now select Use pattern form in the Creation method section, click <Edit pattern> in the Event pattern section, and paste the code from the policy. If you prefer, you can add or remove calls you wish to monitor in the eventName API section. Click <Next>.
Now we are ready to configure the Target section – i.e., the section to which our events will be sent. In the Target 1 section, select <AWS service> and then click on the Lambda function in the list, select the function with the code provided above, leave the rest of the settings unchanged, and click <Next>.
Optionally add tags. Click <Next>.
Check that all the settings are correct and, finally, click <Create rule>.

Once we’ve created a rule in EventBridge, we can say that half the work is done. Let’s assume that we have created a centralized EventBridge rule, and now all the specified API calls are being tracked. If they match, a notification is sent to the Lambda function and it, in turn, will send a formatted message in Slack. This is an example of such a message:

Since EventBridge is a service that works in the scope of a single region, we have to create the same rule for each of the regions where CloudTrail monitoring is needed. But now, instead of the Lambda function, let’s use the EventBridge rule as the Target.

To do this, we must first complete all the EventBridge 1-5 configuration steps. In step 6, instead of AWS service, select the EventBridge event bus in the Target 1 section. Also, insert the ARN of the EventBridge bus in which we created our rule in the previous step into the Event bus in a different account or region. In the Execution role section, you can select Create a new role for this specific resource, and the required CrossRegion role will be created automatically.

The workflow will look like this diagram:

Creating roles

Also, for correct CrossAccount and CrossRegion operations, you need to create roles both on the source account side (from which we want to send events) and on the target account side (our centralized EventBridge bus).

On the source account side, you need to create such a role:

The ARN of our centralized EventBridge bus will be added to the Resource section. Trust relationships will look like this:

On the Target account side, you need to add the access rights directly to the EventBridge bus. To do that, follow the instruction:

Open the EventBridge service.
Go to the Event buses tab.
In the Default event bus section, select our default Event bus (or if you’ve created a custom EventBridge bus, select it.
In the Permissions field, select Manage permissions and insert the code from the CrossAccount example.
Save changes.

Bottom line

If you’ve followed our instructions, Events from all regions and accounts that correspond to our rule will be sent to the centralized EventBridge. The rule will then call the Lambda function to send a message to Slack.

Sergey Plesovskykh, Team Lead DevOps Engineer at Profisea