TLDR: CloudTrail detection engine (CTDE), is an AWS native (i.e. using lambda and CloudTrail) detection engine.

Requirement and Research

For one of the work I did, I needed to come up with a detection for an AWS environment, that is cheap, will take custom detection (preferably Sigma rule for easier maintenance) and somewhat flexible/ customizable.

With that requirements in mind, I needed something that will be able to take, for the time being, CloudTrail. The thinking behind this was a portion of CloudTrail events are of a high impact from a security standpoint, and with how AWS environment is architectured, having a detection on these high impact CloudTrail will provide pretty good coverage with minimal cost upfront.

Muscle memory immediately bring me to SIEM route. This is something that will be a good base to build upon, I can start with CloudTrail and send a bunch other log later. However shipping log there is not going to be cheap, thus failing the first requirement.

Next best thing I was drawn to was GuardDuty. However, GuardDuty failed in being flexible and taking custom detection. Another offering from AWS that I also thought of was CloudWatch with custom metrics. This setup will allow flexibility and custom detection. However the thought of managing custom detection in CloudWatch give me shivers and relying and later expanding on CloudWatch for detection would felt like building on a wobbly ground.

The last alternative I thought of, and eventually the one I chose, was hosting a lambda that consume CloudTrail and Sigma rule, then fire alert when it found matches. The cost shouldn’t be too big, because the organization is not really that big and it will be super flexible, I can also manage the Sigma rule using a repository and detection pipeline to verify the detection.

Design and Development

The tool will comprise of a python script hosted in AWS Lambda, Amazon SNS to send the alert, two S3 bucket to hold CloudTrail events and sigma signature. A new object in the CloudTrail S3 bucket (i.e. new CloudTrail events) will trigger the python. The python script then load CloudTrail events from S3 bucket, and also the sigma signature. The script then compare the required detection from the sigma with the CloudTrail events and fire email notification for matched events.

This setup, as I mentioned in the requirement, will allow user to have the ability to custom their detection to detect high impact CloudTrail events, and leveraging sigma rules flexibility. So kind of having the flexibility of a SIEM detection, without having to ship the log to a real SIEM, at least until the requirements fit to go the SIEM route.

I designed the development to be of at least three stages with incremental addition in the future. I aim for these features to be developed for the stages:

1. Phase 1: Lambda able to take in hard coded CloudTrail API, send un-formatted email alert

2. Phase 2: Lambda able to read Sigma from S3 and consume Sigma rule

3. Next Phases:

   • Lambda able to send formatted email alert

   • Lambda able to do correlation between rules

   • Lambda able to do threshold for a rule

Each of the stages will have similar steps, that is:

1. definition and configuration

2. coding and local testing.

3. packaging and deployment.

4. monitoring.

Lambda

Currently (i.e. phase 2), The Lambda function is triggered by the s3:ObjectCreated:* event whenever a new CloudTrail log file is written to your S3 bucket, and here is the simplified logic:

1. Get the Rules: The function first fetches your set of custom security criteria (Sigma rules) from a dedicated S3 bucket. It reads all these rules, parses them, and makes a simple list of critical event names it needs to watch for (like DeleteRole or DisableLogging). This list is cached for speed.

2. Get the Logs: When a new CloudTrail log file arrives in its S3 bucket, the function downloads it, decompresses it, and extracts all the individual activity records.

3. Check for Matches: The function goes through every CloudTrail record and checks if its eventName property is present in the cached list of critical event names.

4. Send the Alert: If a record matches a rule, the function immediately publishes the full event details as a message to a designated Amazon SNS Topic, which then notifies your security team (e.g., via email or integrated ticketing system).

Accompanying Lambda is the IAM Execution Role and permission to access the S3, and publish SNS to send the alert. I attached AWSLambdaBasicExecutionRole and a custom permission for S3 access and SNS publishing. Limiting the access only to two buckets needed (Sigma rules and CloudTrail).

Sigma rules

To automate Sigma rule deployment for the setup, I use a GitHub repository and workflows to push rules to an S3 bucket, making it a simple pipeline to deploy rules to the setup. I used GitHub OIDC and an IAM role with limited policy for secure, least privilege and short-lived credentials. Since this is more of the way I deployed the detection pipeline, and less about the engine itself, I’ve talked more about this on a different post here.

Caveats

As you might already figured out, cloud logs and in this case CloudTrail can be pretty noisy, which I believe can be mitigated by at least a couple of things:

• A selection of a right collection of sigma signature is → this is what I meant by a high impact CloudTrail events. For instance, I might want to be alerted on a StopLogging events, but probably not on all DescribeInstances events

• Threshold for the detection. This feature is pretty common in a SIEM, but still on phase 3 for the tool. To come back to the previous example, if I can set a threshold for how many DescribeInstances events per 5 minutes, this is something that I might wanted to be alerted on.

• Correlation between loaded signature. This is also a common feature called composite detection. If a IAM event such as AssumeRole, was followed by a bunch of discovery activities (i.e. DescribeInstances), I might want to be alerted on that.

As part of deploying the Sigma rule, one thing I learned was to test Sigma rule extensively and use the result to feed back into CTDE. I end up increasing the Lambda timeout as part of the lesson learned from Sigma rule testing.

Currently threshold feature is in development and correlation is still further down the pipeline, but the tool is operational if you have several high impact CloudTrail event you want to monitor on.

Conclusion

CTDE tries to fill the gap of aws detection for small organization that needs more flexibility than just turning on GuardDuty, but don’t need a full blown SIEM. If this sounds fun, Terraform script is available with the where you can easily deploy the tool in the github here.