Analyze AWS EKS Audit logs with Falco

Published in

FAUN — Developer Community 🐾

4 min readFeb 16, 2021

Falco is a Kubernetes threat detection engine. Falco supports Kubernetes Audit Events to track the changes defined in k8s audit rules made to your cluster.

But unfortunately, AWS EKS is a managed Kubernetes service, and it only can send audit logs to CloudWatch. It means there is no direct way for Falco to inspect the EKS audit events.

We need to implement a solution to ship audit logs from CloudWatch to Falco. There are two solutions.

https://github.com/sysdiglabs/ekscloudwatch
https://github.com/xebia/falco-eks-audit-bridge (I will not recommend this solution. It is a bit complex and requires extra setup like setup S3 bucket and AWS Kinesis Firehose service)

I will explain how to configure ekscloudwatch on EKS with IRSA roles support.

Enable Audit in EKS cluster

Enable IAM Roles for Service Accounts (IRSA) on the EKS cluster if you did not enable yet.

Configure IAM role and policy

Create a policy called ekscloudwatch-eks-cw-policy then

Create a role with a Trust Relationship for and attach the policy above which is called ekscloudwatch-eks-cw-role in this deployment

You can get all of them from the following gist

Install falco

This parameter for chart version: 2.0.16. Always refer to falco docs before deploying.

see https://github.com/falcosecurity/charts/tree/master/falco#kubernetes-audit-log for details. https://github.com/falcosecurity/charts/blob/master/falco/values-k8saud will have all parameter you need to set to enable k8s audit.

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
helm install falco falcosecurity/falco --namespace falco -f ./values-k8saudit.yaml --create-namespace

helm will expose a service with helm application name prefix. It is falco in this deployment.

Update ekscloudwatch serviceaccount, configmap and deployment

Createekscloudwatch.yaml file and copy ServiceAccount,ConfigMap and Deployment from above gist. We must edit them before applying with kubectl

If you install falco with different name(falco) and/or in different namespace, you need to update above deployment yaml and configmap in gist.

1. Edit value of eks.amazonaws.com/role-arn in serviceaccount with your role arn

eks.amazonaws.com/role-arn: "arn:aws:iam::12345678910:role/ekscloudwatch-eks-cw-role"

2. Edit cluster_name: "my-eks-cluster" with your cluster name in configmap

3. Edit endpoint: "http://falco-k8saudit-webhook:9765/k8s-audit" if you installed falco with different name than falco

4. Don’t forget to set your region aws_region. If you don’t set ekscloudwatch will call ec2 imdsv1 to get region name.

Deploy ekscloudwatch

$ kubectl apply -f ekscloudwatch.yaml

If you check falco pods logs you will get some logs like

Falco Resource requirement

If you see the following message in falco pod output, you should increase resources. Default values: https://github.com/falcosecurity/charts/blob/master/falco/values.yaml#L18

Tue Feb 16 19:21:28 2021: Falco internal: syscall event drop. 2788 system calls dropped in last second.

Cleanup

Delete role and policy on AWS.
Delete EKS resources

kubectl delete -f ekscloudwatch.yaml
helm uninstall falco

About Falco k8s audit logs

Here is the list of the rules

Disallowed K8s User
Create Disallowed Pod
Create Privileged Pod
Create Sensitive Mount Pod
Create HostNetwork Pod
Create NodePort Service
Create/Modify Configmap With Private Credentials
Anonymous Request Allowed
Attach/Exec Pod
EphemeralContainers Created
Create Disallowed Namespace
Pod Created in Kube Namespace
Service Account Created in Kube Namespace
System ClusterRole Modified/Deleted
Attach to cluster-admin Role
ClusterRole With Wildcard Created
ClusterRole With Write Privileges Created
ClusterRole With Pod Exec Created
K8s Deployment Created
K8s Deployment Deleted
K8s Service Created
K8s Service Deleted
K8s ConfigMap Created
K8s ConfigMap Deleted
K8s Namespace Created
K8s Namespace Deleted
K8s Serviceaccount Created
K8s Serviceaccount Deleted
K8s Role/Clusterrole Created
K8s Role/Clusterrole Deleted
K8s Role/Clusterrolebinding Created
K8s Role/Clusterrolebinding Deleted
K8s Secret Created
K8s Secret Deleted
All K8s Audit Events
Full K8s Administrative Access
Ingress Object without TLS Certificate Created
Untrusted Node Successfully Joined the Cluster
Untrusted Node Unsuccessfully Tried to Join the Cluster