Falco: The Kubernetes Runtime Security Watchdog

Managing a Kubernetes cluster can feel like juggling chainsaws on a unicycle. Adding runtime security often feels like tossing a flaming torch into the mix. This is where Falco steps in—your reliable companion for runtime security, offering insights into unexpected behaviours without overwhelming you with complexity.

What is Falco?

Falco is an open-source runtime security tool that monitors your Kubernetes cluster for suspicious activity. Think of it as the bouncer for your cluster, keeping an eye on system calls, file access, and network activity. Unlike traditional security tools, Falco focuses on real-time detection, making it ideal for identifying intrusions or anomalies as they happen.

Why Use Falco?

Here's what makes Falco a valuable addition to your Kubernetes setup:

• Real-Time Alerts: Get notified the moment something suspicious occurs
• Custom Rules: Tailor its monitoring to suit your cluster’s unique workloads
• Extensibility: Integrate Falco with tools like Loki, Prometheus, or your existing observability stack
• Open Source: Backed by a vibrant community, Falco is constantly evolving to address emerging threats

Getting Started with Falco

Integrating Falco into your Kubernetes cluster is surprisingly straightforward. You can deploy it as a DaemonSet, ensuring that it runs on every node in your cluster. Once deployed, you can start using Falco's built-in rules or create your own custom rules to match your specific security requirements but one step at a time ...

Installation

The easiest method I found to installing this was to leverage our old friend ... Helm which is made this super simple thanks to the community charts:

$ helm repo add falcosecurity https://falcosecurity.github.io/charts
$ helm repo update
$ helm install falco falcosecurity/falco

Of course, we also need a way to configure this a little to suite the needs of our cluster and leveraging the values we can pipe these into the chart... now, there are a loooot of configuration in the values.yaml so we'll focus on adding the rules from above and format the output to json.

It's worth 🔗 checking out their readme before you run this so you know what is happending when you deploy the chart.

For now, we'll set up a more structured output using JSON so it can be more easily piped into something like loki in the future:

# enable the kubernetes collectors
collectors:
  kubernetes:
    enabled: true

# Required to include the k8smeta plugin in support of the kubernetes collectors
falcoctl:
  config:
    artifact:
      install:
        refs: [falco-rules:3 ghcr.io/falcosecurity/plugins/plugin/k8smeta:0.2.1]

# Structure the output so it can be piped into Loki and reduced the volume of logs
falco:
  json_output: true 
  json_include_output_property: false
  json_include_tags_property: false

Save this as a values.yaml file and apply it using a slight mod to our helm command as follows:

$ helm upgrade --install falco falcosecurity/falco -f values.yaml

This being said, there are already a huge number of rules out of the box that will probably cover 90% of the scenarios you want to cover (to begin with anyway):

This is the list of default rules from their 🔗docs 😆

Understanding Falco Rules

Falco’s power lies in its customisable rule engine. While it ships with a robust set of default rules, tailoring them to your environment can significantly enhance its effectiveness.

Example Rule: Detect Privileged Container Execution

- rule: Run Privileged Container
  desc: Detect any privileged container launches
  condition: evt.type = "execve" and container.privileged = true
  output: "Privileged container launched (user=%user.name container=%container.id)"
  priority: WARNING

This rule alerts you whenever a privileged container is launched, providing the username and container ID for further investigation. By adjusting conditions and priorities, you can fine-tune alerts to your specific needs.

These can be easily added to our deployment by adding something like the following to our values.yaml file will create these additional rules:

# define our custom rules
customRules:
  example-rules.yaml: |-
    - rule: Run Privileged Container
      desc: Detect any privileged container launches
      condition: evt.type = "execve" and container.privileged = true
      output: "Privileged container launched (user=%user.name container=%container.id)"
      priority: WARNING

    - rule: Outbound connection to suspicious IP
      desc: Detect outbound connections to known bad IP addresses
      condition: net.direction=out and fd.sip.is_blacklisted=true
      output: Outbound connection to blacklisted IP detected (command=%proc.cmdline connection=%fd.name user=%user.name)
      priority: WARNING
    
    - rule: Shell spawned in container
      desc: Detect a shell spawned in a container
      condition: spawned_process and container.id != host and proc.name = "bash"
      output: Shell spawned in container (user=%user.name command=%proc.cmdline container_id=%container.id image=%container.image.repository)
      priority: WARNING

    - rule: Sensitive file accessed
      desc: Detect access to sensitive files
      condition: open_read and fd.name in ("/etc/shadow", "/etc/passwd")
      output: Sensitive file opened for reading (user=%user.name command=%proc.cmdline file=%fd.name)
      priority: WARNING

Deploy and access the dashboard

If you need a quick view into the Falco data then luckily they have a sidekick that can be deployed in the same helm chart by adding the following to the values.yaml file:

falcosidekick:
  enabled: true
  webui:
    enabled: true

There's no public ingress for this, for obvious reasons, but you can port-forward and access it like:

$ kubectl port-forward svc/falco-falcosidekick-ui 8080:2802

Then access the dashboard via http://localhost:8080 and login using the credentials stored within the secret `falco-falcosidekick-ui`. It should look something like:

For more info around this and how to trigger some events that you can see in the dashboard check out their 🔗quickstart docs which will show you how to create custom rules, deploy a test app and trigger an event that will show up in the dashboard.

Integrating Falco into Your Workflow

To maximise Falco's value, consider integrating its alerts with your existing monitoring and incident response workflows. Some popular integrations include:

• Loki/Grafana: Send Falco alerts to Loki for centralised log storage and create dashboards in Grafana for visualisation
• Slack or PagerDuty: Configure Falco to send alerts to your team's communication or incident response platform
• Prometheus AlertManager: Use Falco's alert outputs to trigger Prometheus alerts for seamless incident response

Best Practices for Using Falco

These are some of things I've found while playing with Falco, let me know in the comments below you have some other best practices to share:

1. KISS (Keep it Simple and Smart) Principle: Begin with Falco's default rules and incrementally customise them to match your cluster’s workload and risk profile
2. Regular Rule Updates: Keep your rules updated to account for new workloads, Kubernetes features, or emerging threats
3. Integrate with Observability: Combine Falco with tools like Prometheus or Elasticsearch for comprehensive monitoring and analysis
4. Review Alerts Regularly: Tune your configuration to reduce noise while ensuring critical events are detected and acted upon ... consider sending the more important ones to something like slack

Wrap-Up

Falco isn't a one-size-fits-all solution, but it's a powerful addition to your Kubernetes security stack. By configuring it thoughtfully and integrating it with your existing tools, you can ensure your cluster remains secure without being overwhelmed by unnecessary alerts.

Remember: runtime security is about visibility, control, and proactive measures. With Falco, you’re not just reacting to issues ... you're actively defending your infrastructure ... but don't make this your only tool 😉