Setting up Investigations with Hadoop

Overview

Hadoop is an open-source framework capable of processing applications executed in a distributed computing environment. Using the Capsule8 product you can set up investigations by following the procedure explained below:

Requirements

• Hadoop Cluster 
• Capsule8 Sensor running in your environment

1. Configuring the Sensor:

Edit the configuration file /etc/capsule8/capsule8-sensor.yaml.

Add a sink for HDFS, enable the sensor to create directories, and turn on the flight recorder:

cloud_meta: auto
blob_storage_create_buckets_enabled: true
investigations:
  reporting_interval: 30s
  sinks:
    - name: "[namenode hostname/ip]:9000/capsule8-investigations/"
      backend: hdfs
      automated: true
      type: parquet
      partition_format: "hostname_partition={{.Hostname}}/date_partition={{.Time.Format \"2006-01-02\"}}"
      credentials:
        blob_storage_hdfs_user: hadoop
flight_recorder:
  enabled: true
  tables:
    - name: "shell_commands"
      rows: 1000
      enabled: true
    - name: "tty_data"
      rows: 1000
      enabled: true
    - name: "connections"
      rows: 2000
      enabled: true
    - name: "sensor_metadata"
      rows: 500
      enabled: true
    - name: "alerts"
      rows: 100
      enabled: true
    - name: "sensors"
      rows: 10
      enabled: true
    - name: "process_events"
      rows: 4000
      enabled: true
    - name: "container_events"
      rows: 300
      enabled: true

Restart the sensor and verify it worked with:

$ hdfs dfs -ls /capsule8-investigations/ 

This should list all of the tables that were enabled in the config:

drwxr--r-- - root supergroup 0 2020-10-27 18:33 /capsule8-investigations/alerts
drwxr--r-- - root supergroup 0 2020-10-27 18:33 /capsule8-investigations/connections
drwxr--r-- - root supergroup 0 2020-10-27 18:33 /capsule8-investigations/container_events
drwxr--r-- - root supergroup 0 2020-10-27 18:33 /capsule8-investigations/process_events
drwxr--r-- - root supergroup 0 2020-10-27 18:33 /capsule8-investigations/sensor_metadata
drwxr--r-- - root supergroup 0 2020-10-27 18:33 /capsule8-investigations/sensors

2. Editing the Sensor config:

After confirming that the Capsule8 Sensor is properly configured, edit the reporting interval to a more reasonable time.

cloud_meta: auto
blob_storage_create_buckets_enabled: true
investigations:
  reporting_interval: 5m
  #...