bwNetFlow
The bwNetFlow platform uses existing software and provides glue code to allow
for large scale network flow analysis. BwNetFlow mainly uses goflow
and Apache Kafka to process network flows in the form of
protobuf messages.
Overview
We provide a set of tools working with Kafka as consumers, producers, or both
(we call those processors) to establish a flow monitoring analysis pipeline.
While these tools can be combined in any specific way, the core components and
wiring in our use case is as follows:
- Goflow: Cloudflare’s goflow receives NetFlow v9 from our
external border interfaces and produces protobuf messages in a Kafka topic.
- Enricher: Our enricher reads from this topic and adds domain
specific knowlege (customer ID, device info, geolocation, etc.) during
transit into another Kafka topic.
- Splitter: Our splitter reads the enriched topic and splits
the stream of flow messages into customer-specific topics, enabling
multi-tenancy through Kafkas authentication and access control system.
- Reducer: Our reducer has the capability to reduce any topic’s
flows to some subset of its fields. It also has some anonymization routines.
- Dashboard: Our dashboard application reads from the
un-splitted, enriched topic. It then aggregates the flows to counters and
writes these counter values to a Time Series Database (InfluxDB or
Prometheus). This could be run on customer topics too, but InfluxDB (which we
use with this programm) can handle multiple customers through its
authentication features.
These components of our platform have been built using our
kafkaconnector library, which abstracts most of the recurrent
code fragments and already makes resonable assumptions based on what we are doing.
Customers and Users
In our setup, customers interact with the bwNetFlow platform in one of two ways:
- they look at the dashboards we generate using Grafana, InfluxDB, and the
aforementioned dashboard application
- they connect to Kafka themselves, using some custom tool written by them in
a language of their choosing, given it supports Kafka and Protobuf. Some
examples directly pulled from BelWü’s IP department can be found
here. They include fancy printing,
htop
-style CLI
viewers, flow field searches, bogon detection, and even Sqlite dumping of
flows.
- Protobuf to Netflow v9 Converter: Our
protobuf_to_netflow_converter converts all incoming protobuf decoded
messages into NetFlow v9 compliant messages, which is useful for customers
with existing Netflow-consuming setups.
- Protobuf to CSV converter: The protobuf_to_csv_converter writes
all incoming protobuf decoded messages into csv files. The specific protobuf
fields can be specified by the user.
- DDoS detection: This provides a proof-of-concept volume-based DoS
Detection implementation based on protobuf converted NetFlow data.
- NetFlow NEMEA Toolchain: This an example how the bwNetFlow
NetFlow v9 exporter and the NEMEA framework can be chained. This can be
applied similarly to all other available NEMEA modules.
To develop Kafka consumers/producers with C++ as these additional tools mostly
are, our cpp_kafkaconnector
library abstracts most of the recurrent code fragments and makes reasonable
assumptions for our use case.
Deployment
Our main components are available from their respective repositories and also
as container images (currently only via ghcr.io, see the Packages tab on our
organization page). As for container deployments, have a look at the
demo deployment for our customers (or our [fullstack
docker-composable][docker-composable] coming soon…).
Additionally, we’re working to release our Ansible playbooks used for our
currently largest deployment.
See our presentation from SC19’s INDIS workshop
here, or get the executive summary
from our poster below.