NfSen - Netflow Sensor

What is NfSen?

• Display your netflow data: Flows, Packets and Bytes using RRD (Round Robin Database).
• Easily navigate through the netflow data.
• Process the netflow data within the specified time span.
• Create history as well as continuous profiles.
• Write your own plugins to process netflow data on a regular interval.

Table of Content

1. NfSen - Netflow Sensor
   1. What is NfSen?
   2. Table of Content
2. Screenshots
3. Installing NfSen
   1. Prerequisites:
   2. First Installation
      1. Master Config File:
      2. Importing existing netflow data
      3. Start-Stop NfSen
   3. Upgrading NfSen
4. Working with NfSen
   1. Views
   2. Profiles
      1. Profile Types
      2. Creating profiles
      3. Managing Profiles
   3. Navigation
      1. Selecting a time slot or a time window
      2. Statistic Summary
      3. Graph Display Options
      4. Netflow Processing
         1. Filter Syntax
         2. Options
         3. Default Filters
   4. Bookmarks
   5. Command line tool 'nfsen'
5. Plugins
   1. Installing Plugins
   2. Writing Backend Plugins
   3. Testing Backend Plugins
   4. Writing Frontend Plugins
   5. Optional Modules
      1. Notification.pm

Screenshots

NfSen - General Overview Page	NfSen - Flow Overview Page	NfSen - Navigation Page
NfSen - Netflow Processing output	NfSen - Profile Info

Installing NfSen

Prerequisites:

• PHP and Perl:
  NfSen is written in PHP and Perl and should run on any *NIX system.
  At least Perl 5.6.0 and PHP > 4.1 is required including the Perl regex extension.
• RRD tools
  For the netflow graphs, NfSen requires the RRD tools, at least the RRDs Perl Module.
  All about RRD here.
  
• Nfdump tools
  The nfump tools are the backend tools for NfSen and will collect and process the netflow data.
  Make sure you have at least version 1.4. You can download nfdump from sourceforge nfdump.sourceforge.net.
  

First Installation

Master Config File:

##############################
#
# NfSen master config file
#
# Configuration of NfSen:
# Set all the values to fit your NfSen setup and run the 'install.pl'
# script from the nfsen distribution directory.
#
# You should not need to changes anything after NfSen is installed,
# besides the NfSen plugins at the bottom.
# When you make any changes in the plugins section, run 'nfsen reload' 
# to make sure nfsen-run gets notified about your plugins.
#
# Do not change any other settings after NfSen is installed.
# otherwise you must rerun the install.pl script.
#
# The syntax must conform to Perl syntax.
#
##############################
#
# NfSen default layout: 
# Any scripts, modules or profiles are installed by default under $BASEDIR. 
# However, you may change any of these settings to fit your requested layout.

#
# Required for default layout
$BASEDIR = "/data/nfsen";

#
# Where to install the NfSen binaries
$BINDIR="${BASEDIR}/bin";

#
# Where to install the NfSen Perl modules
$LIBEXECDIR="${BASEDIR}/libexec";

#
# Where to install the config files
$CONFDIR="${BASEDIR}/etc";

#
# NfSen html pages directory:
# All php scripts will be installed here.
# URL: Entry point for nfsen: http://<webserver>/nfsen/nfsen.php
$HTMLDIR = "/var/www/nfsen/";

#
# Where to install the docs
$DOCDIR="${HTMLDIR}/doc";

#
# Var space for NfSen
$VARDIR="${BASEDIR}/var";

#
# The Profiles stat directory, where all profile information
# RRD DBs and gif pictures of the profile are stored
$PROFILESTATDIR="${BASEDIR}/profiles";

#
# The Profiles directory, where all netflow data is stored
$PROFILEDATADIR="${BASEDIR}/profiles";

#
# Where go all the backend plugins
$BACKEND_PLUGINDIR="${BASEDIR}/plugins";

#
# Where go all the frontend plugins
$FRONTEND_PLUGINDIR="${HTMLDIR}/plugins";

#
# nfdump tools path
$PREFIX = '/usr/local/bin';

#
# BASEDIR unrelated vars:
#
# Run nfcapd as this user
# This may be a different or the same uid than your web server.
# Note: This user must be in group $WWWGROUP, otherwise nfcapd
# is not able to write data files!
$USER = "netflow";

# user and group of the web server process
# All netflow processing will be done with this user
$WWWUSER = "www";
$WWWGROUP = "www";

# Receive buffer size for nfcapd - see man page nfcapd(1)
$BUFFLEN = 200000;

# Netflow sources
# Define an ident string, port and colour per netflow source
# ident identifies this netflow source. e.g. the router name, 
# Upstream provider name etc.
# port nfcapd listens on this port for netflow data for this source
# col colour in nfsen graphs for this source
#
# Syntax: 
# 'ident' => { 'port' => '<portnum>', 'col' => '<colour>' }
# Ident strings must be 1 to 19 characters long only, containing characters [a-zA-Z0-9_].

%sources = (
 'upstream1' => { 'port' => '9995', 'col' => '#0000ff' },
 'peer1' => { 'port' => '9996', 'col' => '#ff0000' },
);

#
# Low water mark: When expiring files, delete files until
# size = max size * low water mark
# typically 0.9
$low_water = 0.9;

#
# syslog facility for periodic jobs
# nfsen uses level 'debug', 'info', 'warning' and 'err'
# Note: nfsen is very chatty for level 'debug' and 'info'
# For normal operation, you may set the logging level in syslog.conf
# to warning or error unless you want to debug NfSen
$syslog_facility = 'local3';

#
# plugins
# plugins are run for each timeslot, after the roll over of new data files.
# A plugin may run for any profile or for a specific profile only.
# Syntax: [ 'profile', 'module' ]
# profile: ',' separated list of profiles, or "*' for any profile
# module: Module name.
# The module follows the standard Perl module conventions, with at least two
# additional functions: Init() and run(). See demoplugin.pm for a simple template.
# Plugins are installed under 
#
# $BACKEND_PLUGINDIR and $FRONTEND_PLUGINDIR

@plugins = (
 # profile # module
 # [ '*', 'demoplugin' ],
);

#
# Notification module
# The Notification module is an optional module. If you want your plugins to
# notify any result by email, use this module.
# Make sure you have installed Mail::Internet before using the module
#
# Use this from address 
$MAIL_FROM = 'your@from.example.net';

# Use these recipients
$RCPT_TO = 'any@example.net, another@example.net';

# Use this SMTP server
$SMTP_SERVER = 'localhost';

1;

./install.pl etc/nfsen.conf

• Create the NfSen environment under BASEDIR
• Copy the php/html files into the HTMLDIR
• Create the live profile.
• Prepares the RRD DBs for the live profile.
• Creates and configures config.php
  

// print "<a href='doc/NfSen.html' target='_blank' >Help</a>\n";

Importing existing netflow data

• cd BASEDIR/profiles/live
• Copy already existing data into the appropriate directory or make sure you have a softlink from the source directory to your flows.
• cd BASEDIR/bin and rebuild the profile live:
  ./nfsen -r live
  

Start-Stop NfSen

BINDIR/nfsen.rc start

Upgrading NfSen

1. Change to the directory where you unpacked NfSen.
2. If you upgrade from NfSen v1.1, add the missing directory config statements to your existing nfsen.conf file. To reflect the NfSen v.1.1 directory layout, add the following lines after the HTMLDIR variable:
   
   

   # Where to install the binaries
   $BINDIR="${BASEDIR}/bin";
   #
   # Where to install the NfSen Perl modules
   $LIBEXECDIR="${BASEDIR}/libexec";
   #
   # Where to install the config files
   $CONFDIR="${BASEDIR}/etc";
   #
   # Var space for NfSen
   $VARDIR="${BASEDIR}/var";
   # 
   # Where to install the docs
   $DOCDIR="${HTMLDIR}/doc";
   #
   # The Profile stat directory, where all information and RRD DBs of the profile are stored
   $PROFILESTATDIR="${BASEDIR}/profiles";
   #
   # The Profile directory, where all netflow data is stored
   $PROFILEDATADIR="${BASEDIR}/profiles";
   #
   # Where go all the backend plugins
   $BACKEND_PLUGINDIR="${BASEDIR}/libexeec";
   #
   # Where go all the frontend plugins
   $FRONTEND_PLUGINDIR="${HTMLDIR}/plugins";

   
   
3. If you feel ok with the current directory layout goto step 4.
   If you want to rearrange the directory layout:
   
   
   • Stop nfsen: BASEDIR/bin/nfsen.rc stop
   • Move/copy directories to the new desired location and change the config to reflect the changes.
   • If you separate PROFILESTATDIR and PROFILEDATADIR:
     Copy/move all *rrd *gif files as well as the profile.dat file into the appropriate PROFILESTATDIR/<profilename>
     Copy/move the filter.txt/history_filter.txt file and all source directories to the appropriate PROFILEDATADIR/<profilename>
     
     Example for profile 'port80':
     

     cd $BASEDIR/profiles/port80
     mv *rrd *gif profile.dat $PROFILESTATDIR/port80
     mv filter.txt <source1> <source2> ... $PROFILEDATADIR/port80 

   
   
4. Run:
   

   ./install.pl <path/to/your/existing/nfsen/etc/nfsen.conf>

5. Start NfSen:
   

   BINDIR/nfsen.rc start

Working with NfSen

• Web Interface
• Command line interface

Views

Tab Navigation

Profiles

Profile Selection

Profile Types

Creating profiles

Successful creation of new profile.

Progress of building the profile

Managing Profiles

Delete Profile

Confirm Delete Profile

Navigation

Navigation Display

The page is divided into two parts: The upper part allows you to navigate through the netflow data as well as selecting a time slot or time window. The lower part contains all the controls to process the netflow data of the selected time slot or time window.

Select Time Span

Selecting a time slot or a time window

A time slot starts at every 5 minutes cycle of the hour ( 0, 5, 10, 15 etc. ) and lasts 5 minutes. A time window consists of several time slots.When entering the 'Details view' a window scale of one days is selected so you will see the last 24 hours of the profile. The time cursor is placed in the middle of the begin and end of these 24 hours and the time window slot is set to one time slot. You will see the selected time slot or time window always in the title of the browser window as well as in the title of the main graph. Selecting a different time slot can be done in a number of ways:

• Clicking into the main graph, for example when you see a suspicious peak.
• Using the time cursor controls:
  
  
  

  >	Next time slot: Advance time by 5 minutes.
  <	Previous time slot: Go back 5 minutes.
  >>	Advance time slot by a full time span of the graph.
  <<	Go back by a full time span of the graph.
  >|	Go to the end of the profile.
  |	Center time cursor in current graph.
  ^	Place cursor at peak, found within +/- 1 hour timespan of current cursor position.

  
  
  
  
• 
  Entering the timeslot of interest in the t_start input field and press the enter key.

The graphs are immediately updated, when selecting a different time slot.
Sometimes it is desirable to process more than a single 5 min timeslot. The time range of interest an be selected by extending the time window. First, place the time cursor on the left edge of he requested time window, using the methods described above. Then, select the right edge of the time window by:

Select 'right' Mark from the pull down menu  and click into the main graph.

or

Enter the the right boundary in the t_end input field and press the enter key.

The main graph will be immediately updated with the selected time window:

Selected Time Window

• A selected time window may be shifted by selecting a new left boundary. The size of the time window remains.
• To adjust the window size, select a new right window boundary.
• 

Border of available Data

Statistic Summary

Disabled sources 'Downstream' and 'Peer1'

Graph Display Options

• Scale:
• Linear y-axis
• Logarithmic y-axis.
• Graph Type:
• Stacked: All sources are drawn on top of each other.
• Line: All sources are drawn independent.

 

Netflow Processing

• Select the netflow sources to process. You may select multiple sources.
• Enter a netflow filter. The syntax conforms to the nfdump filter syntax.
• Select any options.

Filter Syntax

expr and expr, expr or expr, not expr, ( expr ).

Expr can be one of the following filter primitives:

Any
 any Flows. Used as dummy filter. Use 'not any' to block all flows.

protocol
 TCP, UDP, ICMP or PROTO num where num is the protocol number.

IP address
 [SourceDestination] IP a.b.c.d or HOST a.b.c.d with a.b.c.d as any valid IP address. 

[SourceDestination]
 defines the IP address to be selected and can be SRC DST or any combination of SRC and|or DST.
 Omitting SourceDestination is equivalent to SRC or DST.

[inout]
 defines the interface to be selected and can be IN or OUT.

network
 [SourceDestination] NET a.b.c.d m.n.r.s
 [SourceDestination] NET a.b.c.d/num with a.b.c.d as network number, m.n.r.s as netmask or num 
 as maskbits. The network may be given as a.b, a.b.c, where a B or C-class 
 equivalent netmask is assumed.

Port
 [SourceDestination] PORT [comp] num with num as a valid port number. If comp is omitted, '=' is assumed.

Interface
 [inout]  IF num with num as an interface number.

Flags
 flags tcpflags
 With tcpflags as a combination of:
 A ACK.
 S SYN.
 F FIN.
 R Reset.
 P Push.
 U Urgent.
 X All flags on.
The ordering of the flags is not relevant. Flags not mentioned are treated as don't care. In order 
to get those flows with only the SYN flag set, use the syntax 'flags S and not flags AFRPU'.

TOS
 tos value
 Type of service: Value 0..255.

Packets
 packets [comp] num
 Limit the packet count in the netflow record.

Bytes
 bytes [comp] num
 Limit the byte count in the netflow record.

[comp] 
The following comparators are supported:
 =, ==, >, <, EQ, LT, GT. If comp is omitted, '=' is assumed.

AS 
 [SourceDestination] AS num with num as a valid AS number.

Examples:
tcp and ( src ip 172.16.17.18 or dst ip 172.16.17.19)

tcp and ( net 172.16/16 and src port > 1024 and dst port 80 ) and bytes > 2048

Options

Default Filters

Bookmarks

Command line tool 'nfsen'

Plugins

Installing Plugins

#
# plugins
# plugins are run for each timeslot, after the roll over of new data files.
# A plugin may run for any profile or for a specific profile only.
# Syntax: [ 'profile', 'module' ]
# profile: ',' separated list of profiles, or "*' for any profile
# module : Module name.
# The backend plugin is a Perl module and follows the standard 
# Perl module conventions, with at least two additional functions: Init() and run(). 
# See demoplugin.pm for a simple template.
# The frontend plugin is a PHP script with dedicated functions 
# <modulename>__ParseInput(), as well as <modulename_Run();
 
@plugins = (
	# profile # module
	[ '*', 'demoplugin' ],
);

BASEDIR/bin/nfsen reload

Writing Backend Plugins

#
#
package PluginName;

use strict;
use NfConf;

sub run {
	my $profile = shift;
	my $timeslot = shift; # Format: yyyymmddHHMM

	# Do whatever you want to do.
}

sub Init {
	return 1;
}

1;

#
# Demo plugin for NfSen
#
# This plugin demonstrates the use of plugins
# It searches for flows > 500000 packets and sends the output
# using the Notification module

package demoplugin;

use strict;
use NfSen;
use NfConf;

#
# The plugin may send any messages to syslog
# Do not initialize syslog, as this is done by 
# the main process nfsen-run
use Sys::Syslog;
Sys::Syslog::setlogsock('unix');
 
# Use the optional Notification module
use Notification;

my ( $nfdump, $BASEDIR );

#
# Define a nice filter: 
# We like to see flows containing more than 500000 packets
my $nf_filter = 'packets > 500000';

#
# Periodic function
# input: profilename
# timeslot. Format yyyymmddHHMM e.g. 200503031200
sub run {
	my $profile = shift;
	my $timeslot = shift;
	
	syslog('debug', "demoplugin run: Profile: $profile, Time: $timeslot");

	my %profileinfo = NfSen::ReadProfile($profile);
	my $netflow_sources = "$BASEDIR/profiles/$profile/$profileinfo{'sourcelist'}";

	# 
	# process all sources of this profile at once
	my @output = `$nfdump -M $netflow_sources -r nfcapd.$timeslot '$nf_filter'`;
 
	#
	# Process the output and notify the duty team

	# The 2nd but last line contains the stat info of the processed flow:
	# e.g. Flows analysed: 61 matched: 61, Bytes read: 3168
	if ( $output[-2] =~ /matched:\s+(\d+)/ ) {
	my $matched = $1;
	syslog('debug', "demoplugin run: $matched flows");
	if ( $matched ) {
	notify("Packets > 500000: Profile $profile, Timeslot $timeslot", \@output );
	} else {
	syslog('err', "Unparsable output line '$output[-2]'");
	}


	# .... more to come
 
}

sub Init {
	syslog("info", "demoplugin: Init");

	# Init some vars
	$nfdump = "$NfConf::PREFIX/nfdump";
	$BASEDIR = "$NfConf::BASEDIR";

	return 1;
}

sub BEGIN {
	syslog("info", "demoplugin BEGIN");
	# Standard BEGIN Perl function - See Perl documentation
	# not used here
}

sub END {
	syslog("info", "demoplugin END");
	# Standard END Perl function - See Perl documentation
	# not used here
}


1;

Testing Backend Plugins

./testPlugin -p <pluginname> -P <profile> -t <timeslot>

Writing Frontend Plugins

	<input type='text' name='<? echo "${plugin_id}_port";?>' value='' size='5' maxlength='5' >

<?

/*
 * demoplugin frontend plugin template
 */

// Required functions
/* 
 * This function is  called prior to any output to the web browser and is intended
 * for the plugin to parse possible form data. This function is called only, if this
 * plugin is selected in the plugins tab
 */

function demoplugin_ParseInput( $plugin_id ) {

	// Set possible warning or error message
	$_SESSION['warning'] = "Warning set by demo plugin!";
	$_SESSION['error'] = "Error set by demo plugin!";

	// Parse any GET[''] or POST[''] vars

} // End of demoplugin_ParseInput

/*
 * This function is called after the header with the navigation bar have been sent to the
 * browser. It's now up to this function what to display.
 * This function is called only, if this plugin is selected in the plugins tab
 */

function demoplugin_Run( $plugin_id ) {

	print "<h3>Hello I'm the demo plugin with id $plugin_id</h3>\n";

} // End of demoplugin_Run

?>

Optional Modules

Notification.pm

nfsen.conf:
....
#
# Notification module
# The Notification module is an optional module. If you want your plugins to
# notify any result by email, use this module.
# Make sure you have installed Mail::Internet before using the module
#
# Use this from address 
$MAIL_FROM = 'your@from.example.net' ;

# Use these recipients
$RCPT_TO = 'any@example.net', another@example.net" ;

# Use this SMTP server
$SMTP_SERVER = 'localhost' ;


Somewhere in your plugin:

use Notification;
...

@output = some command;
notify("Suspicious Flows found", \@output);