vermont/README.md

# VERMONT - VERsatile MONitoring Tool

[![Build Status](https://travis-ci.org/tumi8/vermont.svg?branch=master)](https://travis-ci.org/tumi8/vermont)

Vermont is an open-source software toolkit for the creation and processing of network flow data, based on monitored Internet packet data.
See the [Wiki](https://github.com/tumi8/vermont/wiki) for full details of all its features.

## REQUIREMENTS

VERMONT has been tested on Linux and FreeBSD systems.

For compilation, GNU C/C++ compiler and standard libraries are required,
as well as the following Ubuntu/Debian packages (or equivalent packages
of other Linux distributions):
 - cmake 
 - libboost-filesystem-dev
 - libboost-regex-dev
 - libboost-test-dev
 - libboost-thread-dev
 - libxml2-dev 
 - libpcap-dev 
 - libsctp-dev (if not available, disable cmake option SUPPORT_SCTP)

The following packages are optional:
 - cmake-curses-gui (ccmake, interactive user interface of cmake)
 - libpq-dev (for PostgreSQL support)
    ==> cmake option SUPPORT_POSTGRESQL
 - libmysqlclient-dev (for MySQL support)
    ==> cmake option SUPPORT_MYSQL
 - libgsl-dev (for connection-based sampling with Bloom filters)
    ==> cmake option USE_GSL
 - libczmq-dev (for receiving IPFIX reports over ZMQ)
    ==> cmake option SUPPORT_ZMQ

For DTLS support, OpenSSL 1.0.0 or higher is required. It is recommended 
to build OpenSSL based on the latest CVS revision. See DTLS instructions below.


## BUILDING AND INSTALLATION

This project uses cmake for setting platform- and user-specific compile 
options. In order to generate the Makefile for actual compilation, you 
need to call in the root of the source directory:
``` shell
$ cmake .
```

In order to change the default compile options, use:
``` shell
$ cmake -D OPTION1=value1 -D OPTION2=value2 ...
```

To get a list of the most important options, call:
``` shell
$ cmake -LH
```

As a user-friendly alternative, you can use the interactive user interface.
Please note that this requires the package cmake-curses-gui, if you are using
Ubuntu/Debian.
``` shell
$ ccmake .
```

If some libraries are installed in custom directories, use:
``` shell
$ cmake -D CMAKE_PREFIX_PATH=/custom/directory1:/custom/directory2

```

After successfully generating the Makefile with cmake, start the 
compilation with:
``` shell
$ make
```

Although not strictly necessary, VERMONT binaries and data files can be 
copied to the usual install location by running:
``` shell
$ make install 
```


### BUILDING WITH DTLS-OVER-UDP SUPPORT

VERMONT's DTLS support is based on OpenSSL version 1.0.0 (and maybe higher). 

Since the DTLS implementation in OpenSSL is fairly new and not as mature as 
the TLS/SSL implementation, you should use the latest version of OpenSSL which 
you can get from http://openssl.org/source/.

At the time of writing (July 2010), the latest version is 1.0.0a.
``` shell
$ wget http://openssl.org/source/openssl-1.0.0a.tar.gz
$ tar xzf openssl-1.0.0a.tar.gz
$ cd openssl-1.0.0a/
```

If you want to profit from the most recent bugfixes, you can check out the 
sources from the OpenSSL CVS repository instead:
``` shell
$ cvs -z9 -d anonymous@cvs.openssl.org:/openssl-cvs co openssl
$ cd openssl/

```

In order to avoid incompatibilities with other packages of your distribution,
you probably do not want the new version of OpenSSL to become the default 
OpenSSL library on your system. Therefore, it is recommended to install the 
new version in a local directory by using the --prefix option of the config
script.

To build OpenSSL and install it into a built/ subdirectory within the OpenSSL
source directory, call the following commands:
``` shell
$ ./config -d no-dso no-shared --prefix=`pwd`/built
$ make 
$ make install
```

The configure option "no-dso" turns off the use of shared-library methods which 
avoids linking problems related to libdl on the Linux platform.
With the option "no-shared", only static libraries are built which makes it 
easier to link VERMONT to the correct version of OpenSSL.

In order to compile VERMONT with DTLS-over-UDP support, change into the root
of VERMONT's source directory and execute cmake with the OpenSSL include and 
library paths (replace "/path/to/openssl" by your OpenSSL source directory):
``` shell
$ cmake -DSUPPORT_DTLS=YES -DCMAKE_INCLUDE_PATH=/path/to/openssl/built/include -DCMAKE_LIBRARY_PATH=/path/to/openssl/built/lib
```

On 64 bit platforms, the library path might be different (mind the "64" at the 
very end!):
``` shell
$ cmake -DSUPPORT_DTLS=YES -DCMAKE_INCLUDE_PATH=/path/to/openssl/built/include -DCMAKE_LIBRARY_PATH=/path/to/openssl/built/lib64
```

If you have previously built VERMONT with OpenSSL located in another 
directory, you might need to manually remove the file CMakeCache.txt before 
calling cmake.


### BUILDING WITH DTLS-OVER-SCTP SUPPORT

At the time of writing (July 2010), DTLS over SCTP can be used on FreeBSD only!
This is due to the fact that FreeBSD is currently the only OS which supports 
the SCTP-AUTH extension (see RFC 4895) which is required by DTLS.

The current version of OpenSSL (1.0.0a) has no native support for SCTP. You 
have to download additional patches from

http://sctp.fh-muenster.de/

and apply them to the OpenSSL sourcese before building OpenSSL. Make sure that
the patches fit to your local version of OpenSSL. Otherwise, you might need to
manually adapt the patch files.

Also, make sure to add the command line argument "sctp" when running OpenSSL's
./config to build SCTP support into OpenSSL.

In order to compile VERMONT with DTLS-over-SCTP support, you need to run cmake
with the following three options:

`-DSUPPORT_SCTP -DSUPPORT_DTLS -DSUPPORT_DTLS_OVER_SCTP`

In addition, you need to indicate the include and library paths to your patched
version of OpenSSL as explained for DTLS-over-UDP.


## USAGE AND CONFIGURATION

In order to run VERMONT, a configuration file is needed which specifies the 
modules to be used and their parameters:
``` shell
$ ./vermont -f <config-file>
```

Example configuration files can be found in [`configs/`](config).
Documentation of the available modules and their configuration parameters
can be found at https://github.com/tumi8/vermont/wiki/Moduleconfiguration

Use `Ctrl-C` to stop VERMONT. If VERMONT does not exit properly, enter Ctrl-C
for a second time.


## OPERATION AS COLLECTOR: TUNING SOCKET RECEIVE BUFFER

VERMONT can be used as an IPFIX/PSAMP and NetFlow.v9 collector. As the 
incoming IPFIX/PSAMP/NetFlow messages usually arrive in bursts, losses 
may occur due to insufficient buffer space.

As a solution, the socket receive buffer can be increased. To check the
current settings, use the following system calls on Linux systems with 
`/proc` file system:
``` shell
$ cat /proc/sys/net/core/rmem_default
$ cat /proc/sys/net/core/rmem_max
```

In order to configure a new value X (in bytes), call:
``` shell
$ sysctl -w net/core/rmem_default=X
$ sysctl -w net/core/rmem_max=X
```

If you want Vermont to use a different buffer size than the default one,
you can specify it using the `<buffer>` directive in the `<listener>` section.


## OPTIMIZED PACKET CAPTURING WITH PCAP

To reduce the number of times packets need to be copied on their way from
the network interface card to the user space (i.e., VERMONT), we recommend 
the utilization of pcap library 1.0.0 or higher.

For earlier versions of pcap, the pcap-mmap patch can be applied to 
improve the performance: http://public.lanl.gov/cpw/


## EFFECTS OF RECEIVE OFFLOAD MECHANISMS

Several mechanisms have been implemented in modern network interface cards,
drivers, and kernels to offload common functions from the protocol stack and 
the application. One particular focus is on TCP segmentation and reassembly.

Receive offload mechanisms aim at reassembling subsequent TCP segments into
a single large packet before passing it to the IP/TCP protocol stack and 
finally to the application. In the Linux kernel, this is done by generic 
receive offload (GRO) if the network interface card and the driver support 
NAPI. Latest Intel 10GE controllers (e.g., 82599) support receive side 
coalescing (RSC) which performs TCP reassembly in hardware.

If any receive offload mechanism is enabled, VERMONT (like any other 
pcap-based application) does not observe the actually captured TCP packets 
but the reassembled ones. One consequence is that packet counts of flows will 
be smaller than the true number of packets.

In order to avoid such distortions, all receive offload mechanisms need to 
be disabled. In the case of GRO (and the older LRO), this can be done with 
ethtool. The following call returns a list of the current status of all 
offload mechanisms for interface <dev>:
``` shell
$ ethtool -k <dev>
```

If GRO is not shown, you probably need to install a newer version of ethtool.
To disable GRO (and LRO), execute:
``` shell
$ ethtool -K <dev> gro off
$ ethtool -K <dev> lro off
```

Hardware-based RSC can be deactivated at compile time of the driver as 
explained here:
http://downloadmirror.intel.com/19004/eng/README-2.0.72.4.txt