MAJI - An implementation of an IPFIX meter

Version 1.0.2

Maji is Copyright (c) 2008-2011 The University of Waikato, Hamilton, 
New Zealand.
All rights reserved.

See the file COPYING for full licensing details for this software.

A detailed list of changes in this version of Maji can be found at 
https://secure.wand.net.nz/trac/traceflow_ipfix/wiki/ChangeLog

Report any bugs, questions or comments to contact@wand.net.nz

Authors:
	Shane Alcock (WAND)

With contributions from:
	Perry Lorier (WAND)
	Benjamin Black (fastip.com)
	Lance Fitz-Herbert (WAND)
	Refael Whyte (WAND)
	Rong Zheng


Introduction
============
Maji is an implementation of an IPFIX meter, developed by the WAND Research
Group at the University of Waikato, New Zealand. Maji has been developed in
close consultation with the IPFIX draft specifications (some of which are now
published RFCs) but it is not a feature-complete implementation at this stage. 
Maji does not provide any form of useful IPFIX collector, nor does it come with 
any tools for processing the IPFIX records produced by the meter. 

Required Libraries
==================
libtrace 3.0.11 or better
	* available from http://research.wand.net.nz/software/libtrace.php

libconfuse 
	* available from http://www.nongnu.org/confuse/
	* Debian / Ubuntu users can install the libconfuse-dev package instead

libsqlite3
	* optional but required for SQL export
	* available from http://www.sqlite.org/download.html
	* Debian / Ubuntu users can install the libsqlite3-dev package

libsctp
	* optional but required for SCTP export
	* Debian / Ubuntu users can install the libsctp-dev package

libwandevent 2.0.0 
	* available from http://research.wand.net.nz/software/libwandevent/libwandevent-2.0.0.tar.gz

openssl
	* optional but required for TLS transport
	* 1.0.0 required for DTLS transport
	* available from http://www.openssl.org/
	* Debian / Ubuntu users can install the libssl-dev package

Installation
============
After having installed the required libraries, running the following series
of commands should install maji

	./configure
	make
	make install

By default, maji installs to /usr/local/maji - this can be changed by
appending the --prefix=<new location> option to ./configure.

If you require SQL support, be sure to check after the ./configure stage that
maji will be compiled with SQL support. If not, check that you have 
libsqlite3-dev installed on your machine. If it is installed in a non-standard
location, you may need to add LDFLAGS="-L<non standard library location>" and
CPPFLAGS="-I<non standard header file location"> to the ./configure command.

If you require TLS support, use the --with-openssl[=path] option to ./configure
where path is the location of the SSL shared library.

Running Maji
============
Maji can be run using the following command:

	maji -c <maji config file> 

The libtrace uri tells Maji where it should capture/read packets from. The
libtrace wiki (http://www.wand.net.nz/trac/libtrace/wiki/SupportedTraceFormats)
provides details on what trace formats are supported and how the uri's for each
format should look. For example, if you want Maji to meter packets on 
interface eth0 using libpcap the libtrace uri should be 'pcapint:eth0' 
(without the quotes).

The -c option simply tells Maji where to read its configuration options from.
The section "Configuration" below explains how the config file works - you can
also consult the file example.config to see an example configuration file.

Maji can also be run as a daemon by using the 'majid' executable instead of
'maji'. 

One final note, you should use the installed maji executable (by default, 
located in /usr/local/maji/bin) rather than the executable in the
maji source directory. There appear to be some unresolved issues that prevent
that particular executable from running correctly- the installed version has no 
such problems.

Configuration
=============
An example configuration file (called example.config) should be included with
the maji source. Configuration is done using the libconfuse library which has 
its own particular syntax. The configuration options and their syntax are as 
follows:

 * report_freq - Defines how often, in seconds, IPFIX flow records should be
 		 exported for expired flows. Expired flows will not be dealt
		 with outside of these reporting intervals. In addition, this
		 defines how often option template records will be exported.

		 Syntax is as follows:

		 report_freq = <seconds>

 * domain_id - Within IPFIX, measurement data from multiple "Observation
 	       Domains" may be accumulated by a single collector. This option
	       specifies which Observation Domain id number should be 
	       applied to the flow records exported by this meter.
	       
	       Syntax is as follows:

	       domain_id = <id>

 * module_location - Specifies the directories where the information element
 		     modules have been installed so that maji can locate them.
		     By default, the standard elements will all be installed
		     into /usr/local/maji/info_elements - if you
		     have installed maji elsewhere, you'll need to change this
		     option to match that location. 

		     Multiple module locations may be specified if required,
		     for instance if user-defined information elements have
		     been defined in a separate location.

		     Avoid using relative paths when providing a module
		     location.

		     Syntax is as follows:

		     module_location = {
		     	"/module/location/one", "/module/location/two"
		     }
		     
 * template - Describes an IPFIX template. Templates consist of two sets of 
	      information elements, one describing the flow key and the other
	      describing what should be measured per flow. The example config
	      defines a template using the standard 5-tuple as a key and
	      counts packets and bytes for each flow.

	      When defining a template, the information element names must be
	      exactly the same as the name of the source file where the 
	      element is defined in the info_elements directory. For example, 
	      the sourceIPv4Address element is defined in sourceIPv4Address.cc.
	      
	      The syntax for this option is (note the placement of braces and
	      double quotes) :

	      template
	      {
	      		keys = { "key_element_one", "key_element_two",
			"key_element_three" }
			elements = {"per_flow_measure_one", 
			"per_flow_measure_two" }
	      }
	    
 * option_template - Describes an IPFIX option template. Rather than measuring
 		     a distinct flow, option templates measure meta-data about
		     the IPFIX metering process. Option template records are
		     sent once per reporting interval, regardless of how many
		     flows might have expired during that interval. The syntax
		     for option templates is more or less identical to 
		     regular templates.

		     Note that we use the term "scope" instead of "keys" for
		     defining the keys for an option template. This is to 
		     maintain consistency with the IPFIX specifications. 
		     For all intents and purposes, scope and keys are 
		     interchangable terminology.

		     The syntax for this option is:

		     option_template
		     {
		     	scope = { "option_key_one", "option_key_two" }
	      		elements = {"option_measure_one", "option_measure_two" }
		     }

 * input - Specifies a input source where packet data is to be read from by
 	   the meter. The uri option should contain a libtrace URI that
	   describes the input source. See 
	   http://wand.net.nz/trac/libtrace/wiki/SupportedTraceFormats for more
	   details about valid libtrace URIs.

	   Multiple input sources can be specified using separate "input"
	   config options.

	   Syntax:

	   input {
		   uri="<libtrace URI>"
	   }

 * output - Specifies an exporter to be used for outputting IPFIX records.
 	    The uri option describes which exporter to use, along with a 
	    base option (if required). The options field can be used to provide
	    getopt-style command line arguments to the exporter. More details
	    on the options accepted by each of the exporters is provided below
	    in the "Supported Export Formats" section.

	    Finally, the use_tls option can be used to request that the
	    exporter encrypt the records using TLS. If TLS was not enabled
	    during installation, setting this option to true will have no
	    effect. Similarly, only some export formats support using TLS
	    encryption - refer below for more details.

	    Multiple exporters may be specified using separate "output" config
	    options.

	    Syntax:

	    output {
	    	uri="<export format>:<base location or port number>"
		options="<exporter exclusive command line arguments>"
		use_tls="<true|false>"
	    }	

 * tls - Specifies the global TLS configuration paths. The ca_file option 
 	 provides the location of the CA certificate. The cert_file option
	 specifies the location of the certificate itself. The key option
	 describes the location of the key file.

	 Note that this is a global specification. Maji does not currently
	 support using different TLS configuration for individual exporters.

	 Syntax:

	 tls {
		 ca_file="/tls/directory/cacert.pem"
		 cert_file="/tls/directory/cert.pem"
		 key_file="/tls/directory/key.pem"
	 }

Information Elements
====================
All the standard information elements supported by maji are located within
<install directory>/info_elements. At this stage, not all information elements
defined within the IPFIX standards are implemented within maji, but we think
we've covered the most important ones. The biggest absences at the time of
writing were the flow duration and timestamp elements - we have flowStartSeconds
and flowEndSeconds but nothing more. Feel free to submit any information
element requests to us if the element you want isn't implemented yet.

Maji also supports user-defined information elements outside of those defined
in the IPFIX specs. Consult the development notes (in the devnotes/ directory) 
for more details as to how one might go about this.

Supported Export Formats
========================
Maji supports several formats for exporting IPFIX records. Each export type
takes a different set of options which will be described in this section.

 * Standard Output - a.k.a. writing IPFIX records to the terminal, mainly 
   useful for debugging and checking that things are working correctly. Takes
   no options.

   Syntax:
   	output {
		uri="stdout:"
		options = ""
	}

 * SQLite - writes the IPFIX records to an SQLite3 database. You must have
   libsqlite3 installed to be able to use this exporter. The database will be
   named after the time when the database was created. Each template will 
   create a separate table within the database. The table names will be derived
   from the information elements used by the templates. I warn you now, the
   table names are not pretty!

   The SQLite exporter takes options that allow the user to specify a 
   frequency for rotating the database file, i.e. closing the existing
   database and creating a new one. Doing so allows you to keep separate 
   databases for different time periods. The -R option can be specified as
   either "h" or "d" to describe an hourly or daily base rotation respectively.
   The -C option specifies how many times to rotate within the base rotation
   period, e.g. -R d -C 4 would rotate 4 times daily, i.e. every six hours. 
   All rotation times are calculated from the start of the day, not when the
   meter was started so the above example would always rotate at 0000, 0600,
   1200 and 1800 hrs.

   The intended location of the SQLite database files is specified as part
   of the uri, after the first colon. Avoid using a relative path when
   specifying the location.

   Syntax:
   	output {
		uri="sql:<location to store database files>"
		options="-R <'h' or 'd'> -C <times to rotate>"
	}

 * UDP - writes IPFIX records to the network via UDP. This exporter requires
   that a listening collector be available to connect to. Maji comes bundled 
   with a simple UDP collector that will merely print IPFIX records to a 
   terminal. An alternative collector is required to do anything more 
   sophisticated with the received records.
   
   The IP address and port of the collector should be specified as part of the
   uri as shown below.

   The UDP exporter supports three additional options: 
   	-r specifies how frequently template definitions should be 
	retransmitted in seconds. Default is 600 seconds.
	-p specifies that templates should be retransmitted after a certain
   	number of IPFIX messages have been exported. Default is 200 messages.
	-N specifies that the exporter should attempt to write NetFlow 9 
	records rather than IPFIX records. This is an experimental feature.	
   
   The message-based retransmit is designed to be used in conjunction with the
   timer-based one. Retransmission will occur every N seconds or every M 
   messages, whichever occurs first. Once a retransmission is performed, both
   the timer and the message count is reset to zero. In addition, if the -p
   option is not specified, message-based retransmission will not occur at all
   and only the timer will be used. There is no way to disable timer-based
   retransmission.	

   The UDP exporter now supports the use of TLS to encrypt exported records.

   Syntax:
   	output {
		uri="udp:<collector ip>:<collector port>"
		options="-r <retransmit seconds> -p <retransmit messages> [-N]"
		use_tls="<true|false>"
	}
	
 * TCP - writes IPFIX records to the network via TCP. This exporter requires
   that a listening collector be available to connect to. Maji comes bundled 
   with a simple TCP collector that will merely print IPFIX records to a 
   terminal. An alternative collector is required to do anything more 
   sophisticated with the received records.
   
   The IP address and port of the collector should be specified as part of the
   uri as shown below.
   
   The TCP exporter takes one additional option:
	-N specifies that the exporter should attempt to write NetFlow 9 
	records rather than IPFIX records. This is an experimental feature.	

   The TCP exporter now supports the use of TLS to encrypt exported records.

   Syntax:
   	output {
		uri="tcp:<collector ip>:<collector port>"
		options="[-N]"
		use_tls="<true|false>"
	}

 * SCTP - writes IPFIX records to the network via SCTP. This exporter requires
   that a listening collector be available to connect to. Maji comes bundled
   with a simple SCTP collector that will merely print IPFIX records to a
   terminal. An alternative collector is required to do anything more
   sophisticated with the received records.

   The IP address and port of the collector should be specified as part of the
   uri as shown below.

   The SCTP exporter takes one additional option:
	-N specifies that the exporter should attempt to write NetFlow 9 
	records rather than IPFIX records. This is an experimental feature.	

   As per the recently published IETF draft "IPFIX Export per SCTP Stream",
   http://www.ietf.org/internet-drafts/draft-ietf-ipfix-export-per-sctp-stream-01.txt,
   the SCTP exporter included with Maji uses a separate SCTP stream for each
   defined data template. All option templates, however, are exported using a
   single dedicated stream. Ultimately, this should have little apparent 
   effect for a user, though.

   Syntax:
   	output {
		uri="sctp:<collector ip>:<collector port>"
		options="[-N]"
	}

 * Stdout - writes IPFIX records directly to the terminal via stdout. 
   Primarily intended for use in testing and debugging Maji as, aside from
   being able to manually inspect IPFIX records, it has little practical use.

   The stdout exporter takes no additional options.

   Syntax:
   	output {
		uri="stdout:"
		options=""
	}

Collectors
==========
The following collectors are included with maji:
	simple_udp_reader - a UDP collector
	simple_tcp_reader - a TCP collector
	simple_sctp_reader - a SCTP collector

As mentioned above, these collectors are simplistic. All they do is print a
textual description of the received IPFIX records to the terminal and were
primarily developed to test and debug the exporters. Nonetheless, they
are probably a good starting point for anyone looking to develop a more useful
collector. See devnotes/collectors for more details on developing collectors.

All of the simple collectors operate in the same fashion. A collector can be
run using the following command line:
	
	simple_<protocol>_reader -c <directory> [-p <port>] 

Options supported by the collectors:
	-p <port>	Specifies the port to listen for a connection on. By
			default, the collectors listen on port 4739.
	-c <directory>	Specifies the location of the information element
			modules (i.e. the .so libraries for the IEs). This 
			MUST be specified for the collector to run.


