An alchemists view from the bar

Network Security Alchemy

Posts Tagged ‘openfpc

Installing and getting started with OpenFPC 0.9

with 2 comments

Installing OpenFPC – Version 0.9.

Hi, this is a simple walk through of installing OpenFPC on Ubuntu LTS 14.04, although the steps should be similar for any Debian based distribution. Getting things running should be pretty simple, but there are couple of gotchas along the way. For the impatient and those that only want the highlights, here is the process at a high level:

  •  Install the Ubuntu package dependencies that are in the Ubuntu package archives. Note that the install script will also check these for you.
  • Download & install cxtracker
  • Download OpenFPC
  • Untar and run the OpenFPC install script (
    At this point you could go and edit the example config files that were placed in /etc/openfpc, but instead I suggest you get things functioning in a default configuration before trying to over complicate things.
  • Create a user for OpenFPC and set the password them.
  • Create the session database
  • Start OpenFPC
  • Grab a coffee and wait for some packets to come in
  • Try out some basic searches and traffic extraction.

All of the below sections talk though the details of how to achieve the above.

For your first time using OpenFPC, I stringly suggest you start off with the default installation. You can get to the advanced functions like proxy nodes later. By default you will have a single device sniffing traffic and connections from eth0 with a name of Default_Node.

# Install package dependencies:
$ sudo apt-get install \
daemonlogger \
tcpdump \
tshark \
libdatetime-perl \
libprivileges-drop-perl \
libarchive-zip-perl \
libfilesys-df-perl \
mysql-server \
libdbi-perl \
libterm-readkey-perl \
libdate-simple-perl \
libdigest-sha-perl \
libjson-pp-perl \
libdatetime-perl \
libswitch-perl \

# Download Cxtracker
Cxtracker is a connection capturing tool designed for general nsm functions. In the context of OpenFPC it finds connections on the network and stores them to disk in a CSV file. A second program (openfpc-cx2db) then parses these session files and uploads them to the OpenFPC session database. This session database allows you to search for network traffic very quickly and identify the sessions you would like to extract. In OpenFPC the connection data is not centrally stored, instead an OpenFPC proxy can aggregate a single search and make it take place across multiple nodes (the things capturing session and packat data), and then combine the results into one dataset for the user.

lward@dev-ny:~$ wget
2014-09-17 07:47:20 (153 MB/s) – ‘cxtracker_0.9.5-1_i386.deb’ saved [12116/12116]

lward@dev-ny:~$ sudo dpkg -i cxtracker_0.9.5-1_i386.deb

# Download OpenFPC.

This documentation was created for openfpc-0.9.5, and documentation has a bad habit of getting out of date quickly. The installation process shouldn’t change much between minor releases, so I suggest you go and install the latest release and hope that these docs are still relevant for it.

lward@dev-ny:~$ wget

# Extract and install OpenFPC
Before you run the installer, there are likely a couple of things you should note.
– Because openfpc-queued needs to use tcpdump to extract session data that is stored on disk, the Ubuntu apparmour profile that prevents it from *reading* files anywere outside of a users home directory isn’t viable. The installer will disable apparmour for tcpdump (and only tcpdump) by creating /etc/apparmor.d/disable/usr.sbin.tcpdump. If you don’t want this, make sure you re-enable it, or edit the installer to not do this. Note that you’ll have to make sure that all pcap operations take place in the openfpc user’s ~, and that’s less than ideal for a file organization point of view.

  • A node called “Default_Node” is created by default. To change its configuration you can edit /etc/openfpc/openfpc-default.conf
  • A user called openfpc is added to the system for all components to drop privileges to (you don’t want daemons running as root)
  • Pay attention for any errors that pop up

lward@dev-ny:~$ tar -zxvf openfpc-0.9.tgz

lward@dev-ny:~/openfpc-0.9$ sudo ./ install

* OpenFPC installer – Leon Ward ( v0.9
* A set if scripts to help manage and find data in a large network traffic
* archive.


[*] Detected distribution as DEBIAN

[*] Installation Complete

# Create a user for OpenFPC.
The location checked for the openfpc password file is defined in the instance configuration file. For us in our simple install that’s /etc/openfpc/openfpc-default.conf that was created when running In that file you’ll notice a line that defines where to look for a passwd file, our default config looks for /etc/openfpc/openfpc.passwd.

lward@dev-ny:~/openfpc-0.9$ sudo openfpc-password -a add -u admin \
-f /etc/openfpc/openfpc.passwd
Creating new user file /etc/openfpc/openfpc.passwd…
[*] Adding user admin
Enter new password:
Retype password:
Password Okay
[*] Done.

# Create the session database.
To make database creation simple, there is a tool for creating and dropping the correct database that matches the configuration you define in the openfpc config file (in our simple default that’s /etc/openfpc/openfpc-default.conf).
openfpc-dbmaint uses the data in that config file to create the database with the expected permissions. This tool requires you to have root access to use. There are multiple database types that can be created, in our simple default example you only will need a session DB. For more options you can see openfpc-dbmain –help.

lward@dev-ny:~/openfpc-0.9$ sudo openfpc-dbmaint create session /etc/openfpc/openfpc-default.conf
[*] Enter mysql “root” credentials to connect to your local mysql server in order to create the databases
DB root Username: root
DB root Password:

Enter password:
[*] Working on Instance /etc/openfpc/openfpc-default.conf .
Would you like session capture ENABLED on Default_Node? (y/n)y
[-] Enabling session capture in Default_Node config
[-] Found cxtracker.
[*] Creating Session database on Default_Node
– Session DB Created
– Adding function INET_ATON6 to DB ofpc_session_default
[*] Restarting OpenFPC Node Default_Node
Stopping Daemonlogger… Not running
Stopping OpenFPC Queue Daemon (Default_Node)… Not running
Stopping OpenFPC cxtracker (Default_Node)… Not running
Stopping OpenFPC Connection Uploader (Default_Node)… Not running
Starting Daemonlogger (Default_Node)… Done
Starting OpenFPC Queue Daemon (Default_Node)… Done
Starting OpenFPC cxtracker (Default_Node)… Done
Starting OpenFPC Connection Uploader (Default_Node) … Done

After creating the database you’ll notice that openfpc is automatically restarted, as openfpc wasn’t running before we executed this command you’ll notice that it starts up. Hopefully you’ll have output like the above.
# Testing the install and getting started.
You can start, stop and check the status of openfpc using the openfpc command. Passing the -v (for verbose) will provide you with some information about the configuration of the system. In the below output you can see that there are two instances configured on my system, one is DISABLED (Example_Proxy), and another is active (Default_Node).
lward@Dev:~/openfpc$ sudo ./openfpc -a status -v
[*] OpenFPC instance openfpc-example-proxy.conf
– NODENAME: Example_Proxy
– DESCRIPTION: “An example OpenFPC Proxy config.;
– PORT: 4243
– PASSWORD FILE /etc/openfpc/openfpc.passwd
[*] OpenFPC instance openfpc-default.conf
– NODENAME: Default_Node
– DESCRIPTION: “An OpenFPC node.;
– PORT: 4242
– PASSWORD FILE /etc/openfpc/openfpc.passwd
– PACKET STORE: /var/tmp/openfpc/pcap
– openfpc-daemonlogger is /usr/bin/daemonlogger
Daemonlogger (Default_Node) : Running
– openfpc-queued is /usr/bin/openfpc-queued
OpenFPC Queue Daemon (Default_Node): Running
– openfpc-cxtracker is /usr/bin/cxtracker
OpenFPC Connection Tracker (Default_Node) : Running
– openfpc-cx2db is /usr/bin/openfpc-cx2db
OpenFPC Connection Uploader (Default_Node) : Running

To actually interact with you OpenFPC Node (Default_Node), you can use the openfpc-client. The openfpc-client is a client application that talks with either an OpenFPC Node or OpenFPC Proxy over the network. This allows you to use a local tool on your workstation to search, extract, save and fetch pcaps from the remote device capturing data. By default openfpc-client tries to connect to the server localhost on TCP:4242. Check openfpc-client –help to find out how to specify a remote node (–server –port).

lward@dev-ny:~$ openfpc-client -a status

* openfpc-client 0.9 *
Part of the OpenFPC project -

Username: admin
Password for user admin :
Status from: Default_Node
* Node: Default_Node
- Node Type : NODE
- Description : "An OpenFPC node."
- Packet storage utilization : 7 %
- Session storage utilization : 7 %
- Space available in save path : 7 %
- Space used in the save path : 2047640 (2.05 GB)
- Session storage used : 2047640 (2.05 GB)
- Packet storage used : 2047640 (2.05 GB)
- PCAP file space used : 156M
- Local time on node : 1410955045 (Wed Sep 17 07:57:25 2014 America/New_York)
- Newest session in storage : 1410954011 (Wed Sep 17 07:40:11 2014 America/New_York)
- Oldest session in storage : 1410441644 (Thu Sep 11 09:20:44 2014 America/New_York)
- Oldest packet in storage : 1410353440 (Wed Sep 10 08:50:40 2014 America/New_York)
- Storage Window : 5 Days, 22 Hours, 19 Minutes, 27 Seconds
- Load Average 1 : 0.00
- Load average 5 : 0.01
- Load average 15 : 0.05
- Number of session files lagging : 0
- Number of sessions in Database : 8
- Node Timezone : America/New_York

In the output above I can see some important status information about this device. Note the amount of data captured, disk usage, and session database size. The session database will auto-trim to only keep session data for the packets that are available for extraction. Make sure you have some data captured and lets go grab some full packet data.

Here I will simply ask to fetch (extract and send to my workstation) all traffic to a destination port of 53 in the last 10 minutes. For more advanced constraints check out openfpc-client –help.

lward@dev-ny:~$ openfpc-client -a fetch -dpt 53 –last 600

* openfpc-client 0.9 *
Part of the OpenFPC project –

Username: admin
Password for user admin :
Date : Wed Sep 17 07:58:56 2014
Filename: /tmp/pcap-openfpc-1410955136.pcap
Size : 17K
MD5 : 938638229b7e508646e5dbbb3ba231b3

The above shows me the filename I’ve just created, by default the pcap file is written to /tmp, you can choose a better filename with the -w option. If we look at the contents of this file we will see the full packet contents.

lward@dev-ny:~$ tshark -r /tmp/pcap-openfpc-1410955136.pcap
1 0.000000000 -> DNS 76 Standard query 0x17d0 A
2 0.000004000 -> DNS 76 Standard query 0x34c8 AAAA
3 0.034045000 -> DNS 108 Standard query response 0x17d0 A A

To save you performing large extractions to see if sessions that match your constraints exist you can use the –search option. The –search option asks openfpc to look though its session database to find out if the traffic you’re interested in exists. This is much faster than actually extracting the full pcap data itself.
lward@dev-ny:~$ openfpc-client -a search -dpt 53

* openfpc-client 0.9 *
Part of the OpenFPC project –

Username: admin
Password for user admin :
Custom Search
Start: Wed Sep 17 07:01:22 2014 (America/New_York)
End : Wed Sep 17 08:01:22 2014 (America/New_York)
Node : Default_Node
Rows : 4
SQL : SELECT start_time,INET_NTOA(src_ip),src_port,INET_NTOA(dst_ip),dst_port,ip_proto,src_bytes, dst_bytes,(src_bytes+dst_bytes) as total_bytes
FROM session IGNORE INDEX (p_key) WHERE unix_timestamp(CONVERT_TZ(`start_time`, ‘+00:00′, @@session.time_zone))
between 1410951682 and 1410955282 AND dst_port=’53’ ORDER BY start_time DESC LIMIT 20
Row Start Time Source IP sPort Destination dPort Proto Src Bytes Dst Bytes Total Bytes Node Name
0 2014-09-17 7:07:17 48755 53 udp 14828 18924 33752 Default_Node
1 2014-09-17 7:07:17 34676 53 udp 14828 31724 46552 Default_Node
2 2014-09-17 7:07:17 41495 53 udp 14828 20204 35032 Default_Node
3 2014-09-17 7:07:44 46496 53 udp 34264 53976 88240 Default_Node

One of the more useful features of OpenFPC is to actually request data in in the formats outputted by different tools. This enables you to simply ‘paste’ the log line from some tool into openfpc-client and it will go grab the session for you. Unsurprisingly OpenFPC supports the search format as one of these log formats. This means for any session that we find in the database with the search action, we can go and ask for is with a fetch (or store) action. E.g.

$ openfpc-client -a fetch –logline ” 1 2014-09-17 7:07:17 34676 53 udp 14828 31724 46552 Default_Node”

* openfpc-client 0.9 *
Part of the OpenFPC project –

Username: admin
Password for user admin :
Date : Fri Oct 3 16:57:14 2014
Filename: /tmp/pcap-openfpc-1412351834.pcap
Size : 660
MD5 : 39fdb557d751b2cebe31b2d5b9aa5d3c

Hopefully this is enough information to get you started!


Written by leonward

October 3, 2014 at 7:00 pm

Posted in Security, snort

Tagged with

Big OpenFPC release – 0.6

with 2 comments

Pushing forwards closer to a 1.0 release for OpenFPC, one of the major components has now been updated – The GUI.

To introduce this new release I’ve put together a short screen-cast of OpenFPC to show the installation, setup procedure, and a bit of general usage. So if you’re tasked with rolling together your own full packet capture/network traffic recorder/forensics system, perhaps you may want to take a look below.


For those who don’t want to sit through five minutes of video to see what the new GUI looks like, here are a few screenshots of the system in action.

Version 0.6 is now available at . Expect a few bugs, and if you report them, Ill own the task of fixing them.


Written by leonward

June 13, 2011 at 12:37 pm

Posted in OpenFPC, Security, snort, Uncategorized

Tagged with , , ,

A new look for OpenFPC – New GUI in devopment

with 2 comments

Developing open source software has its ups and downs. It’s great to hear that your work is helping others solve problems they have, but on the flip-side some people simply love to focus on negatives and never offer to help improve through collaboration.

A user of OpenFPC recently decided they didn’t like the web UI much, and rather than simply complaining about it, they decided to collaborate and work on an overhaul. It’s efforts of people like this that make OSS all the more rewarding.

The UI isn’t quite ready to be released in an installable form, but I thought I would provide a couple of screenshots to wet current users appetite. David, thanks for your effort!

Written by leonward

April 25, 2011 at 6:56 pm

Posted in OpenFPC, Security

Tagged with , ,

An OpenFPC Example: Clustering packet capture over multiple links/devices/countries.

with 3 comments

It’s been a while since my last post, but it’s because I’ve been busy working on ofpc. To rectify that, I thought I would share some of the concepts that are behind how OpenFPC should be able to grow rapidly into a distributed system.

One of the more useful features of ofpc is its self-referencing method for scaling out master/master/slave devices. This concept gets interest when I explain it to people, however it’s not really documented anywhere. So let me introduce it here with a working example……

There are a few common situations where the master/slave relationship can provide real value via clustering.

  • Geographically separated network links with guaranteed or possible asymmetric traffic paths
  • Multi-link trunks
  • High(er) speed links where you need to spread traffic load over multiple slaves

Firstly, please forgive my terrible retro-diagram skills.

OpenFPC Cluster diagram

So here’s the situation:

There are two pipes between network “A” and network “B”, and for whatever the reason, you don’t know if the traffic you want to grab from the buffer could be in the archive of SLAVE1 or SLAVE2. You do know however it’s going to be in one or more of them. Combined they become one *logical* network link.

By requesting the data from the Master queue daemon responsible for these two devices (MASTER in the diagram here), without specifying which slave you want to route your request to, it will search/extract from all of the slaves below it. The master ofpc-queued doesn’t need to be on a separate bit of hardware, it’s just represented in the diagram that way.

Here’s an example of it functioning in my test environment.

lward@UbuntuDesktop:~/code/openfpc$ ./  -a fetch \
 --src-addr= --dst-port=22
* 0.1 *
Part of the OpenFPC project
Username: master
Password for user master : 
Filename: /tmp/extracted-ofpc-1284615954.pcap
Size    : 7.0M
MD5     : a495c1f38dce3dc9dff50ead47a415ab


This ofpc request provided me with a 7MB pcap file made up from the traffic seen by “slave1” and “slave2”, it’s all merged together so I can inspect the traffic as the logical link processes it rather than what can be captured on one physical leg of the link. This isn’t limited to a maximum of two slaves, it can of course be many many more.

If for any given reason I would still prefer to only look at the traffic on one slave, I can either:

  • Make an ofpc request directly to one of the ofpc-slave devices
  • Specify the device to focus on to the master

For example…..

lward@UbuntuDesktop:~/code/openfpc$ ./  -a fetch \
--src-addr= --dst-port=22 -o 4240 --device slave2
* 0.1 *
Part of the OpenFPC project
Username: master
Password for user master :
Filename: /tmp/extracted-ofpc-1284616271.pcap
Size    : 6.0M
MD5     : 68132e2e12c16665913cb1e7f36336f3

If you want to test this feature out, make sure you’re using the latest openfpc code out of svn.


Written by leonward

September 24, 2010 at 12:42 pm

Posted in OpenFPC, Security

Tagged with , , ,

OpenFPC Test Release

leave a comment »

The weekend has landed, and I have time to pull together some of the bits I need for an OpenFPC (Open Full Packet Capture) release, but I need your help.

I know there are bugs that still need squishing (Master-mode install script for example), but if you have time and are interested, please help me test out an alpha release. Go grab it from here (download the latest version number, it may change repeatedly over the next few days) and run the installer.

So far, I have only tested it on Ubuntu 10.4, the Redhat auto-dependency checking isn’t there yet but it should work on that platform if you have the required RPMs installed with a little tweaking.

So what are you waiting for!? Find problems, tell me where the install and setup falls down, and have some fun.


Written by leonward

September 10, 2010 at 5:35 pm

Posted in OpenFPC, Security

Tagged with , ,

OpenFPC – An update: v0.2.97 available (woohoo!)

with 4 comments

It’s been a couple of months since I first posted about the OpenFPC project, so I thought it’s time that I provided a little update.

Firstly, I need to throw some karma over to Edward Fjellskål (, so… Edward++.

Edward and I have merged the OpenFPC and FPCGUI projects, it makes way more sense to combine our efforts as our goals are similar while our approaches have been from different angles. We both see a need to unify all of the home-brew full-packet-capture/network forensics tools we see out there in the wild.

OpenFPC now has a new home,  So, if you’re looking for a distributed wrapper for your daemonlogger instances, or if you’re still trying to get tcpdump to log in a ringbuffer and share access over multiple analysts, devices, and tools, head on over to to read all about it. Here are a couple of quick links for those who want to jump right in:

I’m looking for people to help test and provide feedback now so I can fix problems and tweak things ahead of a full release.

Good luck, and please let me know your feedback.


Written by leonward

August 2, 2010 at 9:53 pm

Posted in OpenFPC, Security

Tagged with , , , , , , ,