An alchemists view from the bar

Network Security Alchemy

Installing and getting started with OpenFPC 0.9

leave a comment »

Installing OpenFPC – Version 0.9.

Hi, this is a simple walk through of installing OpenFPC on Ubuntu LTS 14.04, although the steps should be similar for any Debian based distribution. Getting things running should be pretty simple, but there are couple of gotchas along the way. For the impatient and those that only want the highlights, here is the process at a high level:

  •  Install the Ubuntu package dependencies that are in the Ubuntu package archives. Note that the install script will also check these for you.
  • Download & install cxtracker
  • Download OpenFPC
  • Untar and run the OpenFPC install script (openfpc-install.sh)
    At this point you could go and edit the example config files that were placed in /etc/openfpc, but instead I suggest you get things functioning in a default configuration before trying to over complicate things.
  • Create a user for OpenFPC and set the password them.
  • Create the session database
  • Start OpenFPC
  • Grab a coffee and wait for some packets to come in
  • Try out some basic searches and traffic extraction.

All of the below sections talk though the details of how to achieve the above.

For your first time using OpenFPC, I stringly suggest you start off with the default installation. You can get to the advanced functions like proxy nodes later. By default you will have a single device sniffing traffic and connections from eth0 with a name of Default_Node.

# Install package dependencies:
$ sudo apt-get install \
daemonlogger \
tcpdump \
tshark \
libdatetime-perl \
libprivileges-drop-perl \
libarchive-zip-perl \
libfilesys-df-perl \
mysql-server \
libdbi-perl \
libterm-readkey-perl \
libdate-simple-perl \
libdigest-sha-perl \
libjson-pp-perl \
libdatetime-perl \
libswitch-perl \
libdatetime-format-strptime-perl

# Download Cxtracker
Cxtracker is a connection capturing tool designed for general nsm functions. In the context of OpenFPC it finds connections on the network and stores them to disk in a CSV file. A second program (openfpc-cx2db) then parses these session files and uploads them to the OpenFPC session database. This session database allows you to search for network traffic very quickly and identify the sessions you would like to extract. In OpenFPC the connection data is not centrally stored, instead an OpenFPC proxy can aggregate a single search and make it take place across multiple nodes (the things capturing session and packat data), and then combine the results into one dataset for the user.

lward@dev-ny:~$ wget http://github.com/downloads/gamelinux/cxtracker/cxtracker_0.9.5-1_i386.deb
<snip>
2014-09-17 07:47:20 (153 MB/s) – ‘cxtracker_0.9.5-1_i386.deb’ saved [12116/12116]

lward@dev-ny:~$ sudo dpkg -i cxtracker_0.9.5-1_i386.deb

# Download OpenFPC.

This documentation was created for openfpc-0.9.5, and documentation has a bad habit of getting out of date quickly. The installation process shouldn’t change much between minor releases, so I suggest you go and install the latest release and hope that these docs are still relevant for it.

lward@dev-ny:~$ wget

# Extract and install OpenFPC
Before you run the installer, there are likely a couple of things you should note.
– Because openfpc-queued needs to use tcpdump to extract session data that is stored on disk, the Ubuntu apparmour profile that prevents it from *reading* files anywere outside of a users home directory isn’t viable. The installer will disable apparmour for tcpdump (and only tcpdump) by creating /etc/apparmor.d/disable/usr.sbin.tcpdump. If you don’t want this, make sure you re-enable it, or edit the installer to not do this. Note that you’ll have to make sure that all pcap operations take place in the openfpc user’s ~, and that’s less than ideal for a file organization point of view.

  • A node called “Default_Node” is created by default. To change its configuration you can edit /etc/openfpc/openfpc-default.conf
  • A user called openfpc is added to the system for all components to drop privileges to (you don’t want daemons running as root)
  • Pay attention for any errors that pop up

lward@dev-ny:~$ tar -zxvf openfpc-0.9.tgz
openfpc-0.9/
openfpc-0.9/etc/
openfpc-0.9/etc/openfpc-default.conf
openfpc-0.9/etc/openfpc-example-proxy.conf
openfpc-0.9/etc/init.d/
openfpc-0.9/etc/init.d/openfpc-daemonlogger
openfpc-0.9/etc/init.d/openfpc-cx2db
openfpc-0.9/etc/init.d/openfpc-queued
openfpc-0.9/etc/init.d/openfpc-cxtracker
openfpc-0.9/etc/routes.ofpc
openfpc-0.9/openfpc-install.sh
openfpc-0.9/OFPC/
openfpc-0.9/OFPC/CXDB.pm
openfpc-0.9/OFPC/Config.pm
openfpc-0.9/OFPC/Request.pm
openfpc-0.9/OFPC/Parse.pm
openfpc-0.9/OFPC/Common.pm
openfpc-0.9/docs/
openfpc-0.9/docs/README
openfpc-0.9/docs/TODO
openfpc-0.9/docs/INSTALL
openfpc-0.9/openfpc-dbmaint
openfpc-0.9/openfpc-cx2db
openfpc-0.9/openfpc-client
openfpc-0.9/openfpc-queued
openfpc-0.9/cgi-bin/
openfpc-0.9/openfpc
openfpc-0.9/openfpc-password
lward@dev-ny:~$

lward@dev-ny:~/openfpc-0.9$ sudo ./openfpc-install.sh install

*************************************************************************
* OpenFPC installer – Leon Ward (leon@openfpc.org) v0.9
* A set if scripts to help manage and find data in a large network traffic
* archive.

* http://www.openfpc.org

[*] Detected distribution as DEBIAN

<SNIP>
[*] Installation Complete

# Create a user for OpenFPC.
The location checked for the openfpc password file is defined in the instance configuration file. For us in our simple install that’s /etc/openfpc/openfpc-default.conf that was created when running openfpc-install.sh. In that file you’ll notice a line that defines where to look for a passwd file, our default config looks for /etc/openfpc/openfpc.passwd.

lward@dev-ny:~/openfpc-0.9$ sudo openfpc-password -a add -u admin \
-f /etc/openfpc/openfpc.passwd
Creating new user file /etc/openfpc/openfpc.passwd…
[*] Adding user admin
Enter new password:
Retype password:
Password Okay
[*] Done.

# Create the session database.
To make database creation simple, there is a tool for creating and dropping the correct database that matches the configuration you define in the openfpc config file (in our simple default that’s /etc/openfpc/openfpc-default.conf).
openfpc-dbmaint uses the data in that config file to create the database with the expected permissions. This tool requires you to have root access to use. There are multiple database types that can be created, in our simple default example you only will need a session DB. For more options you can see openfpc-dbmain –help.

lward@dev-ny:~/openfpc-0.9$ sudo openfpc-dbmaint create session /etc/openfpc/openfpc-default.conf
[*] Enter mysql “root” credentials to connect to your local mysql server in order to create the databases
DB root Username: root
DB root Password:

Enter password:
———————————————————
[*] Working on Instance /etc/openfpc/openfpc-default.conf .
Would you like session capture ENABLED on Default_Node? (y/n)y
[-] Enabling session capture in Default_Node config
Done.
[-] Found cxtracker.
[*] Creating Session database on Default_Node
– Session DB Created
– Adding function INET_ATON6 to DB ofpc_session_default
[*] Restarting OpenFPC Node Default_Node
Stopping Daemonlogger… Not running
Stopping OpenFPC Queue Daemon (Default_Node)… Not running
Stopping OpenFPC cxtracker (Default_Node)… Not running
Stopping OpenFPC Connection Uploader (Default_Node)… Not running
Starting Daemonlogger (Default_Node)… Done
Starting OpenFPC Queue Daemon (Default_Node)… Done
Starting OpenFPC cxtracker (Default_Node)… Done
Starting OpenFPC Connection Uploader (Default_Node) … Done
lward@dev-ny:~/openfpc-0.9$

After creating the database you’ll notice that openfpc is automatically restarted, as openfpc wasn’t running before we executed this command you’ll notice that it starts up. Hopefully you’ll have output like the above.
# Testing the install and getting started.
You can start, stop and check the status of openfpc using the openfpc command. Passing the -v (for verbose) will provide you with some information about the configuration of the system. In the below output you can see that there are two instances configured on my system, one is DISABLED (Example_Proxy), and another is active (Default_Node).
lward@Dev:~/openfpc$ sudo ./openfpc -a status -v
###############################################################################
[*] OpenFPC instance openfpc-example-proxy.conf
– NODENAME: Example_Proxy
– DESCRIPTION: “An example OpenFPC Proxy config. http://www.openfpc.org”;
– STATUS : DISABLED
– PORT: 4243
– PASSWORD FILE /etc/openfpc/openfpc.passwd
###############################################################################
[*] OpenFPC instance openfpc-default.conf
– NODENAME: Default_Node
– DESCRIPTION: “An OpenFPC node. http://www.openfpc.org”;
– STATUS : ENABLED
– PORT: 4242
– PASSWORD FILE /etc/openfpc/openfpc.passwd
– INTERFACE: eth0
– FULL PACKET CAPTURE: ENABLED
– PACKET STORE: /var/tmp/openfpc/pcap
– SESSION DATA SEARCH: ENABLED
– SESSION DATABASE NAME: openfpc
– SESSION LAG: 0
– SESSION INSERT FAIL: 0
– openfpc-daemonlogger is /usr/bin/daemonlogger
Daemonlogger (Default_Node) : Running
– openfpc-queued is /usr/bin/openfpc-queued
OpenFPC Queue Daemon (Default_Node): Running
– openfpc-cxtracker is /usr/bin/cxtracker
OpenFPC Connection Tracker (Default_Node) : Running
– openfpc-cx2db is /usr/bin/openfpc-cx2db
OpenFPC Connection Uploader (Default_Node) : Running

 
To actually interact with you OpenFPC Node (Default_Node), you can use the openfpc-client. The openfpc-client is a client application that talks with either an OpenFPC Node or OpenFPC Proxy over the network. This allows you to use a local tool on your workstation to search, extract, save and fetch pcaps from the remote device capturing data. By default openfpc-client tries to connect to the server localhost on TCP:4242. Check openfpc-client –help to find out how to specify a remote node (–server –port).

lward@dev-ny:~$ openfpc-client -a status

* openfpc-client 0.9 *
Part of the OpenFPC project - http://www.openfpc.org

Username: admin
Password for user admin :
=====================================
Status from: Default_Node
=====================================
* Node: Default_Node
- Node Type : NODE
- Description : "An OpenFPC node. http://www.openfpc.org"
- Packet storage utilization : 7 %
- Session storage utilization : 7 %
- Space available in save path : 7 %
- Space used in the save path : 2047640 (2.05 GB)
- Session storage used : 2047640 (2.05 GB)
- Packet storage used : 2047640 (2.05 GB)
- PCAP file space used : 156M
- Local time on node : 1410955045 (Wed Sep 17 07:57:25 2014 America/New_York)
- Newest session in storage : 1410954011 (Wed Sep 17 07:40:11 2014 America/New_York)
- Oldest session in storage : 1410441644 (Thu Sep 11 09:20:44 2014 America/New_York)
- Oldest packet in storage : 1410353440 (Wed Sep 10 08:50:40 2014 America/New_York)
- Storage Window : 5 Days, 22 Hours, 19 Minutes, 27 Seconds
- Load Average 1 : 0.00
- Load average 5 : 0.01
- Load average 15 : 0.05
- Number of session files lagging : 0
- Number of sessions in Database : 8
- Node Timezone : America/New_York
lward@dev-ny:~$

In the output above I can see some important status information about this device. Note the amount of data captured, disk usage, and session database size. The session database will auto-trim to only keep session data for the packets that are available for extraction. Make sure you have some data captured and lets go grab some full packet data.

Here I will simply ask to fetch (extract and send to my workstation) all traffic to a destination port of 53 in the last 10 minutes. For more advanced constraints check out openfpc-client –help.

lward@dev-ny:~$ openfpc-client -a fetch -dpt 53 –last 600

* openfpc-client 0.9 *
Part of the OpenFPC project – http://www.openfpc.org

Username: admin
Password for user admin :
#####################################
Date : Wed Sep 17 07:58:56 2014
Filename: /tmp/pcap-openfpc-1410955136.pcap
Size : 17K
MD5 : 938638229b7e508646e5dbbb3ba231b3

The above shows me the filename I’ve just created, by default the pcap file is written to /tmp, you can choose a better filename with the -w option. If we look at the contents of this file we will see the full packet contents.

lward@dev-ny:~$ tshark -r /tmp/pcap-openfpc-1410955136.pcap
1 0.000000000 192.168.42.12 -> 192.168.42.1 DNS 76 Standard query 0x17d0 A daisy.ubuntu.com
2 0.000004000 192.168.42.12 -> 192.168.42.1 DNS 76 Standard query 0x34c8 AAAA daisy.ubuntu.com
3 0.034045000 192.168.42.1 -> 192.168.42.12 DNS 108 Standard query response 0x17d0 A 91.189.92.57 A 91.189.92.55
<SNIP>

To save you performing large extractions to see if sessions that match your constraints exist you can use the –search option. The –search option asks openfpc to look though its session database to find out if the traffic you’re interested in exists. This is much faster than actually extracting the full pcap data itself.
lward@dev-ny:~$ openfpc-client -a search -dpt 53

* openfpc-client 0.9 *
Part of the OpenFPC project – http://www.openfpc.org

Username: admin
Password for user admin :
=====================================================================================================================================================
Custom Search
=====================================================================================================================================================
Start: Wed Sep 17 07:01:22 2014 (America/New_York)
End : Wed Sep 17 08:01:22 2014 (America/New_York)
Node : Default_Node
Rows : 4
SQL : SELECT start_time,INET_NTOA(src_ip),src_port,INET_NTOA(dst_ip),dst_port,ip_proto,src_bytes, dst_bytes,(src_bytes+dst_bytes) as total_bytes
FROM session IGNORE INDEX (p_key) WHERE unix_timestamp(CONVERT_TZ(`start_time`, ‘+00:00′, @@session.time_zone))
between 1410951682 and 1410955282 AND dst_port=’53’ ORDER BY start_time DESC LIMIT 20
=====================================================================================================================================================
Row Start Time Source IP sPort Destination dPort Proto Src Bytes Dst Bytes Total Bytes Node Name
0 2014-09-17 7:07:17 192.168.42.12 48755 192.168.42.1 53 udp 14828 18924 33752 Default_Node
1 2014-09-17 7:07:17 192.168.42.12 34676 192.168.42.1 53 udp 14828 31724 46552 Default_Node
2 2014-09-17 7:07:17 192.168.42.12 41495 192.168.42.1 53 udp 14828 20204 35032 Default_Node
3 2014-09-17 7:07:44 192.168.42.12 46496 192.168.42.1 53 udp 34264 53976 88240 Default_Node
=====================================================================================================================================================

One of the more useful features of OpenFPC is to actually request data in in the formats outputted by different tools. This enables you to simply ‘paste’ the log line from some tool into openfpc-client and it will go grab the session for you. Unsurprisingly OpenFPC supports the search format as one of these log formats. This means for any session that we find in the database with the search action, we can go and ask for is with a fetch (or store) action. E.g.

$ openfpc-client -a fetch –logline ” 1 2014-09-17 7:07:17 192.168.42.12 34676 192.168.42.1 53 udp 14828 31724 46552 Default_Node”

* openfpc-client 0.9 *
Part of the OpenFPC project – http://www.openfpc.org

Username: admin
Password for user admin :
#####################################
Date : Fri Oct 3 16:57:14 2014
Filename: /tmp/pcap-openfpc-1412351834.pcap
Size : 660
MD5 : 39fdb557d751b2cebe31b2d5b9aa5d3c

Hopefully this is enough information to get you started!

-Leon

Written by leonward

October 3, 2014 at 7:00 pm

Posted in Security, snort

Tagged with

OpenFPC in 2014

with 2 comments

They say that time files, and they’re right (damn them, whoever they are). Lots of things have been going on in my life over the last year, but hey you likely don’t care about that, you’re here because you’re interested to find out if OpenFPC is still alive and growing… and the answer is yes – but with a bit of a twist.

So here are the big changes and updates you may like to know about.

  • Hosting has been moved from Googlecode SVN to git on github (https://github.com/leonward/OpenFPC)
  • I’ve removed the GUI components from the install because I’m struggling to maintain them. I only *ever* used the command line interface anyway, so I expect many others are the same. They’re still in the same git repo for now, but not included in the installer.
  • Session searching now functions from the command line
  • Distributed session databases, each nodes keeps it’s own session data locally
  • If multiple nodes are all linked by a proxynode*, a session search from that proxy will take place *at* all nodes and all results are combined before transmitting them back to the client
  • Multiple TZs are supported. Each node works correctly in it’s own TZ, and when data is combined from multiple nodes in different TZs it functions
  • Added support for parsing passivedns logs (really cool, I’ll put together a walk though of how that works sometime)
  • I’ve wrapped together a release called 0.9 that contains all of these
  • None of the services run as root

There is still a long list of things that I’d like to do with the project, for example I’ve been playing with dancer to provide a full rest api. The next thing I need to do however is update docs, find a stable place to host downloads, sort out the website, then work out what to do with the whole GUI thing for those that used it. All topics for another day.

*I really need to rename “proxy” in the openfpc context… If anyone has a better suggestion for a name I’m all ears.

You can download 0.9 here for now while I try and sort out the old http://www.openfpc.org website and turn it into something maintainable. Alternatively you could just clone it from github

ofpc-0.9-simplesearch

Here is a quick teaser of it in use, searching for sessions destined for TCP:22 that started within the last 10 minutes.

==

 

Written by leonward

September 15, 2014 at 11:00 am

Posted in Uncategorized

Lex Deux Alpes 2014

leave a comment »

Sometimes you just need to get away from it all. This is the video from our 2014 downhillin’ trip to Les Deux Alpes bike park. Loads of fun. You don’t see much of me in the video because the gopro is attached to my head.

Written by leonward

September 14, 2014 at 11:32 am

Posted in Cycling

Tagged with

OpenFPC on Security Onion

with 7 comments

I’ve been asked a couple of times if OFPC can be installed on Security Onion, and I’m happy to say yes it can (as of the time of writing anyway rev 335 in SVN). While poking about with it I spotted a small yet critical bug that I had to squish with 334. Here is what you have to do:

1) Download and install Security Onion

You can download it from here -> http://sourceforge.net/projects/security-onion/files/

2) Grab OpenFPC from SVN

$ svn checkout http://openfpc.googlecode.com/svn/trunk/ openfpc-read-only
A openfpc-read-only/tools
A openfpc-read-only/tools/ofpc-cxsearch.pl
A openfpc-read-only/tools/mk_release.sh
A openfpc-read-only/tools/testParse.pl
<snip>

2) Install some extra packages

Get and install cxtracker and some other stuff

$ wget http://github.com/downloads/gamelinux/cxtracker/cxtracker_0.9.5-1_i386.deb

$ sudo dpkg -i ./cxtracker_0.9.5-1_i386.deb

$ sudo apt-get install libarchive-zip-perl libfilesys-df-perl libdate-simple-perl libdatetime-perl

3) Run the installer

OpenFPC checks for some dependencies during install, these will fail on Onion even after installing the above. One reason is that it checks for Apache 2 however Apache 2.2 is installed on Onion. Use the –foceinstall option instead to continue.

$ sudo ./openfpc-install.sh forceinstall

4) Create the GUI and Session DBs

Security Onion doesn’t have a password set for the root mysql user, this doesn’t sit well with the OpenFPC install scripts as they expect there to be one. When a password is prompted for, simply hit the Enter key. This looks a little confusing, so here is an exact copy/paste below of what to expect. You could of course set a password for root, but I don’t know what else on the platform this may break (if anything).

$ sudo openfpc-dbmaint create gui /etc/openfpc/openfpc-default.conf
[*] Enter mysql “root” credentials to connect to your local mysql server in order to create the databases
DB root Username: root
DB root Password: <Enter> Enter password: <Enter> 

[*] Enter an initial username for the first OpenFPC GUI user.
GUI Username: admin
GUI Password: <a password>
Email address: admin@admin.com
Real Name: My real name
grep: /etc/openfpc/openfpc.passwd: No such file or directory
USER NOT FOUND. Adding admin.
Creating new user file /etc/openfpc/openfpc.passwd…
Adding user admin
Done.
CREATING GUI DATABASE
—————————
Enter password: <Enter> 
Enter password: <Enter> 
Enter password: <Enter> 
Enter password: <Enter> 
GUI DB Created.
Enter password: <Enter> 
Enter password: <Enter> 
New user admin added.
[*] Restarting OpenFPC

<SNIP>

5) Create the session DB, it will also look a little weird if a blank password is used for the mysql ‘root’ user.

$ sudo  openfpc-dbmaint create session /etc/openfpc/openfpc-default.conf
[*] Enter mysql “root” credentials to connect to your local mysql server in order to create the databases
DB root Username: root
DB root Password: <Enter> Enter password:
———————————————————
[*] Working on Instance /etc/openfpc/openfpc-default.conf .
Would you like session capture ENABLED on Default_Node? (y/n)y
[-] Enabling session capture in Default_Node config
Done.
[-] Found cxtracker.
CREATING DATABASE
—————————
Enter password: <Enter>
Enter password: <Enter>
Enter password: <Enter>
Enter password: <Enter>
Enter password: <Enter>
Session DB Created.
Adding function INET_ATON6… to DB openfpc
Enter password: <Enter>
[*] Restarting OpenFPC <SNIP>

6) Try it out.

The command line should be functional now, as should the GUI accessible at https://localhost/openfpc/

$ sudo  openfpc-client -a status

* openfpc-client 0.7 *
Part of the OpenFPC project

Username: admin
Password for user admin : <My Password>
####################################
OpenFPC Node name : Default_Node
OpenFPC Node Type : NODE
OpenFPC Version : 0.6
Oldest Packet : 1347467610 (Wed Sep 12 16:33:30 2012)
Oldest Session : 1347479998 (Wed Sep 12 19:59:58 2012)
Packet utilization : 25%
Session utilization : 25%
Session DB Size : 8 rows
Session lag : 0 files
Storage utilization : 25%
Packet space used : 4726172 (4.73 GB)
Session space used : 4726172 (4.73 GB)
Storage used : 4726172 (4.73 GB)
Load avg 1 : 1.54
Load avg 5 : 1.34
Load avg 15 : 1.16
Errors : 0

You should now be able to configure Snorby to extract data from OpenFPC when alerts fire on the Security Onion.

Written by leonward

September 19, 2012 at 3:40 pm

Posted in OpenFPC, Security

My Coed y Brenin Bike Crash (March 2012)

leave a comment »

I had a good crash this weekend while at a trail center in Wales. I’ll live to ride another day!

Here’s the video taken from the GoPRO mounted to my chest.

 

 

Written by leonward

March 20, 2012 at 2:09 pm

Posted in Fail

Tagged with ,

List of Mountain Bike films on iTunes

leave a comment »

I’m a big buyer of content via iTunes, however sometimes the search interface lets me down. I find it frustrating that there isn’t a way of listing content for an intrest group (such as MTB movies), and whenever I’m taking a long flight, I find a bike movie is perfect viewing.

To help anyone else trying to find a list of decent MTB films that are all available on iTunes, perhaps the below could help. I’ll try to keep it updated as I dig out more over time. Simply search for the film name on iTunes and it should turn up in the store. Note that some of these are marked as TV shows rather than films.

Follow Me – http://www.anthillfilms.com/followme/

Vast – http://www.ionatefilms.com/

Here we go again – http://dh-productions.com/HereWeGoAgain/index.html

Life Cycles – http://www.lifecyclesfilm.com/

 

 

 

Written by leonward

January 14, 2012 at 11:13 am

Posted in Uncategorized

Tagged with

Defining an Achievable Network Segmentation Process

leave a comment »

We all struggle balancing work and personal projects, but somehow I managed to combine both into one with the new Sourcefire blog (http://blog.sourcefire.com).

I’ve had a blog posted there rather than here for once, so if you’re interested in network segmentation go take a read.

The modern enterprise network has undertaken massive changes over recent years. The adoption of cloud computing, consumerization, mobilization, and the explosion of the “app” markets, has driven us all to use technology in new ways. We must embrace these new technologies and the business edge that they can offer, but all the while we need to recognize that just below all of this new technology there is something supporting it that hasn’t changed. The security infrastructure they depend on to deliver safe and controlled service.

Read more here -> http://blog.sourcefire.com/2011/09/defining-achievable-network.html

-Leon

 

Written by leonward

October 3, 2011 at 9:31 am

Posted in Uncategorized

Follow

Get every new post delivered to your Inbox.

Join 783 other followers