An alchemists view from the bar

Network Security Alchemy

Archive for the ‘snort’ Category

Installing and getting started with OpenFPC 0.9

with 2 comments

Installing OpenFPC – Version 0.9.

Hi, this is a simple walk through of installing OpenFPC on Ubuntu LTS 14.04, although the steps should be similar for any Debian based distribution. Getting things running should be pretty simple, but there are couple of gotchas along the way. For the impatient and those that only want the highlights, here is the process at a high level:

  •  Install the Ubuntu package dependencies that are in the Ubuntu package archives. Note that the install script will also check these for you.
  • Download & install cxtracker
  • Download OpenFPC
  • Untar and run the OpenFPC install script (openfpc-install.sh)
    At this point you could go and edit the example config files that were placed in /etc/openfpc, but instead I suggest you get things functioning in a default configuration before trying to over complicate things.
  • Create a user for OpenFPC and set the password them.
  • Create the session database
  • Start OpenFPC
  • Grab a coffee and wait for some packets to come in
  • Try out some basic searches and traffic extraction.

All of the below sections talk though the details of how to achieve the above.

For your first time using OpenFPC, I stringly suggest you start off with the default installation. You can get to the advanced functions like proxy nodes later. By default you will have a single device sniffing traffic and connections from eth0 with a name of Default_Node.

# Install package dependencies:
$ sudo apt-get install \
daemonlogger \
tcpdump \
tshark \
libdatetime-perl \
libprivileges-drop-perl \
libarchive-zip-perl \
libfilesys-df-perl \
mysql-server \
libdbi-perl \
libterm-readkey-perl \
libdate-simple-perl \
libdigest-sha-perl \
libjson-pp-perl \
libdatetime-perl \
libswitch-perl \
libdatetime-format-strptime-perl

# Download Cxtracker
Cxtracker is a connection capturing tool designed for general nsm functions. In the context of OpenFPC it finds connections on the network and stores them to disk in a CSV file. A second program (openfpc-cx2db) then parses these session files and uploads them to the OpenFPC session database. This session database allows you to search for network traffic very quickly and identify the sessions you would like to extract. In OpenFPC the connection data is not centrally stored, instead an OpenFPC proxy can aggregate a single search and make it take place across multiple nodes (the things capturing session and packat data), and then combine the results into one dataset for the user.

lward@dev-ny:~$ wget http://github.com/downloads/gamelinux/cxtracker/cxtracker_0.9.5-1_i386.deb
<snip>
2014-09-17 07:47:20 (153 MB/s) – ‘cxtracker_0.9.5-1_i386.deb’ saved [12116/12116]

lward@dev-ny:~$ sudo dpkg -i cxtracker_0.9.5-1_i386.deb

# Download OpenFPC.

This documentation was created for openfpc-0.9.5, and documentation has a bad habit of getting out of date quickly. The installation process shouldn’t change much between minor releases, so I suggest you go and install the latest release and hope that these docs are still relevant for it.

lward@dev-ny:~$ wget

# Extract and install OpenFPC
Before you run the installer, there are likely a couple of things you should note.
– Because openfpc-queued needs to use tcpdump to extract session data that is stored on disk, the Ubuntu apparmour profile that prevents it from *reading* files anywere outside of a users home directory isn’t viable. The installer will disable apparmour for tcpdump (and only tcpdump) by creating /etc/apparmor.d/disable/usr.sbin.tcpdump. If you don’t want this, make sure you re-enable it, or edit the installer to not do this. Note that you’ll have to make sure that all pcap operations take place in the openfpc user’s ~, and that’s less than ideal for a file organization point of view.

  • A node called “Default_Node” is created by default. To change its configuration you can edit /etc/openfpc/openfpc-default.conf
  • A user called openfpc is added to the system for all components to drop privileges to (you don’t want daemons running as root)
  • Pay attention for any errors that pop up

lward@dev-ny:~$ tar -zxvf openfpc-0.9.tgz
openfpc-0.9/
openfpc-0.9/etc/
openfpc-0.9/etc/openfpc-default.conf
openfpc-0.9/etc/openfpc-example-proxy.conf
openfpc-0.9/etc/init.d/
openfpc-0.9/etc/init.d/openfpc-daemonlogger
openfpc-0.9/etc/init.d/openfpc-cx2db
openfpc-0.9/etc/init.d/openfpc-queued
openfpc-0.9/etc/init.d/openfpc-cxtracker
openfpc-0.9/etc/routes.ofpc
openfpc-0.9/openfpc-install.sh
openfpc-0.9/OFPC/
openfpc-0.9/OFPC/CXDB.pm
openfpc-0.9/OFPC/Config.pm
openfpc-0.9/OFPC/Request.pm
openfpc-0.9/OFPC/Parse.pm
openfpc-0.9/OFPC/Common.pm
openfpc-0.9/docs/
openfpc-0.9/docs/README
openfpc-0.9/docs/TODO
openfpc-0.9/docs/INSTALL
openfpc-0.9/openfpc-dbmaint
openfpc-0.9/openfpc-cx2db
openfpc-0.9/openfpc-client
openfpc-0.9/openfpc-queued
openfpc-0.9/cgi-bin/
openfpc-0.9/openfpc
openfpc-0.9/openfpc-password
lward@dev-ny:~$

lward@dev-ny:~/openfpc-0.9$ sudo ./openfpc-install.sh install

*************************************************************************
* OpenFPC installer – Leon Ward (leon@openfpc.org) v0.9
* A set if scripts to help manage and find data in a large network traffic
* archive.

* http://www.openfpc.org

[*] Detected distribution as DEBIAN

<SNIP>
[*] Installation Complete

# Create a user for OpenFPC.
The location checked for the openfpc password file is defined in the instance configuration file. For us in our simple install that’s /etc/openfpc/openfpc-default.conf that was created when running openfpc-install.sh. In that file you’ll notice a line that defines where to look for a passwd file, our default config looks for /etc/openfpc/openfpc.passwd.

lward@dev-ny:~/openfpc-0.9$ sudo openfpc-password -a add -u admin \
-f /etc/openfpc/openfpc.passwd
Creating new user file /etc/openfpc/openfpc.passwd…
[*] Adding user admin
Enter new password:
Retype password:
Password Okay
[*] Done.

# Create the session database.
To make database creation simple, there is a tool for creating and dropping the correct database that matches the configuration you define in the openfpc config file (in our simple default that’s /etc/openfpc/openfpc-default.conf).
openfpc-dbmaint uses the data in that config file to create the database with the expected permissions. This tool requires you to have root access to use. There are multiple database types that can be created, in our simple default example you only will need a session DB. For more options you can see openfpc-dbmain –help.

lward@dev-ny:~/openfpc-0.9$ sudo openfpc-dbmaint create session /etc/openfpc/openfpc-default.conf
[*] Enter mysql “root” credentials to connect to your local mysql server in order to create the databases
DB root Username: root
DB root Password:

Enter password:
———————————————————
[*] Working on Instance /etc/openfpc/openfpc-default.conf .
Would you like session capture ENABLED on Default_Node? (y/n)y
[-] Enabling session capture in Default_Node config
Done.
[-] Found cxtracker.
[*] Creating Session database on Default_Node
– Session DB Created
– Adding function INET_ATON6 to DB ofpc_session_default
[*] Restarting OpenFPC Node Default_Node
Stopping Daemonlogger… Not running
Stopping OpenFPC Queue Daemon (Default_Node)… Not running
Stopping OpenFPC cxtracker (Default_Node)… Not running
Stopping OpenFPC Connection Uploader (Default_Node)… Not running
Starting Daemonlogger (Default_Node)… Done
Starting OpenFPC Queue Daemon (Default_Node)… Done
Starting OpenFPC cxtracker (Default_Node)… Done
Starting OpenFPC Connection Uploader (Default_Node) … Done
lward@dev-ny:~/openfpc-0.9$

After creating the database you’ll notice that openfpc is automatically restarted, as openfpc wasn’t running before we executed this command you’ll notice that it starts up. Hopefully you’ll have output like the above.
# Testing the install and getting started.
You can start, stop and check the status of openfpc using the openfpc command. Passing the -v (for verbose) will provide you with some information about the configuration of the system. In the below output you can see that there are two instances configured on my system, one is DISABLED (Example_Proxy), and another is active (Default_Node).
lward@Dev:~/openfpc$ sudo ./openfpc -a status -v
###############################################################################
[*] OpenFPC instance openfpc-example-proxy.conf
– NODENAME: Example_Proxy
– DESCRIPTION: “An example OpenFPC Proxy config. http://www.openfpc.org&#8221;
– STATUS : DISABLED
– PORT: 4243
– PASSWORD FILE /etc/openfpc/openfpc.passwd
###############################################################################
[*] OpenFPC instance openfpc-default.conf
– NODENAME: Default_Node
– DESCRIPTION: “An OpenFPC node. http://www.openfpc.org&#8221;
– STATUS : ENABLED
– PORT: 4242
– PASSWORD FILE /etc/openfpc/openfpc.passwd
– INTERFACE: eth0
– FULL PACKET CAPTURE: ENABLED
– PACKET STORE: /var/tmp/openfpc/pcap
– SESSION DATA SEARCH: ENABLED
– SESSION DATABASE NAME: openfpc
– SESSION LAG: 0
– SESSION INSERT FAIL: 0
– openfpc-daemonlogger is /usr/bin/daemonlogger
Daemonlogger (Default_Node) : Running
– openfpc-queued is /usr/bin/openfpc-queued
OpenFPC Queue Daemon (Default_Node): Running
– openfpc-cxtracker is /usr/bin/cxtracker
OpenFPC Connection Tracker (Default_Node) : Running
– openfpc-cx2db is /usr/bin/openfpc-cx2db
OpenFPC Connection Uploader (Default_Node) : Running

 
To actually interact with you OpenFPC Node (Default_Node), you can use the openfpc-client. The openfpc-client is a client application that talks with either an OpenFPC Node or OpenFPC Proxy over the network. This allows you to use a local tool on your workstation to search, extract, save and fetch pcaps from the remote device capturing data. By default openfpc-client tries to connect to the server localhost on TCP:4242. Check openfpc-client –help to find out how to specify a remote node (–server –port).

lward@dev-ny:~$ openfpc-client -a status

* openfpc-client 0.9 *
Part of the OpenFPC project - http://www.openfpc.org

Username: admin
Password for user admin :
=====================================
Status from: Default_Node
=====================================
* Node: Default_Node
- Node Type : NODE
- Description : "An OpenFPC node. http://www.openfpc.org"
- Packet storage utilization : 7 %
- Session storage utilization : 7 %
- Space available in save path : 7 %
- Space used in the save path : 2047640 (2.05 GB)
- Session storage used : 2047640 (2.05 GB)
- Packet storage used : 2047640 (2.05 GB)
- PCAP file space used : 156M
- Local time on node : 1410955045 (Wed Sep 17 07:57:25 2014 America/New_York)
- Newest session in storage : 1410954011 (Wed Sep 17 07:40:11 2014 America/New_York)
- Oldest session in storage : 1410441644 (Thu Sep 11 09:20:44 2014 America/New_York)
- Oldest packet in storage : 1410353440 (Wed Sep 10 08:50:40 2014 America/New_York)
- Storage Window : 5 Days, 22 Hours, 19 Minutes, 27 Seconds
- Load Average 1 : 0.00
- Load average 5 : 0.01
- Load average 15 : 0.05
- Number of session files lagging : 0
- Number of sessions in Database : 8
- Node Timezone : America/New_York
lward@dev-ny:~$

In the output above I can see some important status information about this device. Note the amount of data captured, disk usage, and session database size. The session database will auto-trim to only keep session data for the packets that are available for extraction. Make sure you have some data captured and lets go grab some full packet data.

Here I will simply ask to fetch (extract and send to my workstation) all traffic to a destination port of 53 in the last 10 minutes. For more advanced constraints check out openfpc-client –help.

lward@dev-ny:~$ openfpc-client -a fetch -dpt 53 –last 600

* openfpc-client 0.9 *
Part of the OpenFPC project – http://www.openfpc.org

Username: admin
Password for user admin :
#####################################
Date : Wed Sep 17 07:58:56 2014
Filename: /tmp/pcap-openfpc-1410955136.pcap
Size : 17K
MD5 : 938638229b7e508646e5dbbb3ba231b3

The above shows me the filename I’ve just created, by default the pcap file is written to /tmp, you can choose a better filename with the -w option. If we look at the contents of this file we will see the full packet contents.

lward@dev-ny:~$ tshark -r /tmp/pcap-openfpc-1410955136.pcap
1 0.000000000 192.168.42.12 -> 192.168.42.1 DNS 76 Standard query 0x17d0 A daisy.ubuntu.com
2 0.000004000 192.168.42.12 -> 192.168.42.1 DNS 76 Standard query 0x34c8 AAAA daisy.ubuntu.com
3 0.034045000 192.168.42.1 -> 192.168.42.12 DNS 108 Standard query response 0x17d0 A 91.189.92.57 A 91.189.92.55
<SNIP>

To save you performing large extractions to see if sessions that match your constraints exist you can use the –search option. The –search option asks openfpc to look though its session database to find out if the traffic you’re interested in exists. This is much faster than actually extracting the full pcap data itself.
lward@dev-ny:~$ openfpc-client -a search -dpt 53

* openfpc-client 0.9 *
Part of the OpenFPC project – http://www.openfpc.org

Username: admin
Password for user admin :
=====================================================================================================================================================
Custom Search
=====================================================================================================================================================
Start: Wed Sep 17 07:01:22 2014 (America/New_York)
End : Wed Sep 17 08:01:22 2014 (America/New_York)
Node : Default_Node
Rows : 4
SQL : SELECT start_time,INET_NTOA(src_ip),src_port,INET_NTOA(dst_ip),dst_port,ip_proto,src_bytes, dst_bytes,(src_bytes+dst_bytes) as total_bytes
FROM session IGNORE INDEX (p_key) WHERE unix_timestamp(CONVERT_TZ(`start_time`, ‘+00:00′, @@session.time_zone))
between 1410951682 and 1410955282 AND dst_port=’53’ ORDER BY start_time DESC LIMIT 20
=====================================================================================================================================================
Row Start Time Source IP sPort Destination dPort Proto Src Bytes Dst Bytes Total Bytes Node Name
0 2014-09-17 7:07:17 192.168.42.12 48755 192.168.42.1 53 udp 14828 18924 33752 Default_Node
1 2014-09-17 7:07:17 192.168.42.12 34676 192.168.42.1 53 udp 14828 31724 46552 Default_Node
2 2014-09-17 7:07:17 192.168.42.12 41495 192.168.42.1 53 udp 14828 20204 35032 Default_Node
3 2014-09-17 7:07:44 192.168.42.12 46496 192.168.42.1 53 udp 34264 53976 88240 Default_Node
=====================================================================================================================================================

One of the more useful features of OpenFPC is to actually request data in in the formats outputted by different tools. This enables you to simply ‘paste’ the log line from some tool into openfpc-client and it will go grab the session for you. Unsurprisingly OpenFPC supports the search format as one of these log formats. This means for any session that we find in the database with the search action, we can go and ask for is with a fetch (or store) action. E.g.

$ openfpc-client -a fetch –logline ” 1 2014-09-17 7:07:17 192.168.42.12 34676 192.168.42.1 53 udp 14828 31724 46552 Default_Node”

* openfpc-client 0.9 *
Part of the OpenFPC project – http://www.openfpc.org

Username: admin
Password for user admin :
#####################################
Date : Fri Oct 3 16:57:14 2014
Filename: /tmp/pcap-openfpc-1412351834.pcap
Size : 660
MD5 : 39fdb557d751b2cebe31b2d5b9aa5d3c

Hopefully this is enough information to get you started!

-Leon

Written by leonward

October 3, 2014 at 7:00 pm

Posted in Security, snort

Tagged with

Big OpenFPC release – 0.6

with 2 comments

Pushing forwards closer to a 1.0 release for OpenFPC, one of the major components has now been updated – The GUI.

To introduce this new release I’ve put together a short screen-cast of OpenFPC to show the installation, setup procedure, and a bit of general usage. So if you’re tasked with rolling together your own full packet capture/network traffic recorder/forensics system, perhaps you may want to take a look below.

 

For those who don’t want to sit through five minutes of video to see what the new GUI looks like, here are a few screenshots of the system in action.

Version 0.6 is now available at  http://code.google.com/p/openfpc/downloads/list . Expect a few bugs, and if you report them, Ill own the task of fixing them.

-Leon

Written by leonward

June 13, 2011 at 12:37 pm

Posted in OpenFPC, Security, snort, Uncategorized

Tagged with , , ,

Insta-Snorby 0.4 with OpenFPC

with 3 comments

Snorby had a big launch this weekend with an event that rivaled Apple in terms of hype and excitement! The two-dot-ooh-yeah release has reached the unwashed masses.

The Snorby 2.0 feature that I’m most excited about is the inclusion of support for OpenFPC directly in the Snorby UI (but face it I’m kind of biased here). Many users of Snorby will be unaware of the OpenFPC project, and as they could be eager to try out the bleeding Snorby version, I thought I would include a quick how-to (below) of adding OpenFPC on to the Insta-Snorby appliance.

OpenFPS and Snorby together

I wouldn’t expect real-world users of Snorby / OpenFPC to use the Insta-Snorby VM, but it’s a good introduction / test platform. As a guide to effort, the below ten steps should take about ten minutes to follow (including the download and updating of packages).

If you spot any errors please let me know, this is the bleeding edge after all.

First the obvious bits….

1) Download the Insta-Snorby-0.4.iso

2) Install the .iso on to the hard disk of a virtual (or physical) machine

3) SSH in to the device as root.

Now the less-obvious bits…

4) Prepare the platform.

Update the package archives. This is mandatory, it’s not being performed as part of good practice.

root@Insta-Snorby ~# apt-get update

Install the dependancies from the Ubuntu package archive (note you can copy/paste the below into your ssh session rather than re-type).

apt-get install apache2 daemonlogger tcpdump tshark libarchive-zip-perl \
libfilesys-df-perl libapache2-mod-php5 mysql-server php5-mysql \
libdatetime-perl libdbi-perl libdate-simple-perl php5-mysql \
libterm-readkey-perl libdate-simple-perl

5) Download the latest version of OpenFPC from http://code.google.com/p/openfpc/downloads/list

root@Insta-Snorby ~# wget http://openfpc.googlecode.com/files/openfpc-0.4-266.tgz

Note that 0.4-266 is “current” at the time of writing, but there is a lot of development happening right, so make sure you get the latest and don’t assume 0.4-266 is still “current”

6) Install OpenFPC

root@Insta-Snorby ~# tar -zxf openfpc-0.4-266.tgz 
root@Insta-Snorby ~# cd openfpc-0.4-266/
root@Insta-Snorby ~# ./openfpc-install.sh install

You will be promoted to provide a password for the OpenFPC extract.cgi script. This password protects any attempts to pull out a pcap via the cgi interface used by Snorby via Apache’s basic auth. It saves the password to /etc/openfpc/apache2.passwd.
You will need this username/pass to access any pcaps via Snorby, so REMEMBER IT!

7) Customize OpenFPC

OpenFPC is a client/server system, the openfpc-client does not need to be on the same physical host as the openfpc-queue daemon and therefore it listens on a network socket (default 4242). The default username and password is

Username: openfpc
Password: openfpc

If you want to change these, edit /etc/openfpc/openfpc-default.conf and set…

a) USER=openfpc=openfpc

Set this to whatever username/pass you desire e.g.
USER=snorby=letmein

b) Change the user account that is used to pull PCAP files via the extract.cgi interface to one you have specified with a USER definition. e.g. for the above user definition I would use:

GUIUSER=snorby
GUIUSER=letmein

8) Start up OpenFPC

root@Insta-Snorby ~/openfpc-0.4-266# openfpc –action start

###############################################################################
[*] OpenFPC instance openfpc-example-proxy.conf
 -  NODENAME:              Example_Proxy
 -  DESCRIPTION:           "An example OpenFPC Proxy config. www.openfpc.org"
 -  STATUS :               DISABLED 
 -  PORT:                  4243
###############################################################################
[*] OpenFPC instance openfpc-default.conf 
 -  NODENAME:              Default_Node 
 -  DESCRIPTION:           "An OpenFPC node. www.openfpc.org" 
 -  STATUS :               ENABLED
 -  PORT:                  4242
 -  INTERFACE:             eth0
 -  FULL PACKET CAPTURE:   ENABLED
 -  PACKET STORE:          /var/tmp/openfpc/pcap
 -  SESSION DATA SEARCH:   DISABLED
Starting Daemonlogger (Default_Node)...                                    Done
Starting OpenFPC Queue Daemon (Default_Node)...                            Done

9) Check communications and your openfpc username/password.

Use the command line tool openfpc-client to check things are working. The –action status will provide a status check of a remote OpenFPC instance.

root@Insta-Snorby ~/openfpc-0.4-266# openfpc-client -a status
   
 * openfpc-client 0.4 *
Part of the OpenFPC project
Username: openfpc
Password for user openfpc : 
#################################### 
 OpenFPC Node name   :  Default_Node 
 OpenFPC Node Type   :  NODE
 OpenFPC Version     :  0.4
 Oldest Packet       :  1291638906 (Mon Dec  6 12:35:06 2010)
 Oldest Session      :  0 (Thu Jan  1 00:00:00 1970)
 Packet utilization  :  10% 
 Session utilization :  Disabled% 
 Session DB Size     :  Disabled rows 
 Session lag         :  0 files 
 Storage utilization :  10% 
 Packet space used   :  1867896 (1.87 GB)
 Session space used  :  Disabled (Disabled Bytes)
 Storage used        :  1867896 (1.87 GB)
 Load avg 1          :  0.04 
 Load avg 5          :  0.05 
 Load avg 15         :  0.08 
 Errors              :  0 
root@Insta-Snorby ~/openfpc-0.4-266#

10) Configure the Snorby OpenFPC plugin

Navigate to the Snorby web interface, and browse to Administration.

Enable your OpenFPC integration here

  • Check the box “Enable OpenFPC support”
  • Use the below URL for extraction
  • Hit “Save Settings”

Complete!

Now when you look at an IPS event, you will have a “Packet Capture” button that pulls out the complete session data via OpenFPC.

Many of the advanced OpenFPC capabilities are not addressed in this how-to such as connection/flow capture and searching, compressed extracts, reports, distributed extracts, horizontal scaling, etc etc but I’m keeping this How-to simple. If you want to know more, you know where to look http://www.openfpc.org.

-Leon

Written by leonward

December 6, 2010 at 1:24 pm

Pushing the OpenFPC project forward

leave a comment »

A couple of people have been working harder than normal over the last couple of weeks. Edward, and I are happy to push out another OpenFPC test release to the world.

Here is short list of highlights and changes, however there is one point to pay close attention to.

A very kind web developer has started to help the team work on a central user interface for searching and extraction. Ill introduce him and his work in another future post, however in the short term thanks should be sent over to Eduardo!

0.3 Change highlights

  • Multiple configs can co-exist on a single box
  • Sourcefire IPS event parsing fixed
  • Snort-Fast event type no longer required port numbers. Makes multi-session extracts more simple (http attacks for example)
  • Search via bpf (–bpf command line option to openfpc-client)
  • Passwords no longer echo to screen
  • New init scripts to work with the new openfpc command
  • LSB compliant init scripts
  • Better log output (wlog) and verbose message handeling
  • Added better example configs (openfpc-default.conf and openfpc-example-proxy.conf)
  • Enabling session data is now far more simple
  • Included web-ui, now enabled by default
  • Space now renders in GB rather than Bytes
  • Fixed performance hit on cx2db inserting half open sessions.
  • Improved help text
  • The out-of-the-box proxy and node configurations now work with each other
  • CGI interface for full packet integration with other tools

As always, feedback and bugs are welcomed.

 

Written by leonward

November 22, 2010 at 9:09 pm

OpenFPC 0.1a – Installation and usage

with 2 comments

Firstly, I need to vent my anger at WordPress’s post formatting breakage. It is impossible for me to format this post using the Visual editor, and the HTML generated is so unbelievably ugly I can’t follow it.  So when the formatting of the text below doesn’t look correct blame wordpress not me.

I have uploaded a tarball of OpenFPC-0.1a to Googlecode, and as there isn’t any real documentation yet for OpenFPC I thought I would provide a few tips for people who want to try it out.

Firstly a word of warning: OpenFPC could be changing in the near future. I’m meeting up with ebf0 the author of FPCGui in June for a beer. Because duplicating effort is never a smart thing to do, and as both projects have similar goals it may make sense to pool some resources. Anyway, for the people who emailed me after my last post wanting to know how to get started with OpenFPC here are some tips.

The below is being performed on Ubuntu 10.04 LTS.

1) Check for requirements

OpenFPC depends on a few libraries and tools that need to be installed on the server you decide to dedicate for traffic collection. These include:

  • TCPdump
  • tshark (Part of the wireshark project)
  • mergecap (Part of the wireshark project)
  • daemonlogger (Version 1.2.1 or greater)
  • Perl
  • Perl Getopt::Long

On Ubuntu 10.04 LTS, all of these applications and librariers can be installed via apt without any real challenges. If they are not available in your Operating System’s package manager, you will have to go download and install them yourself.

lward@UbuntuDesktop:~$ sudo apt-get install tcpdump tshark daemonlogger

2) Install OpenFPC

Once those packages are installed, go download OpenFPC from here. This document was written for version 0.1a, but if there is a later release on the download list it’s best to use that. Once you have the tarball, extract it to somewhere in your home directory.

lward@UbuntuDesktop:~$ tar -zxvf openfpc-0.1a.tgz
openfpc-0.1a/
openfpc-0.1a/ofpcParse.pm
openfpc-0.1a/ofpc-extract.pl
openfpc-0.1a/install-openfpc.sh
openfpc-0.1a/README
openfpc-0.1a/openfpc
openfpc-0.1a/openfpc.conf
lward@UbuntuDesktop:~$

If your dependencies are all satisfied, installation should be as simple as running the installer. And that should be it!

OpenFPC installs itself into /opt/openfpc, and looks for a configuration file in /openfpc.conf /etc/openfpc/openfpc.conf, /opt/openfpc/openfpc.conf (first config file found wins).
I suggest you start off by editing the file /opt/openfpc/openfpc.conf. Pay close attention to the FILE_SIZE and DISK_SPACE values. You will want to increase the FILE_SIZE value to over 10M for a production environment, it’s that default because of my testing on a VM. 1G or 2G would probably make more sense.
DISK_SPACE equates to a percentage of disk space to use on the capture partition. All other values should be pretty much obvious.
Once you’re happy with the values you have set, start up openfpc.

TIP: Because openfpc looks for a .conf file in your current working directory, be careful where you start/stop/extract pcap files from. I think I may change this behaviour in the future, but this is how it is for now.

lward@UbuntuDesktop:~$ /etc/init.d/openfpc status
[*] Reading configuration file /opt/openfpc/openfpc.conf
[!] No current buffers found in /var/tmp/openfpc – Have you started it yet?

lward@UbuntuDesktop:~$ sudo /etc/init.d/openfpc start
[sudo] password for lward:
[*] Reading configuration file /opt/openfpc/openfpc.conf
[-] Daemon mode set
[-] Interface set to eth0
[-] Logpath set to /var/tmp/openfpc
[-] Log filename set to “openfpc-pcap”
[-] Pidfile configured to “openfpc-dl”
[-] Pidpath configured to “/var/run”
[-] Rollover configured for 10 megabytes
[-] Rollover configured for 0 none
[-] Pruning behavior set to oldest IN DIRECTORY

DaemonLogger Version 1.2.1
By Martin Roesch
(C) Copyright 2006-2007 Sourcefire Inc., All rights reserved
Checking partition stats for log directory “/var/tmp/openfpc/.”
50% max disk utilization = 2467159 blocks free (out of 4934317)
 Blocksize = 4096
Rollsize = 2560 blocks
[-] It looks like daemonlogger has started successfully
[*] Traffic buffer (Daemonlogger) started on Tue May 25 17:16:28 BST 2010

Currently pcaps need to be requested locally using the ofpc-extract.pl tool, and only Snort-syslog, Snort alert_fast, Sourcefire3D, and Exim4 log entries are supported. If you would like support for another log format added that you use right now, send me samples via email (<myfirstname>@rm-rf.co.uk). Please only ask for log file formats that you are in a position to use and test, don’t just suggest nice-to-have.

So once you have a log entry you want to extract, run the ofpc-extract.pl command. I have one of my Snort instances writing events to syslog. I don’t make use of a nice Snort event UI like Snorby, Sguil, or Base on this device but when an event is triggered I want some more packet data.Here is an alert Ill use as an example (IP address’ are censored of course):

May 26 08:36:45 XXXXX snort: [1:2020:5] RPC mountd TCP unmount request [Classification: Attempted Information Leak] [Priority: 2]: {TCP} X.X.X.X:991 -> Y.Y.Y.Y:41208

To extract the session based on this log message, I will pass the -a “<LOG LINE>” arguments to ofpc-extract.pl

leon@rancid:~$ ofpc-extract.pl -a “May 26 08:36:45 rancid snort: [1:2020:5] RPC mountd TCP unmount request [Classification: Attempted Information Leak] [Priority: 2]: {TCP} 80.68.89.43:991 ->80.68.87.205:41208”
* ofpc-extract.pl  – Part of the OpenFPC Project *
Leon Ward – leon@rm-rf.co.uk
————————————————–
– Searching for traffic..
– Merging …
– Created /tmp/extracted-1274864016.pcap (1.1K Bytes)

I now have the complete session in the file /tmp/extracted-1274864016.pcap.
If you want to create your own wrapper scripts around ofpc-extract, you could use the -q (quiet) flag e.g.

leon@rancid:~$ FILENAME=$(ofpc-extract.pl -a “May 26 08:36:45 rancid snort: [1:2020:5] RPC mountd TCP unmount request [Classification: Attempted Information Leak] [Priority: 2]: {TCP} 80.68.89.43:991 -> 80.68.87.205:41208” -q)
* ofpc-extract.pl  – Part of the OpenFPC Project *
Leon Ward – leon@rm-rf.co.uk
————————————————–
leon@rancid:~$ echo $FILENAME
/tmp/extracted-1274864443.pcap

Good luck with your sniffing!

Written by leonward

May 26, 2010 at 10:15 am

Introducing OpenFPC – An Open Full Packet Capture Framework for Network Traffic Recording

with 6 comments

I’m writing this post in order to introduce a project I’ve been thinking about for years and started working on a couple of months back. OpenFPC.

OpenFPC

Update: Aug/2010: OpenFPC now has a new home: www.openfpc.org. Go there for more info.

OpenFPC is designed as an add-on capability for other network communication and security technologies, it provides those technologies (open source or comercial) with something that their users often demand but the system doesn’t or can’t deliver on. The complete network traffic associated with a network security event. OpenFPC provides a central interface where full traffic session data can be automatically (or manually) requested from a distributed recording infrastructure, and then deliveries it back to the requester in a .pcap file or as a href to the .pcap file.

So what makes,or will make OpenFPC different from other tools available that provide similar functionality?

  • Distributed framework – With a central user interface and request point.
  • Quick and simple to set-up and start functioning. No expert knowledge required
  • Automated extraction of sessions (Syn-to-fin) of known to be “interesting” events for longer-term storage
  • Traffic is requested in the context if the requesting device, system or process
  • No database requirement for traffic/session data indexes
  • Optimised session searching over large traffic archives
  • Web UI, command line, and automated-process interfaces all provided

The above points are in no particular order, but to introduce the project I would like to focus on one feature in particular. In further posts and as the code develops I plan to expand on many of the above with functioning examples.

“Traffic is requested in the context if the requesting device, system or process”

A wise man once said that a picture is worth a thousand words, maybe the below examples will provide the same result and save me a load of typing.

Example 1) Mail log context

Problem: My mail log shows an entry that I have further interest in, this transmission is suspected to contain some malware. I would like to inspect the complete network traffic session to get hold of the malicious stuff if it actually exists.

leon@rancid:~$ tail -n 1 /var/log/exim4/mainlog

2010-05-16 11:35:27 1ODbBv-0008UQ-20 <= xxxxxx@sourcefire.com H=nFAsys00xxxxg108.obsStp.com [74.125.149.199] P=smtp S=1019 id=AALkTiXXXYH8LJFAD3eTy4DHlz5_JRy6A-ADSF-iCXYP@mail.gmail.com

Solution: Extract the session based on the Exim4 (my mail daemon) log line. Below is the log entry that I’m interested in.

leon@rancid:~$ ofpc-extract.pl -a 22010-05-16 11:35:27 1ODbBv-0008UQ-20 <= xxxxxx@sourcefire.com H=nFAsys00xxxxg108.obsStp.com [74.125.149.199] P=smtp S=1019 id=AALkTiXXXYH8LJFAD3eTy4DHlz5_JRy6A-ADSF-iCXYP@mail.gmail.com
* ofpc-extract.pl  – Part of the OpenFPC Project *
Leon Ward – leon@rm-rf.co.uk
————————————————–
– Searching for traffic…..
– Merging …
– Created /tmp/extracted-1274006391.pcap (3.4K Bytes)
Now I have the complete session to inspect and perform analysis on.
Example 2) IPS Context
Problem: Looking through my central log manager I spot the following IPS event that was passed to my log manager over Syslog. Because the event was uploaded to my screen via syslog i’ve lost all packet data.
May 16 11:49:05 rancid snort: [1:1951:7] RPC mountd TCP mount request [Classification: Attempted Information Leak] [Priority: 2]: {TCP} 28.88.89.43:687 -> 88.88.17.305:41208
I would like to extract the complete session to perform further investigation into the event.
leon@rancid:~$ ofpc-extract.pl -a “May 16 11:49:05 rancid snort: [1:1951:7] RPC mountd TCP mount request [Classification: Attempted Information Leak] [Priority: 2]: {TCP} 28.88.89.43:687 -> 88.88.17.305:41208
* ofpc-extract.pl  – Part of the OpenFPC Project *
Leon Ward – leon@rm-rf.co.uk
————————————————–
– Searching for traffic.
– Merging …
– Created /tmp/extracted-1274006982.pcap (1.2K Bytes)
Example 3) Arbitrary extraction
Of course you don’t need to format a request like  a log file message for a session, simply search. Here are the command line options.
leon@rancid:~$ ofpc-extract.pl --help
* ofpc-extract.pl  - Part of the OpenFPC Project *
  Leon Ward - leon@rm-rf.co.uk
--------------------------------------------------
- Usage:
  --mode     or -m <at|window> At a specific time, or search in a window
  --src-addr or -s           Source IP
  --dst-addr or -d           Destination IP
  --src-port or -u           Source Port
  --dst-port or -r           Destination Port
  --write    or -w           Output file
  --http     or -l                      Output in HTML for download
  --verbose  or -v                      Verbose output
  --debug                               Debug output
  --quiet                               Return only a filename or an error
  --all                                 Check in all buffers not just current sniff buffer
  ***** Operation Mode Specific Stuff *****
  "At" mode.
  --each-way or -e         Number of pcaps each-way Default: 1
  --event    or -a         Parse a supported event log line e.g. Snort, Sourcefire, Exim etc
  --timestamp or -t                     Event mode - Look each way of this epoch value
  --sf                                  Timestamp in SF format, convert it
 "Window" mode.
  --start    or -b                      Start timestamp for searching in absolute mode
  --end      or -j                      End timestamp for searching is absolute mode
Conclusion
If you are still reading this post, I assume that this project is something that may interest you and can guess you have a couple of questions:
  • What log formats are currently supported?
    Right now not many, this is a preview release after all. Snort Fast alert, Snort Syslog, Exim4, and Sourcefire 3D Defense Center, Sourcefire 3D Sensor.
  • When can I expect the central manager / WebUI
    Don’t hold your breath right now 🙂 But seriously, I’m working on it and had some ghetto code working that I’m too shamed of to share.
  • What’s next? When can I expect something stable?
    This is the most important question. I’m aiming for a stable non-distributed release without the manager in the next couple of weeks. I think it’s best to get a small but stable feature-set into the community rather than one big buggy release in a couple of months time.
  • Can I help?
    What a wonderful question!? Yes you can! Right now I need to find problems and iron out some bugs. In a couple of weeks Ill be looking for someone who can write a simple web UI for me, my un-themed s don’t look great
  • How do I install, configure and get this preview release running?
    Keep your eyes open for a blog posting in a couple of days talking you through an install on Ubuntu
  • What are the requirements to run? Do you use sancp or anything like that?
    daemonlogger (Thanks Marty!), tcpdump, perl
  • When’s the automated extraction bit coming?
    I’m not sure if this will be before or after the webUI. I have a day job after all!
  • Will you support <foo> log file format?
    Maybe, I want to have built-in support for most common log formats, tell me what you need and send samples.
  • Where can I get the code from?
    Stay tuned for an alpha release this week on http://code.google.com/p/openfpc/

Happy Sniffing!

Written by leonward

May 17, 2010 at 8:00 am

ET RBN Blacklists with Snort and DumbPig

leave a comment »

I spent a few minutes updating DumbPig to work with Marty’s latest blacklist patch, with some great results. It looks like Marty has done a great job in keeping packet performance high while providing a rich blacklist configuration. DumbPig for the as yet unenlightened, processes Snort rulesets and offers advice when a “dumb” rule is detected. Blacklist snort rules are a good example if dumbness so I thought I would focus in a bit on why and how to use these tools.

Before digging into the tech, let me tease you with some performance numbers. My test below was basic, but should provide some relation to the real world.

Test 1) Stock snort.conf and VRT subscription release (processing approx 1.2GB of pcap files)

$ snort –pcap-dir /var/local/pcaps/defcon-2004/ -A fast -l /tmp -c /etc/snort/snort.conf

Run time for packet processing was 46.994580 seconds
Snort processed 2752654 packets.

Test 2) Stock snort.conf, with the “emerging-rbn.rules” (20/july/2009) added

$ snort –pcap-dir /var/local/pcaps/defcon-2004/ -A fast -l /tmp -c /etc/snort/snort.conf

Run time for packet processing was 89.749301 seconds
Snort processed 2752654 packets.

Test 3) Stock snort.conf, but using a DumbPig created blacklist file from the same “emerging-rbn.rules”

$ snort –pcap-dir /var/local/pcaps/defcon-2004/ -A fast -l /tmp -c /etc/snort/snort.conf

Run time for packet processing was 48.348535 seconds
Snort processed 2752654 packets.

So that’s a whopper of a performance increase while maintaining the same IP based detection ability. I don’t claim you will see the same performance in the wild, but an increase like this should get your attention. Feel free to re-create the tests on your own network and let me know the results (for my interst only).

Configuring Snort 2.6.4.1 with the Blacklist patch (v2)

I have seen a few posts on the internet where people have run into issues configuring and using the blacklist patch. Below are the steps I took to build a system in prep for this weeks Snort Users webinar on DumbPig and other tools so I thought I would share the configuration here.

Starting platform: Debain Lenny – base installation + ssh and sudo (with lward in the sudoers)

1) Install the debs we know will be needed for the compile of Snort

lward@webexprep:~$ sudo apt-get install libpcap0.8 libpcap0.8-dev libdumbnet-dev build-essential libpcre3-dev automake autoconf libtool

2) Download and extract the snort 2.8.4.1 source

lward@webexprep:~$ wget http://dl.snort.org/snort-current/snort-2.8.4.1.tar.gz
lward@webexprep:~$ tar -zxvf ./snort-2.8.4.1.tar.gz

3) Download and extract the Blacklist patch

lward@webexprep:~$ cd snort-2.8.4.1/
lward@webexprep:~/snort-2.8.4.1$ wget http://www.snort.org/users/roesch/code/iplist.patch.v2.tgz
lward@webexprep:~/snort-2.8.4.1$ tar -zxvf ./iplist.patch.v2.tgz

4) Read the README.iplist

5) Pactch your Snort source tree

lward@webexprep:~/snort-2.8.4.1$ patch -p1 < iplist.patch

6) Rerun aclocal / automake/ autoconf

lward@webexprep:~/snort-2.8.4.1$ aclocal -I m4
lward@webexprep:~/snort-2.8.4.1$ automake
lward@webexprep:~/snort-2.8.4.1$ autoconf

7) Configure Snort (and enable IP listing).

lward@webexprep:~/snort-2.8.4.1$ ./configure –enable-iplist

(note there should be no errors , if you have m4 prelude messages, see the comment at the bottem of this post).

8) Compile  / install

Debain / Ubuntu users: You will have to “fix” the dnet/dumbnet fsckup created by the decnet libraries poluting the debian package namespace. A simple symlink will suffice.

lward@webexprep:~/snort-2.8.4.1$ sudo ln -s /usr/include/dumbnet.h /usr/include/dnet.h
lward@webexprep:~/snort-2.8.4.1$ make
lward@webexprep:~/snort-2.8.4.1$ sudo make install

9) Set up your snort configuration.

lward@webexprep:~/snort-2.8.4.1$ sudo mkdir /etc/snort
lward@webexprep:~/snort-2.8.4.1$ sudo chown lward /etc/snort/
lward@webexprep:~/snort-2.8.4.1$ cp etc/* /etc/snort/
lward@webexprep:~/snort-2.8.4.1$ vi /etc/snort/snort.conf

(Change your RULE_PATH to /etc/snort/rules)

10) Grab the VRT snort ruleset from snort.org, and stuck it in your home directory. This exact process will depend on your subscription level.

11) Set up the VRT snort rules

lward@webexprep:~/snort-2.8.4.1$ cd
lward@webexprep:~$ pwd
/home/lward
lward@webexprep:~$ ls
snort-2.8.4.1  snort-2.8.4.1.tar.gz  snortrules-snapshot-2.8.tar.gz
lward@webexprep:~$ tar -zxf ./snortrules-snapshot-2.8.tar.gz
lward@webexprep:~$ cp -r rules/ /etc/snort/

12) Grab yourself a pcap to test and play with

lward@webexprep:~$ wget http://rm-rf.co.uk/downloads/Honeynet-RFP-iis.tgz
lward@webexprep:~$ tar -zxvf ./Honeynet-RFP-iis.tgz

13) Test snort

lward@webexprep:~$ snort -c /etc/snort/snort.conf -A fast -l /tmp -r ~/Honeynet-RFP-iis.pcap

14) Enable, and test the blacklist functions

lward@webexprep:~$ echo preprocessor iplist: blacklist TestBlacklist /etc/snort/rules/test.blacklist >> /etc/snort/snort.conf
lward@webexprep:~$ echo 172.16.0.0/16 > /etc/snort/rules/test.blacklist
lward@webexprep:~/snort-2.8.4.1$ snort -c /etc/snort/snort.conf -A fast -l /tmp -r ~/Honeynet-RFP-iis.pcap

<Check that your are getting blacklist events in your /tmp/alert file. Make sure you add a CIDR that exists in your pcap test data!>

15) Install dumbpig

Install required perl modules from CPAN

lward@webexprep:~$ sudo cpan -e “Parse::Snort”
lward@webexprep:~$ sudo cpan -e “LWP::Simple”
lward@webexprep:~$ wget http://rm-rf.co.uk/downloads/dumbpig
lward@webexprep:~$ chmod +x ./dumbpig

16) Grab the latest Emerging threats RBN list

lward@webexprep:~$ wget http://www.emergingthreats.net/rules/emerging-rbn.rules

17) Convert the rule file into a blacklist

lward@webexprep:~$ ./dumbpig -q -r emerging-rbn.rules -b /etc/snort/rules/rbn.blacklist

DumbPig will detect rules that will work best in a blacklist, and add them to the file “rbn.blacklist”. For more usage inforamtion, take a look at the dumbpig page.

lward@webexprep:~$ head -n 5 /etc/snort/rules/rbn.blacklist
# Autogenerated blacklist by DumbPig from emerging-rbn.rules
# Contact leon.ward@sourcefire.com
# For more information about dumbPig visit http://rm-rf.co.uk
114.80.67.30/32 114.80.67.32/32 <snip>    # From Sid 2406000 : “ET RBN Known Russian Business Network IP TCP (1)” : emerging-rbn.rules
114.80.67.30/32 114.80.67.32/32 <snip>    # From Sid 2406001 : “ET RBN Known Russian Business Network IP UDP (1)” : emerging-rbn.rules

18) Reconfigure your snort.conf to the the rbn.blacklist

Something like this should work for you” “preprocessor iplist: blacklist RBN_Hosts /etc/snort/rules/rbn.blacklist”

And that’s it, the rest is up to you to use/abuse as you need.

Troubleshooting

Troubleshooting build problems. If you are having fun with one of the below, start with a clean source tree, re-patch, and follow step 6 (note the aclocal command).

undefined reference to `SetupIpList'
configure.in:1050: warning: macro `AM_PATH_LIBPRELUDE' not found in library

Happy Snorting - Leon

Written by leonward

July 20, 2009 at 12:06 pm

Posted in Security, snort

Tagged with ,