An alchemists view from the bar

Network Security Alchemy

Introducing OpenFPC – An Open Full Packet Capture Framework for Network Traffic Recording

with 6 comments

I’m writing this post in order to introduce a project I’ve been thinking about for years and started working on a couple of months back. OpenFPC.

OpenFPC

Update: Aug/2010: OpenFPC now has a new home: www.openfpc.org. Go there for more info.

OpenFPC is designed as an add-on capability for other network communication and security technologies, it provides those technologies (open source or comercial) with something that their users often demand but the system doesn’t or can’t deliver on. The complete network traffic associated with a network security event. OpenFPC provides a central interface where full traffic session data can be automatically (or manually) requested from a distributed recording infrastructure, and then deliveries it back to the requester in a .pcap file or as a href to the .pcap file.

So what makes,or will make OpenFPC different from other tools available that provide similar functionality?

  • Distributed framework – With a central user interface and request point.
  • Quick and simple to set-up and start functioning. No expert knowledge required
  • Automated extraction of sessions (Syn-to-fin) of known to be “interesting” events for longer-term storage
  • Traffic is requested in the context if the requesting device, system or process
  • No database requirement for traffic/session data indexes
  • Optimised session searching over large traffic archives
  • Web UI, command line, and automated-process interfaces all provided

The above points are in no particular order, but to introduce the project I would like to focus on one feature in particular. In further posts and as the code develops I plan to expand on many of the above with functioning examples.

“Traffic is requested in the context if the requesting device, system or process”

A wise man once said that a picture is worth a thousand words, maybe the below examples will provide the same result and save me a load of typing.

Example 1) Mail log context

Problem: My mail log shows an entry that I have further interest in, this transmission is suspected to contain some malware. I would like to inspect the complete network traffic session to get hold of the malicious stuff if it actually exists.

leon@rancid:~$ tail -n 1 /var/log/exim4/mainlog

2010-05-16 11:35:27 1ODbBv-0008UQ-20 <= xxxxxx@sourcefire.com H=nFAsys00xxxxg108.obsStp.com [74.125.149.199] P=smtp S=1019 id=AALkTiXXXYH8LJFAD3eTy4DHlz5_JRy6A-ADSF-iCXYP@mail.gmail.com

Solution: Extract the session based on the Exim4 (my mail daemon) log line. Below is the log entry that I’m interested in.

leon@rancid:~$ ofpc-extract.pl -a 22010-05-16 11:35:27 1ODbBv-0008UQ-20 <= xxxxxx@sourcefire.com H=nFAsys00xxxxg108.obsStp.com [74.125.149.199] P=smtp S=1019 id=AALkTiXXXYH8LJFAD3eTy4DHlz5_JRy6A-ADSF-iCXYP@mail.gmail.com
* ofpc-extract.pl  – Part of the OpenFPC Project *
Leon Ward – leon@rm-rf.co.uk
————————————————–
– Searching for traffic…..
– Merging …
– Created /tmp/extracted-1274006391.pcap (3.4K Bytes)
Now I have the complete session to inspect and perform analysis on.
Example 2) IPS Context
Problem: Looking through my central log manager I spot the following IPS event that was passed to my log manager over Syslog. Because the event was uploaded to my screen via syslog i’ve lost all packet data.
May 16 11:49:05 rancid snort: [1:1951:7] RPC mountd TCP mount request [Classification: Attempted Information Leak] [Priority: 2]: {TCP} 28.88.89.43:687 -> 88.88.17.305:41208
I would like to extract the complete session to perform further investigation into the event.
leon@rancid:~$ ofpc-extract.pl -a “May 16 11:49:05 rancid snort: [1:1951:7] RPC mountd TCP mount request [Classification: Attempted Information Leak] [Priority: 2]: {TCP} 28.88.89.43:687 -> 88.88.17.305:41208
* ofpc-extract.pl  – Part of the OpenFPC Project *
Leon Ward – leon@rm-rf.co.uk
————————————————–
– Searching for traffic.
– Merging …
– Created /tmp/extracted-1274006982.pcap (1.2K Bytes)
Example 3) Arbitrary extraction
Of course you don’t need to format a request like  a log file message for a session, simply search. Here are the command line options.
leon@rancid:~$ ofpc-extract.pl --help
* ofpc-extract.pl  - Part of the OpenFPC Project *
  Leon Ward - leon@rm-rf.co.uk
--------------------------------------------------
- Usage:
  --mode     or -m <at|window> At a specific time, or search in a window
  --src-addr or -s           Source IP
  --dst-addr or -d           Destination IP
  --src-port or -u           Source Port
  --dst-port or -r           Destination Port
  --write    or -w           Output file
  --http     or -l                      Output in HTML for download
  --verbose  or -v                      Verbose output
  --debug                               Debug output
  --quiet                               Return only a filename or an error
  --all                                 Check in all buffers not just current sniff buffer
  ***** Operation Mode Specific Stuff *****
  "At" mode.
  --each-way or -e         Number of pcaps each-way Default: 1
  --event    or -a         Parse a supported event log line e.g. Snort, Sourcefire, Exim etc
  --timestamp or -t                     Event mode - Look each way of this epoch value
  --sf                                  Timestamp in SF format, convert it
 "Window" mode.
  --start    or -b                      Start timestamp for searching in absolute mode
  --end      or -j                      End timestamp for searching is absolute mode
Conclusion
If you are still reading this post, I assume that this project is something that may interest you and can guess you have a couple of questions:
  • What log formats are currently supported?
    Right now not many, this is a preview release after all. Snort Fast alert, Snort Syslog, Exim4, and Sourcefire 3D Defense Center, Sourcefire 3D Sensor.
  • When can I expect the central manager / WebUI
    Don’t hold your breath right now 🙂 But seriously, I’m working on it and had some ghetto code working that I’m too shamed of to share.
  • What’s next? When can I expect something stable?
    This is the most important question. I’m aiming for a stable non-distributed release without the manager in the next couple of weeks. I think it’s best to get a small but stable feature-set into the community rather than one big buggy release in a couple of months time.
  • Can I help?
    What a wonderful question!? Yes you can! Right now I need to find problems and iron out some bugs. In a couple of weeks Ill be looking for someone who can write a simple web UI for me, my un-themed s don’t look great
  • How do I install, configure and get this preview release running?
    Keep your eyes open for a blog posting in a couple of days talking you through an install on Ubuntu
  • What are the requirements to run? Do you use sancp or anything like that?
    daemonlogger (Thanks Marty!), tcpdump, perl
  • When’s the automated extraction bit coming?
    I’m not sure if this will be before or after the webUI. I have a day job after all!
  • Will you support <foo> log file format?
    Maybe, I want to have built-in support for most common log formats, tell me what you need and send samples.
  • Where can I get the code from?
    Stay tuned for an alpha release this week on http://code.google.com/p/openfpc/

Happy Sniffing!

Advertisements

Written by leonward

May 17, 2010 at 8:00 am

6 Responses

Subscribe to comments with RSS.

  1. Hi Leon,
    This is just what I have been talking about for along time, and shown you with FPCGUI (Full Packet Capture GUI). I all ready have my proof of concept up and running. I need to finish the command line interface, but that is on my todo 🙂

    I have a trillion more thoughts, but I to have a day job 🙂

    Stay tuned 😉

    Edward Fjellskål

    May 18, 2010 at 7:52 am

    • Well lets talk about that over a beer in a couple of weeks 🙂
      I have some ideas how we could merge projects and support both approaches.

      -L

      leonward

      May 21, 2010 at 8:33 am

  2. Agree!

    I have already had an internal discussion here on how FPCGUI and OpenFPC are alike and what the difference are. I made the GUI so I could show ppl how one could search for sessions and then get the pcap straigt in your web browser->Wireshark. The extracting of the pcap is not based on the session data in the DB, and is purely scripts searching for sessions with tcpdump in the pcaps.

    The idea was to have a WebGUI so one can pass arguments via http and get a pcap back, and also to have a commandline way to do the same. One of my aims is to use it to automatic have a pcap-ringbuffer of important events (From events, may they be from an IDS or some syslog or what ever).

    So one pcap-ringbuffer for all the traffic using daemonlogger (say it is able to hold pcaps for X days), then another ringbuffer that holds pcaps extracted automatically from triggered events (That would be much less data, so it might hold YY days of event-pcaps). So when I have an event, and I really want to see the whole traffic related to the event, I more or less would be sure to have it.

    Beers 🙂
    Edward

    Edward Fjellskål

    May 21, 2010 at 9:38 am

  3. Hi Leon!

    This looks like a very interesting project. Actually I’ve been working on something related for my master thesis. I’d very much like to discuss this with you in more detail. Maybe I can participate in some way. If your interested send me an email.

    Tommy

    Tommy

    May 25, 2010 at 6:42 pm

    • Thanks for the offer!
      Ill be in touch when something comes up.

      leonward

      May 26, 2010 at 11:28 am

  4. Update: OpenFPC has a home: http://www.openfpc.org

    leonward

    August 3, 2010 at 6:21 pm


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: