Sourcefire Quick Tip: Custom RNA Service Detection

As part of my Sourcefire “Quick Tip” series, a new video has been uploaded by our marketing guru’s to YouTube.

This time I ramble on about creating custom service detectors for Sourcefire’s RNA engine. This allows you to detect the network applications  in use on your network and track them in the context of IPS events, real-time network change detection, Service and host white-listing as well as general network mapping.

I hope it’s of use to you out there. I haven’t got my next quick-tip subject planned yet, so if anyone has a suggestion please let me know.

Adding RNA Custom Services

If you would like to learn more about RNA, take a read here.

-Leon

Sourcefire Quick Tip: Host Input API and HippiEd

The on-line marketing team over at Sourcefire’s world domination HQ asked me to throw together some content for our Facebook / Youtube pages.

The result is a series of video quick tips showing some rarely talked about features of Sourcefire’s products. The first video is now available on the Youtube channel and Facebook page. The tool mentioned is available here.

HippiEd

Expect a few more over the next couple of weeks.

-Leon

Snoge and Dumbpig now on Google Code

I have moved the code repositories for DumbPig and Snoge over to google code.

Feel free to check the code out of subversion to look at the latest bleeding edge features.

DumbPig -> http://code.google.com/p/dumbpig

Snoge -> http://code.google.com/p/snoge/

ET RBN Blacklists with Snort and DumbPig

I spent a few minutes updating DumbPig to work with Marty’s latest blacklist patch, with some great results. It looks like Marty has done a great job in keeping packet performance high while providing a rich blacklist configuration. DumbPig for the as yet unenlightened, processes Snort rulesets and offers advice when a “dumb” rule is detected. Blacklist snort rules are a good example if dumbness so I thought I would focus in a bit on why and how to use these tools.

Before digging into the tech, let me tease you with some performance numbers. My test below was basic, but should provide some relation to the real world.

Test 1) Stock snort.conf and VRT subscription release (processing approx 1.2GB of pcap files)

$ snort –pcap-dir /var/local/pcaps/defcon-2004/ -A fast -l /tmp -c /etc/snort/snort.conf

Run time for packet processing was 46.994580 seconds
Snort processed 2752654 packets.

Test 2) Stock snort.conf, with the “emerging-rbn.rules” (20/july/2009) added

$ snort –pcap-dir /var/local/pcaps/defcon-2004/ -A fast -l /tmp -c /etc/snort/snort.conf

Run time for packet processing was 89.749301 seconds
Snort processed 2752654 packets.

Test 3) Stock snort.conf, but using a DumbPig created blacklist file from the same “emerging-rbn.rules”

$ snort –pcap-dir /var/local/pcaps/defcon-2004/ -A fast -l /tmp -c /etc/snort/snort.conf

Run time for packet processing was 48.348535 seconds
Snort processed 2752654 packets.

So that’s a whopper of a performance increase while maintaining the same IP based detection ability. I don’t claim you will see the same performance in the wild, but an increase like this should get your attention. Feel free to re-create the tests on your own network and let me know the results (for my interst only).

Configuring Snort 2.6.4.1 with the Blacklist patch (v2)

I have seen a few posts on the internet where people have run into issues configuring and using the blacklist patch. Below are the steps I took to build a system in prep for this weeks Snort Users webinar on DumbPig and other tools so I thought I would share the configuration here.

Starting platform: Debain Lenny – base installation + ssh and sudo (with lward in the sudoers)

1) Install the debs we know will be needed for the compile of Snort

lward@webexprep:~$ sudo apt-get install libpcap0.8 libpcap0.8-dev libdumbnet-dev build-essential libpcre3-dev automake autoconf libtool

2) Download and extract the snort 2.8.4.1 source

lward@webexprep:~$ wget http://dl.snort.org/snort-current/snort-2.8.4.1.tar.gz
lward@webexprep:~$ tar -zxvf ./snort-2.8.4.1.tar.gz

3) Download and extract the Blacklist patch

lward@webexprep:~$ cd snort-2.8.4.1/
lward@webexprep:~/snort-2.8.4.1$ wget http://www.snort.org/users/roesch/code/iplist.patch.v2.tgz
lward@webexprep:~/snort-2.8.4.1$ tar -zxvf ./iplist.patch.v2.tgz

4) Read the README.iplist

5) Pactch your Snort source tree

lward@webexprep:~/snort-2.8.4.1$ patch -p1 < iplist.patch

6) Rerun aclocal / automake/ autoconf

lward@webexprep:~/snort-2.8.4.1$ aclocal -I m4
lward@webexprep:~/snort-2.8.4.1$ automake
lward@webexprep:~/snort-2.8.4.1$ autoconf

7) Configure Snort (and enable IP listing).

lward@webexprep:~/snort-2.8.4.1$ ./configure –enable-iplist

(note there should be no errors , if you have m4 prelude messages, see the comment at the bottem of this post).

Compile  / install

Debain / Ubuntu users: You will have to “fix” the dnet/dumbnet fsckup created by the decnet libraries poluting the debian package namespace. A simple symlink will suffice.

lward@webexprep:~/snort-2.8.4.1$ sudo ln -s /usr/include/dumbnet.h /usr/include/dnet.h
lward@webexprep:~/snort-2.8.4.1$ make
lward@webexprep:~/snort-2.8.4.1$ sudo make install

9) Set up your snort configuration.

lward@webexprep:~/snort-2.8.4.1$ sudo mkdir /etc/snort
lward@webexprep:~/snort-2.8.4.1$ sudo chown lward /etc/snort/
lward@webexprep:~/snort-2.8.4.1$ cp etc/* /etc/snort/
lward@webexprep:~/snort-2.8.4.1$ vi /etc/snort/snort.conf

(Change your RULE_PATH to /etc/snort/rules)

10) Grab the VRT snort ruleset from snort.org, and stuck it in your home directory. This exact process will depend on your subscription level.

11) Set up the VRT snort rules

lward@webexprep:~/snort-2.8.4.1$ cd
lward@webexprep:~$ pwd
/home/lward
lward@webexprep:~$ ls
snort-2.8.4.1  snort-2.8.4.1.tar.gz  snortrules-snapshot-2.8.tar.gz
lward@webexprep:~$ tar -zxf ./snortrules-snapshot-2.8.tar.gz
lward@webexprep:~$ cp -r rules/ /etc/snort/

12) Grab yourself a pcap to test and play with

lward@webexprep:~$ wget http://rm-rf.co.uk/downloads/Honeynet-RFP-iis.tgz
lward@webexprep:~$ tar -zxvf ./Honeynet-RFP-iis.tgz

13) Test snort

lward@webexprep:~$ snort -c /etc/snort/snort.conf -A fast -l /tmp -r ~/Honeynet-RFP-iis.pcap

14) Enable, and test the blacklist functions

lward@webexprep:~$ echo preprocessor iplist: blacklist TestBlacklist /etc/snort/rules/test.blacklist >> /etc/snort/snort.conf
lward@webexprep:~$ echo 172.16.0.0/16 > /etc/snort/rules/test.blacklist
lward@webexprep:~/snort-2.8.4.1$ snort -c /etc/snort/snort.conf -A fast -l /tmp -r ~/Honeynet-RFP-iis.pcap

<Check that your are getting blacklist events in your /tmp/alert file. Make sure you add a CIDR that exists in your pcap test data!>

15) Install dumbpig

Install required perl modules from CPAN

lward@webexprep:~$ sudo cpan -e “Parse::Snort”
lward@webexprep:~$ sudo cpan -e “LWP::Simple”
lward@webexprep:~$ wget http://rm-rf.co.uk/downloads/dumbpig
lward@webexprep:~$ chmod +x ./dumbpig

16) Grab the latest Emerging threats RBN list

lward@webexprep:~$ wget http://www.emergingthreats.net/rules/emerging-rbn.rules

17) Convert the rule file into a blacklist

lward@webexprep:~$ ./dumbpig -q -r emerging-rbn.rules -b /etc/snort/rules/rbn.blacklist

DumbPig will detect rules that will work best in a blacklist, and add them to the file “rbn.blacklist”. For more usage inforamtion, take a look at the dumbpig page.

lward@webexprep:~$ head -n 5 /etc/snort/rules/rbn.blacklist
# Autogenerated blacklist by DumbPig from emerging-rbn.rules
# Contact leon.ward@sourcefire.com
# For more information about dumbPig visit http://rm-rf.co.uk
114.80.67.30/32 114.80.67.32/32 <snip>    # From Sid 2406000 : “ET RBN Known Russian Business Network IP TCP (1)” : emerging-rbn.rules
114.80.67.30/32 114.80.67.32/32 <snip>    # From Sid 2406001 : “ET RBN Known Russian Business Network IP UDP (1)” : emerging-rbn.rules

18) Reconfigure your snort.conf to the the rbn.blacklist

Something like this should work for you” “preprocessor iplist: blacklist RBN_Hosts /etc/snort/rules/rbn.blacklist”

And that’s it, the rest is up to you to use/abuse as you need.

Troubleshooting

Troubleshooting build problems. If you are having fun with one of the below, start with a clean source tree, re-patch, and follow step 6 (note the aclocal command).

undefined reference to `SetupIpList'
configure.in:1050: warning: macro `AM_PATH_LIBPRELUDE' not found in library

Happy Snorting - Leon

DumbPig 0.5 Updates, bug fixes, and real-word usage

The original release of DumbPig seemed to be well received, and promoted a load of quality feedback. I found some time earlier to take action on the first load of requests, and advice so please provide a warm welcome to version 0.5.

Changes in this version.

• Better support for rules copy/pasted out of the Sourcefire Defense Center GUI
  • (Right click on a rule, and go to documentation to see it’s source) (thanks @tomdixn)
• Removal of the mega if/then/else block (Nigel beat me into submission)
• Support for some forgotten keywords
• Censor ability to hide the sensitive parts of private rules
• Check for updates over the internet
• Addition of some new keywords (thanks @tryke)
• Better whitespace normalization

I hope Dumbpig could become a useful tool for the community, Documentation and code are available on the Dumbpig page.

DumbPig – Automated checking for Snort rulesets

From time to time I have the honour of working with “special” snort rule-sets, ones that I’m not allowed to analyse or take away. I understand that some like to keep their detection closed, but when they have errors in them that ….

• Prevent an import into the Sourcefire Defense Center
• Cause detection to run like a pig
• Incorrectly assign event priority
• Behave in a completely different way to how the “expert writer” expected them to

… I have to sit up and bitch for a while. I also see the same problems again and again in public rule sets, so I wanted to make a way to automate my way out of pain.

Introducing DumbPig, “Because your rules are not so good actually”.

DumbPig parses a snort rule-set, and depending on command line options, will recommend “fixes” for dumb Snort rules.

Take these two dumb rules as examples:

alert ip any any -> any 53 (msg: “I found a scary DNS name woooo”; content: “woot”; sid: 1;)

drop tcp 1.1.1.1 any -> $HOME_NET any (msg: “Communication with nasty host”; sid: 2; rev:1;)

Sid:1 above has some obvious problems for use in the real-world

• The IP protocol header doesn’t have port numbers, it is common to see someone write a rule like this in an attempt to search for content in both TCP and UDP data. It is faster and correct to create two rules, one for TCP and one for UDP.
• There isn’t a revision number, this makes it hard to track the rule through it’s evolution. This is required for importing to a Sourcefire Defense Center along with some other tools.
• There is no classificaion data to equate what the discovery of this rule means to the user
• There is no prioritization specification (partially due to the above)

Sid:2 Also has some other issues

• It doesn’t do any deep-packet checks, what’s the point in wasting IPS cycles to do what a firewall or router can do much faster. This being said, Marty has recently added a Snort preprocessor for blacklisting, this rule would be a good candidate for use in this preproc.

[20:30:48]lward@drax~/Documents/Sourcefire/Tools/dumbpig$ ./dumbpig.pl

Usage dumbPig <args>
   -r or --rulefile  <rulefile>
   -s or --sensitivity <1-3> Sensitivity level, Higher the number, the higher the pass-grade
   -b or --blacklist       Enable blacklist output (see Marty's Blog post for details)
   -p or --pause           Pause for ENTER after each FAIL
   -w or --write           Filename to wite CLEAN rules to
   -q or --quiet           Suppress FAIL, only provide summary

It’s simple to use, just give it a rule file and inspect the output report. It also presents the rule in Snort multi-line format (escaped line breaks) to make it easy to read when things get complicated like the

[20:28:15]lward@drax~/Documents/Sourcefire/Tools/dumbpig$ ./dumbpig.pl -r dumb.rules -s 3 -b

DumbPig version 0.1 - leon.ward@sourcefire.com
Because I hate looking for the same dumb problems with "Private" snort rule-sets

  __,,    ( Dumb-pig says     )  
~(  oo ---( "ur rulz r not so )
  ''''    ( gud akshuly" *    )   

Config
----------------------
* Sensitivity level - 3/3
* Blacklist output ENABLED
* Processing File - dumb.rules
* Pause - 0
* Output clean rules to - 0
* Quite mode = 0

--- [FAIL] 1 on line 1 (dumb.rules) -----------------------------
Reasons:
 - IP rule with port number (or var that could be set to a port number).
   This is BAD and invalid syntax.
   The IP protocol doesn't have port numbers.
   If you want to inspect both UDP and TCP traffic on specific ports use two rules,
   its faster and valid syntax.
 - No revision number! Please add a rev: keyword
 - No classification specified - Please add a classtype to add correct priority rating

alert ip any any -> any 53  (
 msg:  "I found a scary DNS name woooo"; \
 content:  "woot"; \
 sid:  1; \
 )
# alert ip any any -> any 53 (msg: "I found a scary DNS name woooo"; content: "woot"; sid: 1;)

--- [FAIL] 2 on line 2 (dumb.rules) -----------------------------
Reasons:
 - No classification specified - Please add a classtype to add correct priority rating
 - No deep packet checks? This rule looks more suited to a firewall or blacklist
 - IP rule without a content match. Put this in a firewall!

drop ip 1.1.1.1 any -> $HOME_NET any  (
 msg:  "Communication with nasty host found"; \
 sid:  2; \
 rev: 1; \
 )
# drop ip 1.1.1.1 any -> $HOME_NET any (msg: "Communication with nasty host found"; sid: 2; rev:1;)

----Blacklist----
Add this lines to your snort.conf
preprocessor iplist: blacklist { 1.1.1.1/32  }
--------------------------------------
Total: 2 fails over 2 lines in dumb.rules
- Contact leon.ward@sourcefire.com

Think of this as a preview release, give me feedback and find me bugs!

Download dumbpig here.
-Leon

Thoughts of the cloud – Part one of …… some.

I (via Sourcefire) recently had the opportunity to present at a could computing security event in Ireland organised by Calyx, unfortunately for me the cloud-specific focus of the event managed to slip past both myself and my colleagues in marketing unnoticed until the day before the event. This forced me to take the audience on an unexpected thirty minute tangent away from the cloud centric content they were expecting (sorry about that), but while I wasn’t talking I was given a great opportunity to witness the confusion that is presented as “The Cloud”.

Clearly cloud-stuff is getting more and more headlines, especially when you link the “C” word together with security.

Each presenter provided a slightly different definition of what cloud computing meant to them (and their marketing departments), but somehow they all managed to agree on key security risks associated with suggestions of …….

• Dumping mission critical internally hosted services and throwing them into the cloud
• Placing your “trusted data” in the trust of another “untrusted” party
• Believing that a cloud provider (SAAS, PAAS, IAAS etc) has a magic ability to look after their systems better than your people look after your systems
• Burying you head in the sand while shouting “la la la don’t remind me that I’m still accountable for all of this with little to no control!”

Quite correctly none of the presenting vendors offered a silver bullet to solve these issues, and if they had offered one up I’m sure someone in the audience would have called them a liar. Instead they provided intelligent thoughts and opinions to potential workarounds, however I don’t remember them touching on what I see as an important and overlooked point.

Cloud technology will rarely provide a complete migration path, it is most likely to be a technology addition. I won’t go into the justification of my point of view right now, but believe me this is the way I see it. This also means that my non-cloud “Know your network better than your enemy” tangent that I took the audience on is still important. I hope the “C” word moves further along the hype-cycle path to sit somewhere better understood soon.

Earlier today I read Amrit Williams’ wonderfully sarcasm-rich post on a similar subject, I relate to it because it touches on my views of cloud addition. Reading it also reminded me to hit the publish button on this posts draft.

-Leon

TweetYard – Sourcefire and Snort alerts to Twitter

Update: – You can download tweetyard here.

There has been some of discussion of late on the snort-* lists of late regarding Unified alerting vs direct DB access.

I stopped storing events in a DB years back when I stopped using ACID (and yes that was back-in-the-day before BASE came into being). My personal Snort requirements are pretty simple and fast output has always worked well for me linked with a load of swatch-foo and custom perl scripts. After hanging my head in shame for not converting to unified yet (cobblers children clearly have no shoes over here) I thought it would be wise to put some effort in.

I used to receive all of my Snort IDS events via email, but email is *so* web 1.0. So I thought I would hook into Twitter for real-time alerting

So far, so good, and it only took about an hour to build. Kudos to Jason Brvenik for his snort-unified.pm and sample barnyard replacement, it was a good base for what I wanted to hack together. Because I put this together more than fun more than anything else, feel free to follow a censored Twitter feed of my IPS events (If you didn’t have enough to deal with already).  I have blanked the IPs of my protected systems in an attempt to raise the smarts-to-abuse bar up 0.2 inches above short skiddie tall.

I will upload the code when I get a spare couple of minutes, but as I will be attached to the Sourcefire booth @ Infosecurity London for the next three days it may take a while. Hooking it into Sourcefire’s Estreamer is also on the cards the next time I get some down-time.

If anyone is at the show, feel free to drop by the Sourcefire booth and say hi (and to bring me a Coffee at the same time).

-Leon

Openpacket.org – example.com PCAP files

I am in the process of uploading a load of pcap files to openpacket.org from my “example.com” collection. Because openpacket doesn’t provide an interface to include supporting data, below is network map that should help anyone who needs to use these pcaps. They were sniffed from a test network I built and should contain a good mix of systems and protocols.

Expect to see:

• http
• https
• pop3
• smtp
• bittorrent
• ldap
• irc
• msn
• smb
• dcerpc
• ssh

Network Map - Example.com

While I fight with Openpcap’s upload limits, a complete archive of example.com can be found here.

-Leon

Snort-2.8.4 – What’s the fuss about?

I’m writing this post on the 14:00 BST 7/April, this timestamp is important because snort 2.8.4 has not yet been released to the public and when it is I get the feeling that there could be a lot of traffic on the Snort forums and mailing lists.

The morning after the 2.8.4 release I imagine a lot of people will have woken up to a Snort sensor that is no longer functioning, now let me take guess at what has most likely has happened.

• You are running Snort version 2.3.x or earlier
• You are using oinkmaster, or similar to provide an automated tool to download new snort rules and restart the detection engine
• You have set oinkmaster to download the “CURRENT” release
• Snort now fails to start because of these new rules

So what’s happened in plain English?

A new preprocessor has been released called DCERPC2. If stuff has just broken for you, the version of rules that your automated update process has downloaded requires this new preprocessor to provide detection for a load of new “stuff” in the NetBIOS category.

You now have two choices:

• Update Snort to 2.8.4 to stay current with the detection that Sourcefire’s VRT provide (Good)
• Don’t update Snort. Run with an older rule set that doesn’t provide detection for current threats (bad)

Unfortunately it doesn’t look like there is much choice in it, go get the latest tarball and install. While you have wget running in the background, lets examine whet DCERPC2 does, and why this change has been made.

Snort is a real-time protocol modeller, it inspects network traffic at the application layer (and below) and builds a state-machine. This isn’t just simple TCP state data like a firewall would track, it follows application protocol state. The Snort language allows a rule-writer to build his/her own application state tables as needed to support pretty much any protocol, so if a vulnerability is found in the brand new “foo” protocol, a state-machine to track “foo” connections can be built by the rule writer as needed.

Here is an example of how application protocol state is tracked. The magic word here is “flowbits”. I have copied a rule out of the smtp.rules as an example. Note that I have also simplified it for display here so don’t expect it to work in the real-world any more.

alert tcp $EXTERNAL_NET any -> $SMTP_SERVERS 465 (msg:"SMTP SSLv3 Client_Hello request"; \
flow:to_server,established; \
flowbits:isnotset,sslv2.client_hello.request; \
flowbits:isnotset,tlsv1.client_hello.request; \
content:"|16 03 00|"; depth:3; \
content:"|01|" ; depth:1; offset:5; \
flowbits:set,sslv3.client_hello.request; \
flowbits:noalert; \
service smtp; classtype:protocol-command-decode;)

When traffic that matches this rule is seen, it sets a “flowbit” on the stream that can be checked in other rules. Here we can see that for this rule to “fire” and have the SSLv3 flowbit set, the stream cannot be SSLv2 or TLSv1. Also note that when this rule does “fire” it won’t create an alert, it just sets the state for further processing.

This method has been traditionally used to track application protocol state in NetBIOS/DCERPC  traffic in the same was as all other protocols. The method has has worked well for years but because of all the different ways you can access potentially access some vulnerable functions in DCERPC, the netbios rule-set has grown noticeably. DCERPCv2 deprecates most of these flowbit based protocol modellers for DCERPC traffic, this allows the VRT to shrink the netbios.rules category down to make it more manageable, this is great for those who hand-hack config files. Anyone using Sourcefire 3D won’t really notice much difference when DCERPC2 gets released in a SEU, but you would have missed out on a lot of pain already because of the built in flow-bit dependency tracking.

If you are running a commercial solution that won’t work with this rule-set, I would like to hear from you, please get in touch.

-Leon