Well, I'm fresh back from a short vacation down in Argentina and I find myself working on a problem today that required the use of a protocol analyzer or "sniffer". It got me to thinking about the fact that many network administrator's today have never actually taken a packet trace or analyzed a packet. Furthermore, when the need does arise, many of the people I deal with don't really know how to get started.
So, with that in mind, here are few things to keep in mind if you find yourself in that situation...
Head Geek's Top 5 Things to Remember about Analyzing Packets
#5 - Don't assume that the answer is in the packet. Remember the top 3 network troubleshooting steps - check the cable, check the cable, check the cable.
#4 - If you're going to go looking at packets, use Wireshark. You can download it for free from wireshark.org. It's the best protocol analyzer out there - period.
#3 - Remember that with today's ethernet switches you have to mirror (span) the data from the port that you want to monitor to the port where your analyzer is connected. A few of us old timers keep a spare 4 port ethernet hub laying around for cases where reconfiguring the switch isn't convenient...
#2 - User capture and view filters to limit the number of packets that you have to look at. For instance, if you're troubleshooting a website performance issue you can probably filter everything out that's not to/from the client machine and the webserver and everthing that's not to/from TCP port 80.
#1 - Combine the data that you get from looking at the individual packets with the data that you get from investigating packet flows leveraging a technology like NetFlow. If you don't have a Netflow appplication, you can download a free evaluation version of the SolarWinds Engineer's Toolset and Orion NetFlow Traffic Analyzer from the SolarWinds website. The evaluation versions are fully functional and work for 30 days.
Packet sniffing isn't for the faint of heart but I've solved some problems that were real buggers with this method. If you've got some cool stories of issues you've solved after investigating the packets post a comment here and I'll send you a SolarWinds T-shirt.
Flame on...Josh
I was working at a bank and when I arrived at work one morning, the IT manager of the dealing room, asked me what I had changed last night on the network. I told him truthfully, nothing, as I was in the pub enjoying a few pots of draught. On questioning him further, he said his main dealing application wasn't working and the DBA said it must be the network as he had rebooted his servers several times. One of my network colleagues was already working on the issue and when I asked him for details he confirmed that he had done all the basic checks and could find nothing amiss.
My colleague was very meticulous and so I assumed it would be a deeper problem and proceeding to capture some packets of the problem with Wireshark (or Ethereal as it was then known). BTW: I agree that Wireshark is the best!!!!
I had come in a bit late, as it had been a heavy night and the system had been down for a few hours. I gazed at the packets of the application failing to connect to the server. There were heaps of Windows name resolution packets. I slowly pieced the puzzle together. The application was making a DNS request but the admins had used the server's IP address in the workstation configuration and not the server DNS name. So a DNS query went out with the IP address of the server. For years this worked as when that query failed it used the IP address to connect directly. (I later confirmed that the app worked like this!)
The server IP address was 10.0.32.99. Now before the DNS query failed, it sent out a few queries based on the search suffix, i.e. companyname.co.za and then co.za! Luck would have it that that night a company registered the domain 99.co.za and instead of failing the DNS query succeeded! The app then attempted to connect to the server via the firewall connected to the external Internet, which blocked it!
We reconfigured the app on the workstation to use the DNS name of the server corona.companyname.co.za and the dealing room application started working on all the workstations again.
The IT manager did not understand why not using DNS was not a good idea. (This is not the only time I have seen IP addresses instead of DNS names being used - IP addresses in a config are always a bad idea!) Secondly, the manager wanted to take the guy who had registered 99.co.za to task. I suggested it was a bad idea, as they had not done anything wrong but if he was fuming. I explained it to him about eight times before he understood. When he finally understood he took us network guys out for a very fine dinner!
An interesting and timely post.
I just registered for this site (long time IpMonitor user) and was surprised that my password was sent back to me in clear text.
Here's my question: could I use a packet sniffer (like WireShark) to look for email packets containing "Thwack" and "Your Account is Enabled" and personal and password information?
I've been using this as an example trying to get web sites and forums to stop this practice, but I've never really know technically if it was as easy as I suspect.
Thanks, Richard
PS -- Please stop sending passwords in clear text.
Mo,
First of all, thanks for the heads--up on this. I've sent a note to our community site manager and we'll get this remedied. We use Microsoft Community Server so hopefully this is a known issue.
Regarding using a sniffer to capture data like this, it is possible but only if you're physically connected to a wire that the packets transit on their way from the web server to your e-mail client. So, in a nutshell, it's a longshot...
Josh
Or if you have administrative access to the networking devices. Many networking devices now have the ability to capture packets as a feature. The Junipers always had this and I have used these to dump the capture into Wireshark.
You can now also do it with a Cisco router (but I haven't tried this yet myself!)
www.cisco-tips.com/ios-packet-capture-and-auto-upgrade
Redpineapple,
Great point.
i think a good best practice is that if you have to recover your password, login and change it as soon as you get the new one...
If you want to learn the black magic of packet gazing get Stephen Northcutt's book "Intrusion Detection" on New Riders Press. It's from the days before automagic-do-all-clicky-click IDS-systems when IDS operators actually looked at RAW packets. Next to Steven's classic "TCP/IP Illustrated" this is my number one recommendation to learn packet inspection.
Cheers,
Chris
Oooh! Missed the free T-Shirt offer above!
Cool story: I had set up a pair of dual ASA5500 clusters with a bunch of yucky MTA changes on various media between them, and was getting intermittent dropouts on SMTP connections from various free email providers (GMX, Hotmail, etc.) to the German government network that was behind them.
The heat on the issue was WAY up due to constituents' email getting rejected when they tried to contact their representatives. Not a good situation! We could see the connections being built (full three-way TCP handshake) and then just suddenly stopping. The first step, obviously, is a sort of comparative regression analysis: i.e., figure out whether every device the packets were traversing was seeing the same connection. That's pretty painstaking work to validate, and it's out of the scope of this story to describe the exact solution, but I basically abused scapy.py which is a packet generation tool written in python to remove the parts of the packets that were changing (the L1 address for example) and strip them down to layer four and part of layer three to compare. After all that was done .... check .... recheck .... nothing! Every box in the path of the packets was seeing the same connection. Something else was dropping it.
The next step was to validate the packets bit-by-bit and look for the one that was out of place. Bit-for-bit in this case means 1s and 0s and looking for what was out of place. At some point I realized that I had often SEEN but never ANALYZED what the sackOK was for. (For a detailed discussion see www.firewall.cx/tcp-analysis-section-6.php) The full-text-search for sackOK in the ASA manual brought up something interesting: there was an option to turn off selective acknowledgments! Let's flip that switch .... shall we?
Suddenly no more drops, we tested and tested and tested and no more connections were dropping. The ASA seemed to have an issue with the packets and was dropping them without so much as a log message even with log levels turned all the way up. It turned out that the specific revision of the ASA software had an issue with sackOK and was just dropping the connection when it saw ack numbers being skipped. Opened a case with Cisco and got it dealt with there too.
Most likely, the problem never would have been solved without a packet sniffer and the requisite knowledge. Packet pirates ahoy!