Table of Contents
- Introduction – How to use file rotation in tcpdump
- How to run rotating tcpdump captures as a background process
- How to use tcpdump filters along with rotating packet captures
- How to merge the captured files into a single pcap file for easy analysis
- Special Scenario : How to use nohup along with tcpdump rotating pcap
Introduction – How to use file rotation in tcpdump
Wondering how to do packet captures for longer durations despite having very less storage space ? Or how to run rotating packet capture using tcpdump ?
This article helps you understand a useful mechanism in the tcpdump tool which helps us to rotate the packet capture files based on some criteria.
Command Syntax used to do rotate files in tcpdump capture
Here is the syntax to perform such a packet capture
tcpdump -i <interface-name> -W <file-count> -C <file size> -w <capture-file-location and capture filename>
-i <interface-name> – using this option we can mention the interface this packet capture iis performed
Eg: -i ens37 || I am setting the packet capture on interface ens37
-W <file-count> – using this option we can enter the number of files to rotate the packet capture when some criterias are met
Eg: -W 5 || here I am asking the tcpdump to rotate 5 files while capturing the packets
-C <file size> – using this option we are setting the file size limit of each files so that when it attains that size it moves the capture to the next file
Eg: -C 100 || We are setting the file size limit to be 100MB , when the file size is 100MB , tcpdump moves the capture to the next file part of the rotated files
-w <capture-file-location> – using this option we can set the location on which the packet captures are meant to be saved , and also we can set the packet capture actual filename , this filename is used while rotating the files which slight modifications such as numbering the rotated files
Eg: -w /tmp/test.pcap || this sets the capture file to be test.pcap located inside the /tmp directory
-w test.pcap || this sets the capture file to be test.pcap located in the present working directory
Practical example
Now lets see an actual example,
In the following topology we have a continuous ping and some SSH traffic running behind the Ips specified , now we are going to create a rotated pcap on the ens37 interface on the machine centos-discoveringsystems.com
Gateway (192.168.44.1 )-------- ens37 (192.168.44.152 ) centos-discovering-systems.com
The following command , does the packet capture on the ens37 and rotates the packet capture on 5 files(-W) with the size of 5M (-C) and saves the files as dsrotate.pcap prefix (-w) in the present working directory. After running it for some time , i had to hit Ctrl+c to stop the packet capture
[root@discoveringsystems-centos ~]# tcpdump -i ens37 -W 5 -C 5 -w dsrotate.pcap
tcpdump: listening on ens37, link-type EN10MB (Ethernet), capture size 262144 bytes
^C <hit Ctrl+c to stop this >
23311 packets captured
23320 packets received by filter
0 packets dropped by kernel
We can see that there are 5 files created due to the file rotation size limit set to 5MB . Starting from the dsrotate.pcap0 through dsrotate.pcap4 . Also we can see that the full rotation of 5 files happened and then it started to rewrite on the dsroatate.pcap2 which has grown to around 1.5M by the time we checked this output
[root@discoveringsystems-centos ~]# ls -ltrh | grep -i dsr
-rw-r--r--. 1 tcpdump tcpdump 4.8M Dec 30 11:37 dsrotate.pcap3
-rw-r--r--. 1 tcpdump tcpdump 4.8M Dec 30 11:38 dsrotate.pcap4
-rw-r--r--. 1 tcpdump tcpdump 4.8M Dec 30 11:39 dsrotate.pcap0
-rw-r--r--. 1 tcpdump tcpdump 4.8M Dec 30 11:39 dsrotate.pcap1
-rw-r--r--. 1 tcpdump tcpdump 1.5M Dec 30 11:40 dsrotate.pcap2
How to run rotating tcpdump captures as a background process
When we are running this rotating tcpdump , we found that it runs as foreground process in the bash session and then finally we need to hit Ctrl+c to stop it . Sometimes this is hard as when we initiate the bash shell through the ssh or other means , there is a chance that the ssh session gets timed out or we may accidentally close the session which can potentially stop the foreground process and our rotated packet capture ends. To avoid these issues we can run it as a background process. In this section we will discuss how to run the same as a background process
After executing the following command we made the same tcpdump thing into a background process
[root@discoveringsystems-centos ~]# tcpdump -i ens37 -W 5 -C 5 -w dsrotate.pcap &
[1] 7404
[root@discoveringsystems-centos ~]# tcpdump: listening on ens37, link-type EN10MB (Ethernet), capture size 262144 bytes
To check all the current background running process
[root@discoveringsystems-centos ~]# jobs
[1]+ Running tcpdump -i ens37 -W 5 -C 5 -w dsrotate.pcap &
And we can confirm that the file size grows and gets rotated as well
[root@discoveringsystems-centos ~]# ls -ltrh | grep -i dsr
-rw-r--r--. 1 tcpdump tcpdump 4.8M Dec 30 11:37 dsrotate.pcap3
-rw-r--r--. 1 tcpdump tcpdump 4.8M Dec 30 11:38 dsrotate.pcap4
-rw-r--r--. 1 tcpdump tcpdump 4.8M Dec 30 11:57 dsrotate.pcap0
-rw-r--r--. 1 tcpdump tcpdump 4.8M Dec 30 11:58 dsrotate.pcap1
-rw-r--r--. 1 tcpdump tcpdump 752K Dec 30 11:58 dsrotate.pcap2
[root@discoveringsystems-centos ~]# ls -ltrh | grep -i dsr
-rw-r--r--. 1 tcpdump tcpdump 4.8M Dec 30 11:37 dsrotate.pcap3
-rw-r--r--. 1 tcpdump tcpdump 4.8M Dec 30 11:38 dsrotate.pcap4
-rw-r--r--. 1 tcpdump tcpdump 4.8M Dec 30 11:57 dsrotate.pcap0
-rw-r--r--. 1 tcpdump tcpdump 4.8M Dec 30 11:58 dsrotate.pcap1
-rw-r--r--. 1 tcpdump tcpdump 864K Dec 30 11:58 dsrotate.pcap2
Now , how to close this background process , there are two ways , one is killing the process or bring it to the foreground and closing it with Ctrl+c
# we checked the background process job ID
[root@discoveringsystems-centos ~]# jobs
[1]+ Running tcpdump -i ens37 -W 5 -C 5 -w dsrotate.pcap &
# We used the Job ID to bring it to the foreground and then hit Ctrl+c to stop
[root@discoveringsystems-centos ~]# fg %1
tcpdump -i ens37 -W 5 -C 5 -w dsrotate.pcap
^C15329 packets captured
15369 packets received by filter
0 packets dropped by kernel
[root@discoveringsystems-centos ~]# jobs
[root@discoveringsystems-centos ~]#
Let’s say we want to use the other way of stopping it , That is by killing the process
# note down the process ID when this is executed , here the process id is 11122
[root@discoveringsystems-centos ~]# tcpdump -i ens37 -W 5 -C 5 -w dsrotate.pcap &
[1] 11122
[root@discoveringsystems-centos ~]# tcpdump: listening on ens37, link-type EN10MB (Ethernet), capture size 262144 bytes
[root@discoveringsystems-centos ~]# jobs
[1]+ Running tcpdump -i ens37 -W 5 -C 5 -w dsrotate.pcap &
# kill the process using the process Id 11122
[root@discoveringsystems-centos ~]# kill -6 11122
[root@discoveringsystems-centos ~]#
[1]+ Aborted tcpdump -i ens37 -W 5 -C 5 -w dsrotate.pcap
[root@discoveringsystems-centos ~]# jobs
[root@discoveringsystems-centos ~]#
How to use tcpdump filters along with rotating packet captures
Since we are running a rotating packet capture with specific size and a number of files to rotate we might end up seeing it easily getting rotated without capturing the flow or packets we actually want to capture during troubleshooting an intermittent issue . To avoid this issue , we need to be as specific as possible using the tcpdump packet filters while capturing.
In the following example , I am capturing only the ICMP / ping packets (using tcpdump filters) and rotating them as discussed earlier. This helps me to focus on the ICMP flow alone. If there is an intermittent ping drop , we can easily go back and check the captured files to pinpoint on the time of the issue and which side is at fault
[root@discoveringsystems-centos ~]# tcpdump -i ens37 icmp -W 5 -C 5 -w dsrotate.pcap &
[1] 14517
[root@discoveringsystems-centos ~]# tcpdump: listening on ens37, link-type EN10MB (Ethernet), capture size 262144 bytes
[root@discoveringsystems-centos ~]# job
bash: job: command not found...
[root@discoveringsystems-centos ~]# jobs
[1]+ Running tcpdump -i ens37 icmp -W 5 -C 5 -w dsrotate.pcap &
[root@discoveringsystems-centos ~]#
We can use the same technique to capture packets specifically using different tcpdump filters available based on our needs
How to merge the captured files into a single pcap file for easy analysis
In this example we are going to see a way with which we can merge all the pcap files into a single pcap file which are collected part of the rotating tcpdump
The following pcap files are going to be merged into a single pcap file for easy analysis using mergecap tool included in the wireshark package
[root@discoveringsystems-centos ~]# ls -ltrh | grep -i dsr
-rw-r--r--. 1 tcpdump tcpdump 4.8M Dec 30 11:59 dsrotate.pcap2
-rw-r--r--. 1 tcpdump tcpdump 4.8M Dec 30 11:59 dsrotate.pcap3
-rw-r--r--. 1 tcpdump tcpdump 4.8M Dec 30 12:00 dsrotate.pcap4
-rw-r--r--. 1 tcpdump tcpdump 1.3M Dec 30 12:04 dsrotate.pcap1
-rw-r--r--. 1 tcpdump tcpdump 3.1M Jan 3 10:13 dsrotate.pcap0
To install the wireshark and mergecap , please use the following commands for different Linux and mac releases
#Ubuntu , debian
apt-get install wireshark-common
#Centos
yum install wireshark
#Fedora
dnf install wireshark-cli
#Mac Os X
brew install wireshark
Mergecap usage format : By default it merges based on frame timestamps
mergecap -w <merged-pcap-file> <input-pcap> [<input-pcap2> ...]
Sometimes while trying to merge the pcap files , you might see this problem , this is due to the fact that one of the pcap files , in this case dsrotate.pcap1 has a packet which is not fully captured while using the tcpdump
[root@discoveringsystems-centos ~]# mergecap -w dssingle.pcap dsro*
mergecap: Error reading dsrotate.pcap1: Less data was read than was expected
This example is more clear than the merge error output
[root@discoveringsystems-centos ~]# tcpdump -r dsrotate.pcap1
reading from file dsrotate.pcap1, link-type EN10MB (Ethernet)
12:03:58.765757 IP discoveringsystems-centos.ds.com > 192.168.44.1: icmp
12:03:58.766024 IP 192.168.44.1 > discoveringsystems-centos.ds.com: ICMP echo reply, id 3805, seq 9449, length 1480
.
.
.
tcpdump: pcap_loop: truncated dump file; tried to read 1514 captured bytes, only got 654
We can fix the problem using the tcprewrite tool , this fixes the length issues by adding the padding to the packets and recalculates the checksums ,etc to fix it
tcprewrite --fixlen=pad -i dsrotate.pcap1 -o dsrotate.pcap1
Then running the same mergecap went through fine without any errors this time
[root@discoveringsystems-centos ~]# mergecap -w dssingle.pcap dsro*
[root@discoveringsystems-centos ~]#
Here is the merged pcap file
[root@discoveringsystems-centos ~]# ls -ltrh | grep -i dssingle
-rw-r--r--. 1 root root 18M Jan 3 11:17 dssingle.pcap
Special Scenario : How to use nohup along with tcpdump rotating pcap
In some OS , if you run the tcpdump as a background process we might still see the packets captured numbers on the screen as standard output , even though the process is running as background process , to avoid seeing that we can leverage the nohup command along with our rotating packet capture , this is specially useful when doing packet captures on the Arista switches, etc
nohup command itself can help command we are running from being aborted automatically when you exit the shell accidentally or the shell is closed due to the time out.
While using nohup , we need to consider these rules
Standard input will be redirected to /dev/null
Standard output will be redirected to nohup.out file created in the present working directory
Standard error is redirected to the standard output
Here is the example from an Arista switch while doing rotating pcap, When executing the following command in bash we see that the rotation of the file happens and we won’t see any standard input which you will see otherwise if you are not using nohup. One more thing to note here is any inputs you provide won’t be taken by the running command and you may have to use another shell to continue other inputs you want to provide
# we are dropping down into bash and getting to the flash directory has it will have more space when compared to other directories
arista-switch#bash
Arista Networks EOS shell
[admin@arista-switch]$ cd /mnt/flash
# using nohup
[admin@arista-switch flash]$ sudo nohup tcpdump -nevi ma1 port 22 -W 5 -C 5 -w dsrotate.pcap
nohup: ignoring input and appending output to 'nohup.out'
random-text jhbcvjfbvcrjcvb
#without using the nohup
[admin@arista-switch flash]$ sudo tcpdump -nevi ma1 port 22 -W 5 -C 5 -w dsrotate.pcap
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
Got 4
Got 6
Got 312
If you want to run this both as background process and want to redirect all the standard output to nohup.out file , you can use the following syntax
# using nohup and background process option
[admin@arista-switch flash]$ sudo nohup tcpdump -nevi ma1 port 22 -W 5 -C 5 -w dsrote.pcap &
[1] 25326
[admin@arista-switch flash]$ nohup: ignoring input and appending output to 'nohup.out'
#To kill this background process
[admin@arista-switch flash]$ sudo killall tcpdump
[admin@arista-switch flash]$
[1]+ Done sudo nohup tcpdump -nevi ma1 port 22 -W 5 -C 5 -w dsrotate.pcap