bash script: capturing tcp traffic on a remote server sometimes works, sometimes fails. No errors

dot

Background

I am running BusyBox in the remote server.

I have a bash script that does two things:
1. via ssh, starts a sub process to monitor tcp traffic using tcpdump command. Save results to a file - either on remote machine or local machine. Tried both.
2. starts a second sub process to generate tcp traffic.

Code Snippet:

#html_tcpdumpfile="$(ssh remotemachine.mydomain.net \"mktemp\")"
html_tcpdumpfile=$(mktemp)

test_steps=(
    #"{ ssh remotemachine.mydomain.net \"timeout -t 20 tcpdump -nvi eth0 port 5060 > "$html_tcpdumpfile" \" ; }" 
    "{ ssh remotemachine.mydomain.net \"timeout -t 20 tcpdump -i eth0 port 5060 \"> $html_tcpdumpfile; }"   
    "{ ssh remotemachine.mydomain.net \"timeout -t 15 cat /tmp/htmlemail.txt | /etc/postfix/process_email.py \"; }"
 )
pids=()
for index in ${!test_steps[@]}; do       
      (echo "${test_steps[$index]}" | bash) &
      pids[${index}]=$!
      echo "$pids[${index}] is the pid"
done

#shouldn't really need this because of the timers but... just in case...
for pid in ${pids[*]}; 
do   
  wait $pid; 
done;
# ============ ANALYZE TEST RESULTS
echo "========== html_tcpdumpfile CONTENTS ============="
cat $html_tcpdumpfile
echo "========== html_tcpdumpfile CONTENTS ============="

Problem

Sometimes, the tcpdump command doesn't capture anything, and at other times it does. There are no error messages when it fails to capture.

What I've tried So far

  1. As you can see, I've tried to change the location of the dump file between the remote machine and the local one. That doesn't seem to make a difference.

  2. I've proven that TCP traffic is ALWAYS generated... each time I run the script because I have another ssh session open and i can see the traffic being generated. It's just that my script intermittently fails to capture it.

  3. I've tried to increase the timeout value on the tcp session to something huge to make sure I give it enough time. But I don't think that's the problem.

Any suggestions would be appreciated. Thanks.

EDIT 1

I tried to introduce a sleep in between launching each subprocess:

pids=()
for index in ${!test_steps[@]}; do       
      (echo "${test_steps[$index]}" | bash) &
      sleep 5
      pids[${index}]=$!
      echo "$pids[${index}] is the pid"
done

But that doesn't make a difference either.

EDIT 2

I changed the tcpdump command to look like this:

test_steps=(     
    "{ ssh remotemachine.mydomain.net \"timeout -t 30 tcpdump -nlc 100 -i eth0 port 5060 \"> $rtf_tcpdumpfile; }" 
    "{ ssh remotemachine.mydomain.net \"timeout -t 20 tail -f /var/log/messages \" > $syslog; }"    
    "{ ssh remotemachine.mydomain.net \"timeout -t 15 cat /tmp/htmlemail.txt | /etc/postfix/process_email.py \"; }"
 )

The tcpdump still fails to capture intermittently, but ... what's interesting is that the syslog is always successfully captured. (the python script actually writes to the syslog when it's invoked and so I can see /prove that the script is working)

Rui F Ribeiro

First off, if you are dealing with an appliance/iOT with a limited space, I would deal with the output in the calling side, i.e. using the > after the ssh commands as in

ssh "command" > output.txt

As for tcpdump I would not kill it as a policy all the time, risking losing buffers. You might have not output maybe because of that.

I would place a limit on packets captured. I would also try not to solve DNS. As in, for capturing 100 packets:

tcpdump -nc 100 -i eth0 port 5600

When you are storing the capture file on the local system, you should only run cat locally and not remotely and locally.

Likewise, when you are running both tcpdump and cat remotely, you are launching both at the same time, and both the remote and local cat won't have nothing to show.

Following the suggestion of @MarkPlotnick, I also added -lto tcpdump to make it line buffered. That may obviate the need for the -c option. I would use both.

So I would change that script for:

#!/bin/bash
html_tcpdumpfile=$(mktemp)

ssh remotemachine.mydomain.net "timeout -t 20 tcpdump -nlc 100 -i eth0  port 5060 " > $html_tcpdumpfile

cat $html_tcpdumpfile

rm $html_tcpdumpfile

Or we might not even need to create explicitly a temporary file:

#!/bin/bash

ssh remotemachine.mydomain.net "timeout -t 20 tcpdump -nlc 100 -i eth0  port 5060 " \
| less

Lastly, I would advise deleting all the temp files created, specially on the remote side.

PS: the OP mentioned in comments the remote system is BusyBox and as such the timeout options are different than in the coretutils package. I also edit the question for it to mention BusyBox.

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

TOP Ranking

  1. 1

    Failed to listen on localhost:8000 (reason: Cannot assign requested address)

  2. 2

    pump.io port in URL

  3. 3

    How to import an asset in swift using Bundle.main.path() in a react-native native module

  4. 4

    Loopback Error: connect ECONNREFUSED 127.0.0.1:3306 (MAMP)

  5. 5

    Compiler error CS0246 (type or namespace not found) on using Ninject in ASP.NET vNext

  6. 6

    BigQuery - concatenate ignoring NULL

  7. 7

    Spring Boot JPA PostgreSQL Web App - Internal Authentication Error

  8. 8

    ggplotly no applicable method for 'plotly_build' applied to an object of class "NULL" if statements

  9. 9

    ngClass error (Can't bind ngClass since it isn't a known property of div) in Angular 11.0.3

  10. 10

    How to remove the extra space from right in a webview?

  11. 11

    Change dd-mm-yyyy date format of dataframe date column to yyyy-mm-dd

  12. 12

    Jquery different data trapped from direct mousedown event and simulation via $(this).trigger('mousedown');

  13. 13

    maven-jaxb2-plugin cannot generate classes due to two declarations cause a collision in ObjectFactory class

  14. 14

    java.lang.NullPointerException: Cannot read the array length because "<local3>" is null

  15. 15

    How to use merge windows unallocated space into Ubuntu using GParted?

  16. 16

    flutter: dropdown item programmatically unselect problem

  17. 17

    Pandas - check if dataframe has negative value in any column

  18. 18

    Nuget add packages gives access denied errors

  19. 19

    Can't pre-populate phone number and message body in SMS link on iPhones when SMS app is not running in the background

  20. 20

    Generate random UUIDv4 with Elm

  21. 21

    Client secret not provided in request error with Keycloak

HotTag

Archive