doug dobies
about | strengths | projects | linkedin | github | twitter |

Linux one liners

 

Contents

  1. Linux one liners
    1. Summary
  2. CPU related
    1. high cpu/proc queue
  3. Memory related
    1. low memory
    2. low memory (ignore process “inactive memory” like nimbus)
    3. Drop memory caches
    4. Very Detailed vmstat on memory
    5. Paging and Swapping levels (adjustable interval)
    6. Percentage of memory used
    7. Past two hours of memory usage from sar (may need modification if sar output in a language other than English)
    8. Daily memory utilization percentage from sar (again may change depending on language/sar output)
    9. Calculate memory usage by adding up usage in /proc
    10. Add Shared memory
  4. Disk space related
    1. low disk (current directory, takes into account sparse files)
    2. low disk (current directory)
    3. low disk (current directory, w/top 20 on largest directories)
    4. disk usage
    5. largest files in a directory (in MB)
    6. Find the largest files in a filesystem
    7. Compress files in current directory, older than 24 hours
    8. Find files last modified 2 days ago
    9. Remove files older than 7 days
    10. Find percentage of blocks that aren’t reserved for root (usually 95)
  5. Netstat and Networking Fun
    1. General netstat
    2. General netstat command to show all ports
    3. Show local listening TCP/UDP ports, and the process they belong to
    4. See how many connections each service has
    5. Ton of network statistic info
    6. Get MAC Addresses from dmesg
    7. Automatically Update MAC addresses in ifcfg scripts
    8. Compare MAC Addresses between actual and ifcfg scripts
    9. TCP dump examples
    10. Fail over and test Network interfaces
    11. Show IPs of interfaces in a somewhat friendly way
  6. HTTP tricks
    1. To test if webserver is up and serving content
    2. See average memory of apache
    3. watch incoming connections to web and out from web to app
    4. open web connections
    5. List and Count HTTP Connections
    6. Same as above but using the access log
    7. See how many people are hitting a single page at once, good for spotting scrappers
    8. Show Apache Processes and cpu/mem status
    9. Find out when the apache service was started
    10. Check if Trace is enabled
  7. SSL Fun
    1. Convert the PKCS#7 file into a PEM file by typing the following command (Maybe needed for F5)
  8. Mail tricks
    1. test if mail is sending from Linux box
    2. Test if a mail server is up and open to connections
  9. MySQL
    1. MySQL cluster status
    2. MySQL replication (slave behind host)
    3. MySQL Engine upgrade
    4. Check Database size
    5. Show large queries from binlog
  10. Oracle Stuff
    1. Oracle listener error
    2. See if oracle listener is up
    3. To get listener instance status
  11. RHCS Commands
    1. Cluser Fail over (work with any RH cluster service)
  12. SAN commands
    1. rescan luns
  13. Other Odds and ends
    1. Grub kernel option for pci bus scanning compatability
    2. Show busy/wait processes
    3. List open files for a PID
    4. Show last two hours of stats from sar
    5. Remove ext3 journal and recreate
    6. directory tree permissions
    7. record filesystem ownerships and modes
  14. Dig Fun
    1. Reverse name lookup
    2. Dig name servers for information
    3. Find out the name servers for a zone
    4. Request all records for a zone from an authoritative server
    5. Look up the domain name corresponding to the IP address 172.16.118.1
  15. Reference materials (not much use)
    1. some spamassassin settings
    2. signals
    3. Virthost httpd.conf entries
    4. mail relay config using postfix
    5. Rescan SCSI bus
    6. Check for rogue “httpd” processes
    7. Check for processes whose binary doesn’t match what they claim to be

 


Summary

 

Collection of linux one liners. I will update this page occasionally.

CPU related

 

high cpu/proc queue

 

resize;clear;echo;date;echo "Top 10 Processes by CPU %";echo ""; ps -eo user,%cpu,%mem,rsz,args,pid,lstart|sort -rnk2|awk 'BEGIN {printf "%12s\t%s\t%s\t%s\t%s\n","USER","%CPU","%MEM","RSZ","COMMAND","PID","Started"}{printf "%12s\t%g'%'\t%g'%'\t%d MB\t%s\n",$1,$2,$3,$4/1024,$5}'|head -n10;echo; echo "== Last 90 mins ==";echo;sar|head -n3;sar -u|tail -n10;echo;sar -q|head -n3;sar -q|tail -n10;echo;echo "== Current 5 Second Intervals ==";echo;sar -u 5 12;echo;sar -q 5 12

 

Memory related

 

low memory

 

resize;clear;echo;date;echo; vmstat -a -S m|tail -n1|awk 'BEGIN {FS=" "}{printf "\nAvail\tActive\tTotal\tPercent Avail\n%sMB\t%sMB\t%sMB\t%s\n\n",$4+$5,$6,$4+$5+$6,($4+$5)/($4+$5+$6)*100}';echo;echo "Top 10 Processes by MEM %";echo;ps -eo user,%cpu,%mem,rsz,args|sort -rnk4|awk 'BEGIN {printf "%8s\t%6s\t%6s\t%8s\t%s\n","USER","%CPU","%MEM","RSZ","COMMAND"}{printf "%8s\t%6s\t%6s\t%8s MB\t%-12s\n",$1,$2,$3,$4/1024,$5}'|head -n10; echo ""; echo "== Last 90 Minutes ==";echo; sar -r|head -n3; sar -r|tail -n10;echo; sar -B|head -n3; sar -B|tail -n10;echo

 

low memory (ignore process “inactive memory” like nimbus)

 

resize;clear;echo;date;echo; free -m | egrep '(Mem|-)' | tr -d '\n' | awk 'BEGIN {FS=" "}{printf "\nAvail\tActive\tTotal\tPercent Avail\n%sMB\t%sMB\t%sMB\t%s\n\n",$10,$9,$2,($10/$2)*100}';echo;echo "Top 10 Processes by MEM %";echo;ps -eo user,%cpu,%mem,rsz,args|sort -rnk4|awk 'BEGIN {printf "%8s\t%6s\t%6s\t%8s\t%s\n","USER","%CPU","%MEM","RSZ","COMMAND"}{printf "%8s\t%6s\t%6s\t%8s MB\t%-12s\n",$1,$2,$3,$4/1024,$5}'|head -n10; echo ""; echo "== Last 90 Minutes ==";echo; sar -r|head -n3; sar -r|tail -n10;echo; sar -B|head -n3; sar -B|tail -n10;echo

 

Drop memory caches

 

This may or may not help depending on what is taking up memory. If you cat /proc/meminfo and slab is taking up a considerable amount of RAM (nfs_inode_cache and such as seen in slabtop), you can bump up /proc/sys/vm/vfs_cache_pressure to have the linux kernel reclaim slab more quickly.

echo 1 > /proc/sys/vm/drop_caches

 

Very Detailed vmstat on memory

 

vmstat -a -s -S m

 

Paging and Swapping levels (adjustable interval)

 

T=5; ( vmstat -s; sleep $T; vmstat -s ) | awk -v T="$T" -F'[(/)]' '/pages paged in/{ pgpio=pgpin; pgpin=$1 } /pages paged out/{ pgpoo=pgpon; pgpon=$1 } /pages swapped in/{ pgsio=pgsin; pgsin=$1 } /pages swapped out/{ pgsoo=pgson; pgson=$1 } END { printf "== Current %i second intervals ==\nPaged in:    %6.0f = %6.3f KB/s\nPaged out:   %6.0f = %6.3f KB/s\nSwapped in: %6.0f = %6.3f KB/s\nSwapped out: %6.0f = %6.3f KB/s\n", T, pgpin-pgpio, ((pgpin-pgpio)*4)/T, pgpon-pgpoo, ((pgpon-pgpoo)*4)/T, pgsin-pgsio, ((pgsin-pgsio)*4)/T, pgson-pgsoo, ((pgson-pgsoo)*4)/T}';

 

Percentage of memory used

 

MEMU=`free -m | egrep "(Mem|-)" | tr -d '\n' | awk '{printf "%s/%s\n",$9*100,$2}' | bc`; echo -e "\n$MEMU% USED\n"

 

Past two hours of memory usage from sar (may need modification if sar output in a language other than English)

 

memtot=`free -m | awk '$1 ~ /Mem/ {print $2}'`; echo -e "\nTime\t\tPercent\n";sar -r -s `date --date "-2 hours" +%T`|tail -n +4 |awk -v memtot="$memtot" '{printf "%s %s %8d % \n",$1,$2,(($4-$6-$7)/1024)*100/memtot}'|grep -v Average

 

Daily memory utilization percentage from sar (again may change depending on language/sar output)

 

cd /var/log/sa; for safile in `ls -tr sa[0-9]*`;do memtot=`free -m | awk '$1 ~ /Mem/ {print $2}'`;sadate=`sar -r -f /var/log/sa/$safile | head -1 |awk '{print $4}'`; sar -r -f /var/log/sa/$safile |tail -n +4 |awk -v memtot="$memtot" '{printf "%s %s %8d % \n",$1,$2,(($4-$6-$7)/1024)*100/memtot}'|grep -v Average | grep "00:01" |awk -v sadate="$sadate" '{SUM+=$3;TOT+=1} END { printf "%s Daily Average %8d %\n",sadate,SUM/TOT}';done

 

Calculate memory usage by adding up usage in /proc

 

cd /proc; MYSUM=0;for num in `ls -d [0-9]*`;do MEM=`cat ./$num/status 2>/dev/null | grep VmRSS | awk '$1 ~ /VmRSS/ {print $2}'`;MYSUM=$((MYSUM+MEM));done;echo $MYSUM | awk '{printf "%s MB\n",$1/1024}'

 

Add Shared memory

 

ipcs -m | tail -n +4 | awk '{SUM += $5} END {printf"%8d MB\n", SUM/1024/1024}'

 

Disk space related

 

low disk (current directory, takes into account sparse files)

 

FS='./';resize;clear;date;df -h $FS; echo "Largest Directories:"; du -hcx --max-depth=2 $FS 2>/dev/null | grep [0-9]G | sort -grk 1 | head -15 ;echo "Largest Files:"; nice -n 19 find $FS -mount -type f -print0 2>/dev/null| xargs -0 du -k | sort -rnk1| head -n20 |awk '{printf "%8d MB\t%s\n",($1/1024),$NF}'

 

low disk (current directory)

 

FS='./';resize;clear;date;df -h $FS; echo "Largest Directories:"; du -hcx --max-depth=2 $FS 2>/dev/null | grep [0-9]G | head -15 | sort -rk 1;echo "Largest Files:"; nice -n 19 find $FS -mount -type f -ls 2>/dev/null| sort -rnk7| head -n20 |awk '{printf "%8d MB\t%s\n",($7/1024)/1024,$NF}'

 

low disk (current directory, w/top 20 on largest directories)

 

FS='./';resize;clear;date;df -h $FS; echo "Largest Directories:"; nice -n19 find $FS -mount -type d -print0 2>/dev/null|xargs -0 du -k|sort -runk1|head -n20|awk '{printf "%8d MB\t%s\n",($1/1024),$NF}';echo "Largest Files:"; nice -n 19 find $FS -mount -type f -print0 2>/dev/null| xargs -0 du -k | sort -rnk1| head -n20 |awk '{printf "%8d MB\t%s\n",($1/1024),$NF}';

 

disk usage

 

du -hcx --max-depth=5 | grep [0-9]G

or sorted with largest dir on top:

du -hcx --max-depth=5 | grep [0-9]G | sort -n -r

 

largest files in a directory (in MB)

 

du -sm * | sort -nr | head -10

 

Find the largest files in a filesystem

 

find . -type f -exec du -h {} \; | sort -nrk 1 | head

 

Compress files in current directory, older than 24 hours

 

find . -type f -atime +24 -exec gzip '{}' \;

 

Find files last modified 2 days ago

 

find /etc/httpd -name *.log -mtime -2 -print

 

Remove files older than 7 days

 

find ./ -maxdepth 1 -type f -mtime +7 -exec rm -rf {} \;

 

Find percentage of blocks that aren’t reserved for root (usually 95)

 

for DEV in `df | egrep '^/dev' | awk '{print $1}'`; do echo -n "$DEV =>  "; echo `dumpe2fs -h $DEV 2>/dev/null | grep 'lock count' | tr -t '\n' ' ' | awk '{printf "100*%s/%s", $3-$7, $3}'` | bc; done

 

Netstat and Networking Fun

 

General netstat

 

netstat -anpt | egrep '^Proto|ESTAB'

netstat -anpt | sort -k3 -rn | head

 

General netstat command to show all ports

 

netstat -antp

 

Show local listening TCP/UDP ports, and the process they belong to

 

netstat -tulpn

 

See how many connections each service has

 

netstat -np | awk '{print $7}' | awk -F/ '{count[$2]++}END{for(j in count) print count[j],j}' | sort -nr

 

Ton of network statistic info

 

netstat -sw

 

Get MAC Addresses from dmesg

 

dmesg | egrep -i "^eth.*\b(([a-f0-9]{2}:){5}[a-f0-9]{2}|[0-9a-f]{12})\b" | sed -r 's/([a-f0-9]{2})([a-f0-9]{2})([a-f0-9]{2})([a-f0-9]{2})([a-f0-9]{2})([a-f0-9]{2})/\1:\2:\3:\4:\5:\6/i'"

 

Alternative (Only shows in the form of eth2: 00:1B:21:9C:2C:15):

dmesg | sed -nr "s/^(eth[0-9]{1,2}).*\b(([a-f0-9]{2}:){5}[a-f0-9]{2}|[0-9a-f]{12})\b/\1: \2/ip" | sed -r 's/([a-f0-9]{2})([a-f0-9]{2})([a-f0-9]{2})([a-f0-9]{2})([a-f0-9]{2})([a-f0-9]{2})/\1:\2:\3:\4:\5:\6/i'

 

Here’s one a little shorter, outputs uppercase and may or may not have colon separators

dmesg | egrep -o "eth[0-9].*([a-f0-9]{2}:?){5}[a-f0-9]{2}" | awk '{print $1" " toupper($NF)}'

 

Automatically Update MAC addresses in ifcfg scripts

 

Using dmseg

 

for ETH in `ifconfig -a | grep '^e' | awk '{print $1}'`; do MAC=`dmesg | sed -nr "s/^$ETH.*\b(([a-f0-9]{2}:){5}[a-f0-9]{2}|[0-9a-f]{12})\b/\1/ip" | sed -r 's/([a-f0-9]{2})([a-f0-9]{2})([a-f0-9]{2})([a-f0-9]{2})([a-f0-9]{2})([a-f0-9]{2})/\1:\2:\3:\4:\5:\6/i' | tr [:lower:] [:upper:]`; sed -i "s/^HWADDR.*/HWADDR=$MAC/g" /etc/sysconfig/network-scripts/ifcfg-$ETH; done;

 

Using lshw, ethtool and proc

 

for ETH in `ifconfig -a | grep '^e' | awk '{print $1}'`; do VENDOR=`lshw -class network | grep $ETH -B 3 | sed -n 's/.*vendor:\ \(.*\).*/\1/p'`; if [ "$VENDOR" == "Intel Corporation" ]; then BOND=`ip addr | sed -n "s/[0-9]*:\ $ETH:.*\(bond[0-9:]*\).*/\1/p"`; if [ $BOND ]; then MAC=`cat /proc/net/bonding/$BOND | grep Slave\ Interface:\ $ETH -A 3 | sed -n 's/Permanent\ HW\ addr:\ \(.*\)/\1/p' | tr [:lower:] [:upper:]`; else MAC=`ethtool -e $ETH length 320 | grep '0x0000' | awk '{print $2":"$3":"$4":"$5":"$6":"$7}' | tr [:lower:] [:upper:]`; fi; elif [ "$VENDOR" == "Broadcom Corporation" ]; then MAC=`ethtool -e $ETH length 320 | grep '0x0130' | awk '{print $8":"$9":"$10":"$11":"$12":"$13}' | tr [:lower:] [:upper:]`; else echo "$ETH: UNKNOWN VENDOR"; exit 1; fi; sed -i "s/^HWADDR.*/HWADDR=$MAC/g" /etc/sysconfig/network-scripts/ifcfg-$ETH; done;

 

Compare MAC Addresses between actual and ifcfg scripts

 

Using dmesg

 

for ETH in `ifconfig -a | grep '^e' | awk '{print $1}'`; do MACDMESG=`dmesg | sed -nr "s/^$ETH.*\b(([a-f0-9]{2}:){5}[a-f0-9]{2}|[0-9a-f]{12})\b/\1/ip" | sed -r 's/([a-f0-9]{2})([a-f0-9]{2})([a-f0-9]{2})([a-f0-9]{2})([a-f0-9]{2})([a-f0-9]{2})/\1:\2:\3:\4:\5:\6/i' | tr [:lower:] [:upper:]`; MACIFCFG=`sed -nr "s/^HWADDR=(.*)$/\1/p" /etc/sysconfig/network-scripts/ifcfg-$ETH | tr [:lower:] [:upper:]`; if [ "$MACIFCFG" == "$MACDMESG" ]; then echo $ETH: MATCH; else echo -e "$ETH: NO MATCH\n\tifcfg: $MACIFCFG\n\tdmesg: $MACDMESG"; fi; done;

 

Using lshw, ethtool and proc

 

for ETH in `ifconfig -a | grep '^e' | awk '{print $1}'`; do VENDOR=`lshw -class network | grep $ETH -B 3 | sed -n 's/.*vendor:\ \(.*\).*/\1/p'`; MACIFCFG=`sed -nr "s/^HWADDR=(.*)$/\1/p" /etc/sysconfig/network-scripts/ifcfg-$ETH | tr [:lower:] [:upper:]`; if [ "$VENDOR" == "Intel Corporation" ]; then KEY='0x0000'; BOND=`ip addr | sed -n "s/[0-9]*:\ $ETH:.*\(bond[0-9:]*\).*/\1/p"`; if [ $BOND ]; then REALMAC=`cat /proc/net/bonding/$BOND | grep Slave\ Interface:\ $ETH -A 3 | sed -n 's/Permanent\ HW\ addr:\ \(.*\)/\1/p' | tr [:lower:] [:upper:]`; else REALMAC=`ethtool -e $ETH length 320 | grep $KEY | awk '{print $2":"$3":"$4":"$5":"$6":"$7}' | tr [:lower:] [:upper:]`; fi; elif [ "$VENDOR" == "Broadcom Corporation" ]; then KEY='0x0130'; REALMAC=`ethtool -e $ETH length 320 | grep $KEY | awk '{print $8":"$9":"$10":"$11":"$12":"$13}' | tr [:lower:] [:upper:]`; else echo "$ETH: UNKNOWN VENDOR"; exit 1; fi; if [ "$MACIFCFG" == "$REALMAC" ]; then echo $ETH: MATCH; else echo -e "$ETH: NO MATCH\n\tConfigured: $MACIFCFG\n\tCorrect:    $REALMAC"; fi; done;

 

TCP dump examples

 

tcpdump -i eth0 -nn host XXX.XXX.XXX.XXX and port 111 -c 1000

tcpdump -C 10 -W 50 -nn -U -i eth0 -w /home/tcpdump/tcpdump.pcap -Z &

    Basic communication // see the basics without many options

    # tcpdump -nS

    Basic communication (very verbose) // see a good amount of traffic, with verbosity and no name help

    # tcpdump -nnvvS

    A deeper look at the traffic // adds -X for payload but doesn't grab any more of the packet

    # tcpdump -nnvvXS

    Heavy packet viewing // the final "s" increases the snaplength, grabbing the whole packet

    # tcpdump -nnvvXSs 1514
Here's a capture of exactly two (-c2) ICMP packets (a ping and pong) using some of the options described above. Notice how much we see about each packet.

hermes root # tcpdump -nnvXSs 1514 -c2 icmp
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 1514
bytes 23:11:10.370321 IP (tos 0x20, ttl  48, id 34859, offset 0, flags
[none], length: 84) 69.254.213.43 > 72.21.34.42: icmp 64: echo request seq 0

        0x0000:  4520 0054 882b 0000 3001 7cf5 45fe d52b  E..T.+..0.|.E..+
        0x0010:  4815 222a 0800 3530 272a 0000 25ff d744  H."*..50'*..%..D
        0x0020:  ae5e 0500 0809 0a0b 0c0d 0e0f 1011 1213  .^..............
        0x0030:  1415 1617 1819 1a1b 1c1d 1e1f 2021 2223  .............!"#
        0x0040:  2425 2627 2829 2a2b 2c2d 2e2f 3031 3233  $%&'()*+,-./0123
        0x0050:  3435 3637                                4567
23:11:10.370344 IP (tos 0x20, ttl  64, id 35612, offset 0, flags [none],
length: 84) 72.21.34.42 > 69.254.213.43: icmp 64: echo reply seq 0
        0x0000:  4520 0054 8b1c 0000 4001 6a04 4815 222a  [email protected]"*
        0x0010:  45fe d52b 0000 3d30 272a 0000 25ff d744  E..+..=0'*..%..D
        0x0020:  ae5e 0500 0809 0a0b 0c0d 0e0f 1011 1213  .^..............
        0x0030:  1415 1617 1819 1a1b 1c1d 1e1f 2021 2223  .............!"#
        0x0040:  2425 2627 2829 2a2b 2c2d 2e2f 3031 3233  $%&'()*+,-./0123
        0x0050:  3435 3637                                4567
2 packets captured
2 packets received by filter
0 packets dropped by kernel
hermes root #

Expressions

Expressions allow you to trim out various types of traffic and find exactly what you're looking for. Mastering the expressions and learning to combine them creatively is what makes one truly powerful with tcpdump. There are three main types of expression: type, dir, and proto.

Type options are host, net, and port. Direction is indicated by dir, and there you can have src, dst, src or dst, and src and dst. Here are a few that you should definitely be comfortable with:

    host // look for traffic based on IP address (also works with hostname if you're not using -n)

    # tcpdump host 1.2.3.4

    src, dst // find traffic from only a source or destination (eliminates one side of a host conversation)

    # tcpdump src 2.3.4.5
    # tcpdump dst 3.4.5.6

    net // capture an entire network using CIDR notation

    # tcpdump net 1.2.3.0/24

    proto // works for tcp, udp, and icmp. Note that you don't have to type proto

    # tcpdump icmp

    port // see only traffic to or from a certain port

    # tcpdump port 3389

    src, dst port // filter based on the source or destination port

    # tcpdump src port 1025
    # tcpdump dst port 389

    src/dst, port, protocol // combine all three

    # tcpdump src port 1025 and tcp
    # tcpdump udp and src port 53

You also have the option to filter by a range of ports instead of declaring them individually, and to only see packets that are above or below a certain size.

    Port Ranges // see traffic to any port in a range
    tcpdump portrange 21-23

    Packet Size Filter // only see packets below or above a certain size (in bytes)
    tcpdump less 32
    tcpdump greater 128

    [ You can use the symbols for less than, greater than, and less than or equal / greater than or equal signs as well. ]
    // filtering for size using symbols
    tcpdump > 32
    tcpdump <= 128

Writing to a File

tcpdump allows you to send what you're capturing to a file for later use using the -w option, and then to read it back using the -r option. This is an excellent way to capture raw traffic and then run it through various tools later.

The traffic captured in this way is stored in tcpdump format, which is pretty much universal in the network analysis space. This means it can be read in by all sorts of tools, including Wireshark, Snort, etc.

Capture all Port 80 Traffic to a File

# tcpdump -s 1514 port 80 -w capture_file

Then, at some point in the future, you can then read the traffic back in like so:

Read Captured Traffic back into tcpdump

# tcpdump -r capture_file
Getting Creative

Expressions are nice, but the real magic of tcpdump comes from the ability to combine them in creative ways in order to isolate exactly what you're looking for. There are three ways to do combinations, and if you've studied computers at all they'll be pretty familar to you:

    AND
    and or &&
    OR
    or or ||
    EXCEPT
    not or !

More Examples

# TCP traffic from 10.5.2.3 destined for port 3389

tcpdump -nnvvS and src 10.5.2.3 and dst port 3389

# Traffic originating from the 192.168 network headed for the 10 or 172.16 networks

tcpdump -nvX src net 192.168.0.0/16 and dst net 10.0.0.0/8 or 172.16.0.0/16

# Non-ICMP traffic destined for 192.168.0.2 from the 172.16 network

tcpdump -nvvXSs 1514 dst 192.168.0.2 and src net and not icmp

# Traffic originating from Mars or Pluto that isn't to the SSH port

tcpdump -vv src mars and not dst port 22

As you can see, you can build queries to find just about anything you need. The key is to first figure out precisely what you're looking for and then to build the syntax to isolate that specific type of traffic.
Grouping

Also keep in mind that when you're building complex queries you might have to group your options using single quotes. Single quotes are used in order to tell tcpdump to ignore certain special characters -- in this case the "( )" brackets. This same technique can be used to group using other expressions such as host, port, net, etc. Take a look at the command below:

# Traffic that's from 10.0.2.4 AND destined for ports 3389 or 22 (incorrect)

tcpdump src 10.0.2.4 and (dst port 3389 or 22)

If you tried to run this otherwise very useful command, you'd get an error because of the parenthesis. You can either fix this by escaping the parenthesis (putting a \ before each one), or by putting the entire command within single quotes:

# Traffic that's from 10.0.2.4 AND destined for ports 3389 or 22 (correct)

tcpdump 'src 10.0.2.4 and (dst port 3389 or 22)'

Advanced

You can also filter based on specific portions of a packet, as well as combine multiple conditions into groups. The former is useful when looking for only SYNs or RSTs, for example, and the latter for even more advanced traffic isolation.

[ Hint: An anagram for the TCP flags: Unskilled Attackers Pester Real Security Folk ]

Show me all URGENT (URG) packets...

# tcpdump 'tcp[13] & 32!=0'

Show me all ACKNOWLEDGE (ACK) packets...

# tcpdump 'tcp[13] & 16!=0'

Show me all PUSH (PSH) packets...

# tcpdump 'tcp[13] & 8!=0'

Show me all RESET (RST) packets...

# tcpdump 'tcp[13] & 4!=0'

Show me all SYNCHRONIZE (SYN) packets...

# tcpdump 'tcp[13] & 2!=0'

Show me all FINISH (FIN) packets...

# tcpdump 'tcp[13] & 1!=0'

Show me all SYNCHRONIZE/ACKNOWLEDGE (SYNACK) packets...

# tcpdump 'tcp[13] & =18'

[ Note: Only the PSH, RST, SYN, and FIN flags are displayed in tcpdump's flag field output. URGs and ACKs are displayed, but they are shown elsewhere in the output rather than in the flags field ]

Keep in mind the reasons these filters work. The filters above find these various packets because tcp[13] looks at offset 13 in the TCP header, the number represents the location within the byte, and the !=0 means that the flag in question is set to 1, i.e. it's on.

As with most powerful tools, however, there are multiple ways to do things. The example below shows another way to capture packets with specific TCP flags set.

Capture TCP Flags Using the tcpflags Option...

# tcpdump 'tcp[tcpflags] & & tcp-syn != 0'
Specialized Traffic

Finally, there are a few quick recipes you'll want to remember for catching specific and specialized traffic, such as IPv6 and malformed/likely-malicious packets.

IPv6 traffic

# tcpdump ip6

Packets with both the RST and SYN flags set (why?)

# tcpdump 'tcp[13] = 6'

Traffic with the 'Evil Bit' Set

# tcpdump 'ip[6] & 128 != 0'

 

Fail over and test Network interfaces

 

ifenslave -c bond0 eth1 && ping -c5 `route -n | awk '/^0.0.0.0/{print $2}'`;\

ifenslave -c bond0 eth0 && ping -c5 `route -n | awk '/^0.0.0.0/{print $2}'`;\

tail -25 /var/log/messages | grep bonding && cat /proc/net/bonding/bond*

 

Show IPs of interfaces in a somewhat friendly way

 

hname=`hostname -s`;ifconfig | egrep -B1 "inet addr" | grep -v -B1 "127.0.0.1" |awk ' { if (/^eth|^bond/) print $1; else if(/inet addr/) print "- " $2 }' | sed ':a;N;$!ba;s/\n/ /g;s/addr://g;s/eth/\n'"$hname - "'eth/g;s/bond/\n'"$hname - "'bond/g;'

 

HTTP tricks

 

To test if webserver is up and serving content

 

echo -e "GET / HTTP/1.1\n\n" | nc -v -w 5 www.google.com 80 | head -9

 

See average memory of apache

 

ps aux | grep ^apache | awk '{sum += $6} END {print "Average HTTP RSS: " sum/1024/NR"MB"}'

 

watch incoming connections to web and out from web to app

 

netstat -pnta | grep :81  | grep 68.16.26.91 | grep -v TIME | wc -l

 

open web connections

 

netstat -plnant | grep ":80" | grep -i established

 

List and Count HTTP Connections

 

netstat -plant|grep :80|awk '{print $5}'|cut -d: -f1|sort|uniq -c|sort -n

 

Same as above but using the access log

 

sudo cat /var/log/httpd/access_log | tail -2500 | awk '{print $1}' | sort -rn | uniq -c | sort -rn | head -40

 

See how many people are hitting a single page at once, good for spotting scrappers

 

cat /var/log/httpd/access_log | tail -2500 | grep "GET /profile" | awk '{print $1}' | sort -rn | uniq -c | sort -rn | head -40

 

Show Apache Processes and cpu/mem status

 

echo "Apache Processes: " `ps xuaw |egrep -i '[0-9] /usr/sbin/httpd ' -c` && iostat 2 2 | grep -i ^avg-cpu -A 1|tail -n 2 && free -m

 

Find out when the apache service was started

 

ps axo user,lstart,cmd|awk '!/awk/&&/httpd/{if($1~/root/)print}'

 

Check if Trace is enabled

 

replace vhosts.80 with whatever file has the vhosts

cat vhosts.80 | awk -F: '{printf"%s %s\n",$1,$2}' | while read ip port; do echo -e "\n\n-----Checking $ip $port-----\n\n"; echo -e "TRACE / HTTP/1.0\nHost: foo\nA: b\nC: d\n\n" | nc $ip $port; done

 

SSL Fun

 

Convert the PKCS#7 file into a PEM file by typing the following command (Maybe needed for F5)

 

openssl pkcs7 -in <PKCSfile>.p7b -text -out <filename>.pem -print_certs

 

Mail tricks

 

test if mail is sending from Linux box

 

echo "test1" | mail -s "test1" [email protected] "

 

Test if a mail server is up and open to connections

 

echo "HELO" | nc -v -w 5 localhost 25

 

MySQL

 

MySQL cluster status

 

mysql -e "show processlist;"

clustat

 

MySQL replication (slave behind host)

 

echo "show slave status\G"|mysql (-S /var/lib/mysql-db3/mysql.sock) | grep Seconds_Seconds_Behind_Master

 

MySQL Engine upgrade

 

Check health

 

Log into db and verify that it is still active via "clustat"

 

Make a backup of the existing data

 

mysqldump --databases Fulfillment --tables scandata | gzip --fast > /root/Fulfillment-scandata.sql.gz

 

Convert the table

 

ALTER TABLE Fulfillment.scandata ENGINE='INNODB'

 

Confirm conversion

 

SHOW CREATE TABLE Fulfillment.scandata\G

 

To confirm conversion

 

SHOW CREATE TABLE Fulfillment.scandata\G

 

Watch what mysql is doing every second

 

watch -n 1 "mysql -se 'show processlist' | grep -v sleep -i"

 

Check Database size

 

If you run the query which is given below in MySQL Query Browser then you will get the two columns first will display the Data Base Name and the> second will display the Data Base Size in MB.

 

SELECT table_schema "Data Base Name", sum( data_length + index_length ) / 1024 / 1024 "Data Base Size in MB" FROM information_schema.TABLES GROUP BY table_schema ;

 

If you have question in your mind like “How to view the Free space available for my Data Base in MySQL”, then run the below query

 

SELECT table_schema "Data Base Name", sum( data_length + index_length ) / 1024 / 1024 "Data Base Size in MB", sum( data_free )/ 1024 / 1024 "Free Space in MB" FROM information_schema.TABLES GROUP BY table_schema;

 

Database size and Free space in GB

 

SELECT table_schema "Data Base Name", sum( data_length + index_length ) / 1024 / 1024 / 1024 "Data Base Size in GB", sum( data_free )/ 1024 / 1024 / 1024 "Free Space in GB" FROM information_schema.TABLES GROUP BY table_schema;

 

The same information from the command prompt (may help scripters)

 

mysql -e "SELECT table_schema 'Data Base Name', sum( data_length + index_length ) / 1024 / 1024 / 1024 'Data Base Size in GB', sum( data_free )/ 1024 / 1024 / 1024 'Free Space in GB' FROM information_schema.TABLES GROUP BY table_schema";

 

Random commands that don’t appear to be linked to any title

 

Create DB:
create database DB_NAME;
GRANT ALL ON DB_NAME.* TO [email protected] IDENTIFIED BY "Password";

 

Show large queries from binlog

 

Following selects lines with > 4000 characters and outputs 100 characters of it:

mysqlbinlog /path/to/binlog | while read line;do curline=`echo $line|wc -c`; if [ "$curline" -gt "4000" ];then data=`echo "$line" | cut -c1-100`; echo "length: $curline Bytes - query: $data";fi; done

 

Oracle Stuff

 

Oracle listener error

 

From alert log ....

08-NOV-2010 18:52:21 * service_died * ODS2 * 12547

TNS-12547: TNS:lost contact

 

See if oracle listener is up

 

su - oracle
lsnrctl status

 

To get listener instance status

 

lsnrctl status orcl (replace orcl with instance name)
lsnrctl services
tnsping <service name>

 

RHCS Commands

 

Cluser Fail over (work with any RH cluster service)

 

'clustat' ~= Will show the status of the cluster

'clusvcadm -R mysql-svc' ~= Will restart the mysql service in place on the same server

'clusvcadm -r mysql-svc -m <node name>' ~= Will relocate the mysql service to that node

'clusvcadm -d mysql-svc' ~= Will disable the mysql service

'clusvcadm -e mysql-svc' ~= Will enable the mysql service

 

SAN commands

 

rescan luns

 

For RHEL5 boxes, the rescan commands are as follows

 

(Do Not do this on RHEL4 outside of a planned maintenance – there is risk of kernel panic)

 

1. Run powermt display dev=all to determine the HBA numbers. The HBA numbers are the first item in each path entry and are to the left of the "qla2xxx"

2. Run echo "- - -" > /sys/class/scsi_host/host#/scan for each HBA (substituting "#" with the appropriate HBA number)

Warning: DO NOT run an issue_lip. This is unnecessary and will likely cause a DB outage and/or kernel panic.

3. Run powermt display dev=all

Verify that new LUN is present

4. powermt update lun_names

5. Run powermt save

to make sure the new lun is saved.

echo "- - -" > /sys/class/scsi_host/host1/scan

echo "- - -" > /sys/class/scsi_host/host2/scan

emcpdiscover

Then run

powermt display dev=all

 

Other Odds and ends

Grub kernel option for pci bus scanning compatability

 

pci=nobfsort

 

Show busy/wait processes

 

ps -el | grep -P '^\d D'

 

List open files for a PID

 

lsof -p 9331

 

Show last two hours of stats from sar

 

sar -B -s `date --date "-2 hours" +%T`

 

Remove ext3 journal and recreate

 

(EXT3-fs error (device hda3) in start_transaction: Journal has aborted)

tune2fs -O ^has_journal /dev/hda3

e2fsck /dev/hda3

tune2fs -j /dev/hda3

 

directory tree permissions

 

namei -m /path/to/file/or/directory

 

record filesystem ownerships and modes

 

find / -wholename '/proc' -prune -o -fprintf /root/perms-backup.`date +%Y%m%d`.txt "chmod %m '%p'\nchown %u:%g '%p'\n"

 

Dig Fun

 

Reverse name lookup

 

dig +trace -x 207.97.209.147  | grep PTR

 

Dig name servers for information

 

dig . ns

 

Find out the name servers for a zone

 

dig @server domain ns

 

Request all records for a zone from an authoritative server

 

dig @server domain axfr

NOTE: This command requires a zone transfer which the server may disallow.

 

Look up the domain name corresponding to the IP address 172.16.118.1

 

dig -x 172.16.118.1

 

Reference materials (not much use)

 

some spamassassin settings

 

local.cf                 spamassassin-default.rc  spamassassin-spamc.rc    v312.pre

[[email protected] ~]# cat /etc/mail/spamassassin/local.cf

# How many hits before a message is considered spam.

#required_score           6

# 12/2 request done by jerry

#require_score            5.7

# 12/6 request done by jerry

#require_score            5.5

# 12/7 request done by Laron

require_score            5.3

# Change the subject of suspected spam

rewrite_header subject         [SPAM]

# Encapsulate spam in an attachment (0=no, 1=yes, 2=safe)

report_safe             1

# Enable the Bayes system

use_bayes               1

# Enable Bayes auto-learning

bayes_auto_learn              1

# Enable or disable network checks

skip_rbl_checks         0

use_razor2              0

use_dcc                 0

use_pyzor               0

# Mail using languages used in these country codes will not be marked

# as being possibly spam in a foreign language.

ok_languages            all

# Mail using locales used in these country codes will not be marked

# as being possibly spam in a foreign language.

ok_locales              all

# Ignore FH_DATE_PAST_20XX (was causing false positives)

score FH_DATE_PAST_20XX 0.0

 

signals

1) HUP

14) ALRM

27) MSG

40) bad trap

53) bad trap

2) INT

15) TERM

28) WINCH

41) bad trap

54) bad trap

3) QUIT

16) URG

29) PWR

42) bad trap

55) bad trap

4) ILL

17) STOP

30) USR1

43) bad trap

56) bad trap

5) TRAP

18) TSTP

31) USR2

44) bad trap

57) bad trap

6) ABRT

19) CONT

32) PROF

45) bad trap

58) bad trap

7) EMT

20) CHLD

33) DANGER

46) bad trap

59) CPUFAIL

8) FPE

21) TTIN

34) VTALRM

47) bad trap

60) GRANT

9) KILL

22) TTOU

35) MIGRATE

48) bad trap

61) RETRACT

10) BUS

23) IO

36) PRE

49) bad trap

62) SOUND

11) SEGV

24) XCPU

37) bad trap

50) bad trap

63) SAK

12) SYS

25) XFSZ

38) bad trap

51) bad trap

13) PIPE

26) bad trap

39) bad trap

52) bad trap

 

Virthost httpd.conf entries

 

#############################################################
<VirtualHost 192.168.1.1:80>
ServerName xxx.example.com
ServerAlias www.xxx.example.com xxx.example
DocumentRoot "/var/www/vhosts/xxx.example.com"
<Directory /> Options FollowSymLinks
AllowOverride None
</Directory>
<Directory "/var/www/vhosts/xxx.example.com">
Options FollowSymLinks
AllowOverride All
Order allow,deny
Allow from all
</Directory>
ErrorLog logs/xxx.example.com_error_log
CustomLog logs/xxx.example.com_log common
</VirtualHost>

<VirtualHost 192.168.1.1:443>
ServerName xxx.example.com
ServerAlias www.xxx.example.com
DocumentRoot "/var/www/vhosts/xxx.example.com"
<Directory />
Options FollowSymLinks
AllowOverride None
</Directory>
<Directory "/var/www/vhosts/xxx.example.com">
Options FollowSymLinks
AllowOverride All
Order allow,deny
Allow from all
</Directory>
SSLEngine On
SSLCertificateFile /etc/httpd/conf/ssl.crt/xxx.example.com.crt
SSLCertificateKeyFile /etc/httpd/conf/ssl.key/xxx.example.com.key
ErrorLog logs/xxx.example.com_error_log
CustomLog logs/xxx.example.com_log common
</VirtualHost>

 

mail relay config using postfix

 

service sendmail stop

chkconfig sendmail off

chkconfig postfix on

service postfix start

on mail.xxx.com:

main.cf: added "relayhost = [10.101.2.52]"

on mx01.xxx.com:

main.cf: added "inet_interfaces = 10.101.2.52, localhost"

main.cf: added "mynetworks = 10.101.2.49/32, 127.0.0.0/8"

On mail, we told postfix to only send mail through 10.101.2.52 which is mxs01. On mx01, we had to specify postfix to listen on 10.101.2.52 (bond0 interface ip) and localhost. Then to only allow relay mail to come from my network and we specified the explicit address of mail.

 

Rescan SCSI bus

 

echo "- - -" > /sys/class/scsi_host/host0/scan

 

Check for rogue “httpd” processes

 

for pid in `ps aux | grep httpd | grep -v grep | awk '{print $2}'`; do ls -lh /proc/$pid/exe | grep -v httpd; done

 

Check for processes whose binary doesn’t match what they claim to be

 

This
command can return false positives, such as /bin/sh not matching
/bin/bash. This is ok. If the process claims to be httpd and it
returns perl, however, this is not ok.

for pid in `ps aux | grep -v '\[' | grep -v grep | awk '{print $2}' | grep -v PID`; do SERVICE=`ps aux | grep -v grep | grep " $pid " | awk '{print $11}' | egrep -v 'nimbus|delloma' | tr -d '-' | tr -d ':'`; [ "X$SERVICE" != "X" ] && ls -lh /proc/$pid | grep ' exe ' | tr -d '-' | grep -v $SERVICE >/dev/null 2>&1 && echo "$pid should be $SERVICE but it is actually `ls -lh /proc/$pid | grep ' exe ' | awk '{print $11}'`"; done

 

<this is a work in progress>