Hi Guys,
First of all thank you for for your love and support. In this article I’m trying to post my knowledge and experience about Linux server monitoring. As a system administrator it is essential to have good knowledge about monitoring weapons so you can keep your server calm and cool and can get better performance through it. so lets start::
The commands discussed below are some of the most basic commands when it comes to system analysis and debugging server issues such as:
- Finding out bottlenecks.
- Disk (storage) bottlenecks.
- CPU and memory bottlenecks.
- Network bottlenecks.
1: top – Display Linux tasks
top provides an ongoing look at processor activity in real time. It displays a listing of the most CPU-intensive tasks on the system, and can provide an interactive interface for manipulating processes. It can sort the tasks by CPU usage, memory usage and runtime. can be better configured than the standard top from the procps suite
Commonly Used Hot Keys
Friends, I always recommend to use all available tools/options with simple commands, so try below keys with “top” command.
Hot Key | Usage |
---|---|
t | Displays summary information off and on. |
m | Displays memory information off and on. |
A | Sorts the display by top consumers of various system resources. Useful for quick identification of performance-hungry tasks on a system. |
f | Enters an interactive configuration screen for top. Helpful for setting up top for a specific task. |
o | Enables you to interactively select the ordering within top. |
r | Issues renice command. |
k | Issues kill command. |
z | Turn on or off color/mono |
#2: vmstat – Report virtual memory statistics
This command reports information about the kernel threads, virtual memory, disks, traps, and CPU activity.
# vmstat 3
Sample Outputs:
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 0 2540988 522188 5130400 0 0 2 32 4 2 4 1 96 0 0 1 0 0 2540988 522188 5130400 0 0 0 720 1199 665 1 0 99 0 0 0 0 0 2540956 522188 5130400 0 0 0 0 1151 1569 4 1 95 0 0 0 0 0 2540956 522188 5130500 0 0 0 6 1117 439 1 0 99 0 0 0 0 0 2540940 522188 5130512 0 0 0 536 1189 932 1 0 98 0 0 0 0 0 2538444 522188 5130588 0 0 0 0 1187 1417 4 1 96 0 0 0 0 0 2490060 522188 5130640 0 0 0 18 1253 1123 5 1 94 0 0
See Memory Utilization Slabinfo
# vmstat -m
Cache Num Total Size Pages ip_conntrack_expect 0 0 136 28 ip_conntrack 8048 11856 304 13 ip_fib_alias 50 118 64 59 ip_fib_hash 42 118 64 59 iser_descriptors 0 0 128 30 ib_mad 0 0 448 8 jbd_1k 0 0 1024 4 bio_map_info 100 105 1064 7 dm_mpath 0 0 1064 7 jbd_4k 26 48 4096 1 dm_uevent 0 0 2608 3 dm_tio 0 0 24 144 dm_io 0 0 40 92 scsi_cmd_cache 143 170 384 10 sgpool-128 32 34 4096 1
#3: w – Display Logged in Sessions
we command displays information about the users currently on the machine, and their processes.
root@linuxserverguide [~]# w 09:17:33 up 18 days, 20:50, 1 user, load average: 2.57, 2.61, 2.45 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT root pts/1 192.168.1.1 06:42 0.00s 0.06s 0.00s w
#4: uptime – Display system uptime
The uptime command can be used to see how long the server has been running. The current time, how long the system has been running, how many users are currently logged on, and the system load averages for the past 1, 5, and 15 minutes.
root@linuxserverguide [~]# uptime
18:02:41 up 41 days, 23:42, 1 user, load average: 0.00, 0.00, 0.00
1 can be considered as optimal load value. The load can change from system to system. For a single CPU system 1 – 3 and
SMP systems 6-10 load value might be acceptable.
#5: ps – Display Process Stats
ps command will report a snapshot of the current processes. To select all processes use the -A or -e option:
root@linuxserverguide [~]# ps -A
Sample Outputs:
PID TTY TIME CMD 1 ? 00:00:02 init 2 ? 00:00:02 migration/0 3 ? 00:00:01 ksoftirqd/0 4 ? 00:00:00 watchdog/0 5 ? 00:00:00 migration/1 6 ? 00:00:15 ksoftirqd/1 .... ..... 4881 ? 00:53:28 java 4885 tty1 00:00:00 mingetty 4886 tty2 00:00:00 mingetty 4887 tty3 00:00:00 mingetty 4888 tty4 00:00:00 mingetty 4891 tty5 00:00:00 mingetty 4892 tty6 00:00:00 mingetty 4893 ttyS1 00:00:00 agetty 12853 ? 00:00:00 cifsoplockd 12854 ? 00:00:00 cifsdnotifyd 14231 ? 00:10:34 lighttpd 14232 ? 00:00:00 php-cgi 54981 pts/0 00:00:00 vim 55465 ? 00:00:00 php-cgi 55546 ? 00:00:00 bind9-snmp-stat 55704 pts/1 00:00:00 ps
ps is just like top but provides more information.
Show Long Format Output
root@linuxserverguide [~]# ps -Al
F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD 4 S 0 1 0 0 75 0 - 2592 - ? 00:00:00 init 1 S 0 2 1 0 -40 - - 0 migrat ? 00:00:02 migration/0 1 S 0 3 1 0 94 19 - 0 ksofti ? 00:00:01 ksoftirqd/0 5 S 0 4 1 0 -40 - - 0 watchd ? 00:00:00 watchdog/0 1 S 0 5 1 0 -40 - - 0 migrat ? 00:00:01 migration/1 1 S 0 6 1 0 94 19 - 0 ksofti ? 00:00:00 ksoftirqd/1 5 S 0 7 1 0 -40 - - 0 watchd ? 00:00:00 watchdog/1 1 S 0 8 1 0 -40 - - 0 migrat ? 00:00:00 migration/2
#6: free – Display Memory Usage
Please see below output carefully, you will find used and free memory by system also it shows buffered memory used by kernel.
root@linuxserverguide [~]# free
Sample Output:
total used free shared buffers cached Mem: 12302896 9739664 2563232 0 523124 5154740 -/+ buffers/cache: 4061800 8241096 Swap: 1052248 0 1052248
#7: iostat – Display Disk Activity
Friends, this is very powerful command which shows CPU avarage load and disk activity simultaneously, see the sample output below to understand it better
root@linuxserverguide [~]# iostat
Sample Outputs:
Linux 2.6.18-128.1.14.el5 (www03.nixcraft.in) 06/26/2009 avg-cpu: %user %nice %system %iowait %steal %idle 3.50 0.09 0.51 0.03 0.00 95.86 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 22.04 31.88 512.03 16193351 260102868 sda1 0.00 0.00 0.00 2166 180 sda2 22.04 31.87 512.03 16189010 260102688 sda3 0.00 0.00 0.00 1615 0
#8: sar – Display System Activity
The sar command is used to collect, report, and save system activity information. To see network counter, enter:
root@linuxserverguide [~]#sar -n DEV | more
To display the network counters from the 24th:
root@linuxserverguide [~]# sar -n DEV -f /var/log/sa/sa24 | more
You can also display real time usage using sar:
root@linuxserverguide [~]# sar 4 5
Sample Outputs:
Linux 2.6.18-128.1.14.el5 (www03.nixcraft.in) 06/26/2009 06:45:12 PM CPU %user %nice %system %iowait %steal %idle 06:45:16 PM all 2.00 0.00 0.22 0.00 0.00 97.78 06:45:20 PM all 2.07 0.00 0.38 0.03 0.00 97.52 06:45:24 PM all 0.94 0.00 0.28 0.00 0.00 98.78 06:45:28 PM all 1.56 0.00 0.22 0.00 0.00 98.22 06:45:32 PM all 3.53 0.00 0.25 0.03 0.00 96.19 Average: all 2.02 0.00 0.27 0.01 0.00 97.70
#9: mpstat – Multiprocessor Usage
The mpstat command displays activities for each available processor, processor 0 being the first one. mpstat -P ALL to display average CPU utilization per processor:
- root@linuxserverguide [~]#
mpstat -P ALL
Sample Output:
Linux 2.6.18-128.1.14.el5 (www03.nixcraft.in) 06/26/2009 06:48:11 PM CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s 06:48:11 PM all 3.50 0.09 0.34 0.03 0.01 0.17 0.00 95.86 1218.04 06:48:11 PM 0 3.44 0.08 0.31 0.02 0.00 0.12 0.00 96.04 1000.31 06:48:11 PM 1 3.10 0.08 0.32 0.09 0.02 0.11 0.00 96.28 34.93 06:48:11 PM 2 4.16 0.11 0.36 0.02 0.00 0.11 0.00 95.25 0.00 06:48:11 PM 3 3.77 0.11 0.38 0.03 0.01 0.24 0.00 95.46 44.80 06:48:11 PM 4 2.96 0.07 0.29 0.04 0.02 0.10 0.00 96.52 25.91 06:48:11 PM 5 3.26 0.08 0.28 0.03 0.01 0.10 0.00 96.23 14.98 06:48:11 PM 6 4.00 0.10 0.34 0.01 0.00 0.13 0.00 95.42 3.75 06:48:11 PM 7 3.30 0.11 0.39 0.03 0.01 0.46 0.00 95.69 76.89