Saving vmstat with timestamp into a file on Redhat Linux Machine

By default vmstat output doesnot include timestamp and therefore, if you want to save it with the timestamp into a file on Redhat Linux machine, you can use following command.

$vmstat -n [delay [count]] | awk '{now=strftime("%Y-%m-%d %T "); print now $0}'>{file path}

e.g. vmstat -n 1 30 | awk '{now=strftime("%Y-%m-%d %T "); print now $0}'>/tmp/vmstat.txt
where:

• n -> display vmstat header once
• 1 -> update vmstat every second
• 30 ->only display 30 records
• awk '{now=strftime("%Y-%m-%d %T "); print now $0}' -> print timestamp
• >/tmp/vmstat.txt -> path where the vmstat is saved

Final result:

2012-03-09 12:05:25 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
2012-03-09 12:05:25  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
2012-03-09 12:05:25  1  0      0 23199964 473756 6719956    0    0     0     2    0    0  0  0 100  0  0
2012-03-09 12:05:26  0  0      0 23199964 473756 6719956    0    0     0     0  118  232  0  0 100  0  0
2012-03-09 12:05:27  0  0      0 23199964 473756 6719956    0    0     0     0  123  260  0  0 100  0  0
2012-03-09 12:05:28  0  0      0 23199964 473756 6719956    0    0     0     0  111  234  0  0 100  0  0
2012-03-09 12:05:29  0  0      0 23199964 473756 6719956    0    0     0     0  133  276  0  0 100  0  0
2012-03-09 12:05:30  1  0      0 23200088 473756 6719956    0    0     0    88  115  263  0  0 100  0  0
2012-03-09 12:05:31  1  0      0 23200088 473756 6719956    0    0     0    12  135  251  0  0 100  0  0
2012-03-09 12:05:32  0  0      0 23200088 473756 6719964    0    0     0     0  113  313  0  0 100  0  0
2012-03-09 12:05:33  0  0      0 23200088 473756 6719964    0    0     0     0  127  270  0  0 100  0  0
2012-03-09 12:05:34  0  0      0 23200088 473756 6719964    0    0     0     0  110  271  0  0 100  0  0
2012-03-09 12:05:35  1  0      0 23200088 473756 6719964    0    0     0     0  142  297  0  0 100  0  0
2012-03-09 12:05:36  0  0      0 23200088 473756 6719964    0    0     0    44  127  264  0  0 100  0  0
2012-03-09 12:05:37  0  0      0 23200088 473756 6719968    0    0     0    12  144  349  0  0 100  0  0
2012-03-09 12:05:38  0  0      0 23200088 473756 6719928    0    0     0     0  143  259  0  0 100  0  0
2012-03-09 12:05:39  0  0      0 23200088 473756 6719928    0    0     0     0  137  273  0  0 100  0  0
2012-03-09 12:05:40  0  0      0 23200088 473756 6719928    0    0     0     0  123  252  0  0 100  0  0
2012-03-09 12:05:41  0  0      0 23200088 473756 6719928    0    0     0   104  198  269  0  0 100  0  0
2012-03-09 12:05:42  0  0      0 23200088 473756 6719928    0    0     0     0  166  280  0  0 100  0  0
2012-03-09 12:05:43  0  0      0 23200088 473756 6719928    0    0     0     0  183  322  0  0 100  0  0
2012-03-09 12:05:44  0  0      0 23200088 473756 6719932    0    0     0     0  205  330  0  0 100  0  0
2012-03-09 12:05:45  0  0      0 23200088 473756 6719932    0    0     0     0  157  260  0  0 100  0  0
2012-03-09 12:05:46  0  0      0 23200088 473756 6719932    0    0     0     0  112  219  0  0 100  0  0
2012-03-09 12:05:47  0  0      0 23200088 473756 6719932    0    0     0    52  144  304  0  0 100  0  0
2012-03-09 12:05:48  0  0      0 23200088 473756 6719932    0    0     0     0  117  230  0  0 100  0  0
2012-03-09 12:05:49  0  0      0 23200088 473756 6719932    0    0     0     0  139  280  0  0 100  0  0
2012-03-09 12:05:50  0  0      0 23200088 473756 6719932    0    0     0     0  126  241  0  0 100  0  0
2012-03-09 12:05:51  0  0      0 23200088 473756 6719932    0    0     0     0  129  243  0  0 100  0  0
2012-03-09 12:05:52  0  0      0 23200088 473756 6719932    0    0     0     0  111  237  0  0 100  0  0
2012-03-09 12:05:53  0  0      0 23200088 473756 6719932    0    0     0    32  132  265  0  0 100  0  0
2012-03-09 12:05:54  0  0      0 23200088 473756 6719932    0    0     0    16  119  260  0  0 100  0  0

vmstat tool for OpenSolaris

I have been interested in learning different tools that are available with Solaris OS. One of these tool is vmstat.

vmstat provides you a one line overall system behavior. I suggest starting out with vmstat tool to understand system behavior before using more detailed tools such as DTRACE.

vmstat provides you a detail on runnable threads, memory, disk, faults, cpu utilization as shown below.



Following definitions have been taken from "Solaris Performance and Tools" book by Richard McDougall.

Counters
kthr:r -> Total number of runnable threads on the dispatcher queue. It provides information on CPU saturation.

faults:in -> number of interrupts per second
faults:sy -> number of system calls per second
faults:cs -> number of context switches per second, both voluntary and involuntary

cpu:us ->percent user time
cpu:sy -> percent system time
cpu:id ->percent idle time


Description
vmstat 3 10 -> means display vmstat information 10 time with a 3 second interval.

For me the most important counters are kthr:r, kthr:b and CPU id.

Bit of explanation on the vmstat picture.
1: First line is a summary since boot.

1: There are no threads waiting in the queue to get CPU time slice. kthr:r = 0. If this value is more than 2 and sustained on a single CPU server then it indicates that CPU saturation.

2: There are no threads waiting on a resource. kthr:b = 0. If this value is more than 0, it might suggest that we may have a bottleneck somewhere. You need to look at these counter over a long duration before making any conclusion.

3: On average, CPU is spending 1% of its time servicing user-mode threads, 2% servicing kernel threads and 97% ideal.

You can save vmstat to a text file from command line for further analysis as follows. To make my log file unique I normally use datetime and pid values:

vmstat [interval] [total count] >>~/[location where to save]/vmstat_`date +%y%m%d_%H%M%S`_$$.txt

i.e.
vmstat 5 5 >>~/Desktop/PerfTools/vmstat/vmstat_`date +%y%m%d_%H%M%S`_$$.txt
This will save a file as vmstat_101111_135557_824.txt in location /Desktop/PerfTools/vmstat/.
This file will consist of 5 vmstat records with 5 second interval between each record.

vmstat 5 >>~/Desktop/PerfTools/vmstat/vmstat_`date +%y%m%d_%H%M%S`.txt
This file will continue saving the data indefinitely with 5 second interval.