Solaris - Consommation de ressources
Révision datée du 5 novembre 2015 à 11:54 par AdminWiki (discussion | contributions) (→Classer les processus par consommation d'I/O disque)
Surveillance disques
Surveiller utilisation des I/O disques
# iostat -xn 2 2
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.1 3.1 9.1 23.5 0.0 0.0 0.0 3.0 0 0 c0d0
34.1 0.0 17.1 0.0 0.0 0.6 0.0 16.9 0 3 c0d1
34.4 0.0 17.2 0.0 0.0 0.6 0.0 16.6 0 3 c0d2
[...]
111.7 56.6 1599.6 990.2 0.0 4.7 0.0 27.8 0 97 vdc34
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 vdc35
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 vdc36
[...]
Les colonnes intéressantes sont "%w" et "%b". "%b" indique le pourcentage d'utilisation du disque (ou de la LUN sur SAN). Ici on voit que le disque vdc34 est utilisé à 97%. Sur ce système, où je n'ai affiché ici que quelques LUNs, de nombreuses LUNs sont utilisées à 100% ou presque et nous ressentons de gros ralentissements. L'activité est trop forte et la baie n'est pas assez performante.
La colonne "%w" indique le pourcentage d'attente pour écrire sur le disque, donc concrètement, si "%w" est supérieur à "%b", c'est que la latence n'est pas sur le disque, mais sur le bus système.
Classer les processus par consommation d'I/O disque
Je n'ai pas trouvé de commande claire pour retourner cette info, mais j'ai trouvé un script perl "rusage" sur le net ici http://www.brendangregg.com/Solaris/prusage. Et un pdf de 45 pages traite en détail de ce sujet ici : http://www.brendangregg.com/Solaris/paper_diskubyp1.pdf.
Voici un exemple d'utilisation et juste après le code source, au cas où le lien ne serait plus accessible. Ici on classe les processus par consommation disque en écriture.
# ./prusage -i 2 2 -s pid PID MINF MAJF INBLK OUBLK CHAR-kb COMM 18085 0 1034624 11 2066703 783785 oracle 18072 0 1004673 28 2006007 2922472 oracle 14414 0 1517 316 926171 525708 tictimed 11107 0 6626 13 777992 1193227 lp 15910 0 1527 137 710801 633187 tictimed 13808 0 964 93 707699 633186 tictimed 14415 0 1188 68 695021 633219 tictimed 14484 0 2700 69 694253 633180 tictimed 13647 0 2211 143 687310 633193 tictimed 15098 0 1186 92 674124 633187 tictimed 13812 0 1268 172 673283 633219 tictimed 13898 0 1042 49 673332 633200 tictimed 12612 0 945 79 671873 633160 tictimed
Et le code source :
#!/usr/bin/perl # # prusage - Process usage stats, Solaris. I/O, sys/usr times, context switches. # A supplement to "ps", can be run as any user. # # 01-Jul-2005, ver 1.00 (check for newer vers, http://www.brendangregg.com) # # # USAGE: prusage [-bchinuwxCT] [-p PID] [-s sort] [-t top] [interval] [count] # # prusage # Default. (-ic 1), fit to screen, 1 secs. # prusage -b # Child times report (must be root or owner) # prusage -i # I/O stats (default) # prusage -u # USR/SYS times # prusage -x # Context Switchs # prusage -w # Wide output # prusage -c # Clear the screen (default) # prusage -C # Don't clear the screen # prusage -T # Don't fit to screen (print all lines) # prusage -p pid # Print this PID only # prusage -s sort # Sort on pid,blks,cpu,utime,inblk,vctx,... # prusage -t lines # Print top lines only # eg, # prusage 2 # 2 second samples (first is historical) # prusage 2 5 # 5 x 2 second samples # prusage -xi 2 # I/O and Context switch reports, 2 secs # prusage -biux 10 # multi output, all reports every 10 secs # prusage -C 10 # 10 second samples, no clear screen # prusage -CT 10 # 10 second samples, all lines # prusage -Ct8 10 5 # 5 x 10 second samples, top 8 lines only # prusage -p 11321 # PID 11321 only # prusage -s pid # sort on PID # # FIELDS: # PID Process ID # MINF Minor Page Faults (satisfied from RAM) # MAJF Major Page Faults (satisfied by disk I/O) # INBLK In Blocks (disk I/O reads) # OUBLK Out Blocks (disk I/O writes) # CHAR-kb Character I/O Kbytes # COMM Command name # USR User Time # SYS System Time # CUSR Child User Time # CSYS Child System Time # WAIT Wait for CPU Time # LOCK User waiting on lock time # TRAP System trap time # VCTX Voluntary Context Switches (I/O bound) # ICTX Involuntary Context Switches (CPU bound) # SYSC System calls # # NOTE: Minor faults always report zero on most versions of Solaris. # # REFERENCE: /usr/include/sys/procfs.h # # SEE ALSO: psio # process I/O # prstat -m # USR/SYS times, ... # /usr/ucb/rusage # historical # # COPYRIGHT: Copyright (c) 2004, 2005 Brendan Gregg. # # This program is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 2 # of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. # # (http://www.gnu.org/copyleft/gpl.html) # # Author: Brendan Gregg [Sydney, Australia] # # 31-Aug-2004 Brendan Gregg Created this. # 12-Mar-2005 " " Processed /proc/*/psinfo as well. # 09-May-2005 " " Processed /proc/*/usage as well. use Getopt::Std; # # --- Default Variables --- # $INTERVAL = 1; # seconds to sample $MAX = 2**32; # max count of samples $NEW = 0; # skip summary output (new data only) $WIDE = 0; # print wide output (don't truncate) $SCHED = 0; # print PID 0 $TOP = 0; # print top many only $FIT = 1; # fit to screen $CLEAR = 1; # clear screen before outputs $STYLE_IO = 1; # default output style, I/O $STYLE_CTX = 0; # output style, Context Switches $STYLE_TIME = 0; # output style, Times $STYLE_CHILD = 0; # output style, Child times $MULTI = 0; # multi reports, multiple styles $TARGET_PID = -1; # target PID, -1 means all $count = 1; # current iteration # # --- Command Line Arguments --- # ### Check usage &Usage() if $ARGV[0] eq "--help"; getopts('bchinuwxp:s:t:CT') || &Usage(); &Usage() if $opt_h; ### Process options $NEW = 1 if $opt_n; $WIDE = 1 if $opt_w; $FIT = 0 if $opt_T; $CLEAR = 0 if $opt_C; $STYLE_IO = 0 if $opt_x || $opt_u || $opt_b; $STYLE_CTX = 1 if $opt_x; $STYLE_TIME = 1 if $opt_u; $STYLE_CHILD = 1 if $opt_b; $STYLE_IO = 1 if $opt_i; $TOP = $opt_t if defined $opt_t; $SORT = $opt_s if defined $opt_s; $TARGET_PID = $opt_p if defined $opt_p; $INTERVAL = shift(@ARGV) || $INTERVAL; $MAX = shift(@ARGV) || $MAX; ### Determine style count $STYLES = $STYLE_IO + $STYLE_CTX + $STYLE_TIME + $STYLE_CHILD; $MULTI = 1 if $STYLES > 1; ### Determine clear seq $CLEARSTR = `clear` if $CLEAR; ### Fit to screen if ($FIT && ! $opt_t) { my ($row,$col) = &getwinsz(); $TOP = int(($row - $STYLES * 2) / $STYLES); } # # --- Main --- # for (;$count <= $MAX; $count++) { ### Get data &GetProcStat(); # fetch and save /proc stats in %PID{$pid} next if $NEW && $count == 1; ### Print data print $CLEARSTR if $CLEAR; &PrintIO($SORT) if $STYLE_IO; &PrintCtx($SORT) if $STYLE_CTX; &PrintTime($SORT) if $STYLE_TIME; &PrintChild($SORT) if $STYLE_CHILD; ### Pause sleep($INTERVAL) unless $count == $MAX; ### Cleanup memory undef %PID; undef %Comm; } # # --- Subroutines --- # # GetProcStat - Gets /proc usage statistics and saves them in %PID. # This can be run multiple times, the first time %PID will be # populated with the summary since boot values. # This reads /proc/*/usage and /proc/*/psinfo. # sub GetProcStat { my $pid; chdir "/proc"; foreach $pid (sort {$a<=>$b} <*>) { next if $pid == $$; next if $pid == 0 && $SCHED == 0; next if $TARGET_PID > -1 && $pid != $TARGET_PID; # # struct prusage # ### Read usage stats open(USAGE,"/proc/$pid/usage") || next; read(USAGE,$usage,256); close USAGE; ### Unpack usage values ($pr_lwpid, $pr_count, $pr_tstamp, $pr_create, $pr_term, $pr_rtime, $pr_utime, $pr_stime, $pr_ttime, $pr_tftime, $pr_dftime, $pr_kftime, $pr_ltime, $pr_slptime, $pr_wtime, $pr_stoptime, $filltime, $pr_minf, $pr_majf, $pr_nswap, $pr_inblk, $pr_oublk, $pr_msnd, $pr_mrcv, $pr_sigs, $pr_vctx, $pr_ictx, $pr_sysc, $pr_ioch, $filler) = unpack("iia8a8a8a8a8a8a8a8a8a8a8a8a8a8a48LLLLLLLLLLLLa40",$usage); ### Process usage values $New{$pid}{utime} = timestruct2int($pr_utime); $New{$pid}{stime} = timestruct2int($pr_stime); $New{$pid}{ttime} = timestruct2int($pr_ttime); $New{$pid}{ltime} = timestruct2int($pr_ltime); $New{$pid}{wtime} = timestruct2int($pr_wtime); $New{$pid}{slptime} = timestruct2int($pr_slptime); $New{$pid}{minf} = $pr_minf; $New{$pid}{majf} = $pr_majf; $New{$pid}{nswap} = $pr_nswap; $New{$pid}{inblk} = $pr_inblk; $New{$pid}{oublk} = $pr_oublk; $New{$pid}{vctx} = $pr_vctx; $New{$pid}{ictx} = $pr_ictx; $New{$pid}{sysc} = $pr_sysc; $New{$pid}{ioch} = $pr_ioch; # and a couple of my own, $New{$pid}{blks} = $pr_inblk + $pr_oublk; $New{$pid}{ctxs} = $pr_vctx + $pr_ictx; $New{$pid}{cpu} = $New{$pid}{utime} + $New{$pid}{stime}; # # struct psinfo # ### Read psinfo stats open(PSINFO,"/proc/$pid/psinfo") || next; read(PSINFO,$psinfo,256); close PSINFO; ### Unpack psinfo values ($pr_flag, $pr_nlwp, $pr_pid, $pr_ppid, $pr_pgid, $pr_sid, $pr_uid, $pr_euid, $pr_gid, $pr_egid, $pr_addr, $pr_size, $pr_rssize, $pr_pad1, $pr_ttydev, $pr_pctcpu, $pr_pctmem, $pr_start, $pr_time, $pr_ctime, $pr_fname, $pr_psargs, $pr_wstat, $pr_argc, $pr_argv, $pr_envp, $pr_dmodel, $pr_taskid, $pr_projid, $pr_nzomb, $filler) = unpack("iiiiiiiiiiIiiiiSSa8a8a8Z16Z80iiIIaa3iiia",$psinfo); ### Save command name $Comm{$pid} = $pr_fname; next unless $STYLE_CHILD; # only child needs the following, # # struct pstatus # ### Read pstatus stats open(PSTATUS,"/proc/$pid/status") || next; read(PSTATUS,$pstatus,128); close PSTATUS; ### Unpack pstatus values ($pr_flags, $pr_nlwp, $pr_pid, $pr_ppid, $pr_pgid, $pr_sid, $pr_aslwpid, $pr_agentid, $pr_sigpend, $pr_brkbase, $pr_brksize, $pr_stkbase, $pr_stksize, $pr_utime, $pr_stime, $pr_cutime, $pr_cstime, $filler) = unpack("iiiiiiiia16iiiia8a8a8a8a",$pstatus); ### Process pstatus values $New{$pid}{cutime} = timestruct2int($pr_cutime); $New{$pid}{cstime} = timestruct2int($pr_cstime); $New{$pid}{ccpu} = $New{$pid}{cutime} + $New{$pid}{cstime}; } ### Cleanup memory foreach $pid (keys %New) { # save PID values, foreach $key (keys %{$New{$pid}}) { $PID{$pid}{$key} = $New{$pid}{$key} - $Old{$pid}{$key}; } } undef %Old; foreach $pid (keys %New) { # save old values, foreach $key (keys %{$New{$pid}}) { $Old{$pid}{$key} = $New{$pid}{$key}; } } } # PrintIO - print a report on I/O statistics: minf, majf, inblk, oublk, ioch. # sub PrintIO { my $sort = shift || "blks"; my $top = $TOP; my $pid; ### Print header printf("%6s %5s %5s %8s %8s %9s %s\n","PID", "MINF","MAJF","INBLK","OUBLK","CHAR-kb","COMM"); ### Print report foreach $pid (&SortPID("$sort")) { printf("%6s %5s %5s %8s %8s %9.0f %s\n",$pid, $PID{$pid}{minf},$PID{$pid}{majf},$PID{$pid}{inblk}, $PID{$pid}{oublk},$PID{$pid}{ioch}/1024, trunc($Comm{$pid},33)); last if --$top == 0; } print "\n" if $MULTI; } # PrintTime - print a report on Times: utime, stime, wtime, ltime, ttime. # sub PrintTime { my $sort = shift || "cpu"; my $top = $TOP; my $pid; ### Print header printf("%6s %8s %8s %8s %6s %6s %s\n","PID", "USR","SYS","WAIT","LOCK","TRAP","COMM"); ### Print report foreach $pid (&SortPID("$sort")) { printf("%6s %8.2f %8.2f %8.2f %6.2f %6.2f %s\n",$pid, $PID{$pid}{utime},$PID{$pid}{stime},$PID{$pid}{wtime}, $PID{$pid}{ltime},$PID{$pid}{ttime},trunc($Comm{$pid},32)); last if --$top == 0; } print "\n" if $MULTI; } # PrintCtx - print a report on Context Swithes: utime, stime, vctx, ictx, sysc. # sub PrintCtx { my $sort = shift || "ctxs"; my $top = $TOP; my $pid; ### Print header printf("%6s %7s %7s %9s %8s %10s %s\n","PID", "USR","SYS","VCTX","ICTX","SYSC","COMM"); ### Print report foreach $pid (&SortPID("$sort")) { printf("%6s %7.2f %7.2f %9s %8s %10s %s\n",$pid, $PID{$pid}{utime},$PID{$pid}{stime},$PID{$pid}{vctx}, $PID{$pid}{ictx},$PID{$pid}{sysc},trunc($Comm{$pid},27)); last if --$top == 0; } print "\n" if $MULTI; } # PrintChild - print a report on Times: utime, stime, wtime, ltime, ttime. # sub PrintChild { my $sort = shift || "ccpu"; my $top = $TOP; my $pid; ### Print header printf("%6s %8s %8s %8s %8s %s\n","PID", "USR","SYS","CUSR","CSYS","COMM"); ### Print report foreach $pid (&SortPID("$sort")) { printf("%6s %8.2f %8.2f %8.2f %8.2f %s\n",$pid, $PID{$pid}{utime},$PID{$pid}{stime},$PID{$pid}{cutime}, $PID{$pid}{cstime},trunc($Comm{$pid},32)); last if --$top == 0; } print "\n" if $MULTI; } # SortPID - sorts the PID hash by the key given as arg1, returning a sorted # array of PIDs. # sub SortPID { my $sort = shift; ### Sort numerically if ($sort eq "pid") { return sort {$a <=> $b} (keys %PID); } else { return sort {$PID{$b}{$sort} <=> $PID{$a}{$sort}} (keys %PID); } } # getwinsz - gets the terminal window size and returns it as x, y. # The default size returned is 24x80 if an error is encountered. # sub getwinsz { my $row = 24; my $col = 80; my ($xpix,$ypix,$winsize); my $TIOCGWINSZ = 21608; # check /usr/include/sys/termios.h open(TTY, "+</dev/tty") || return($row,$col); ioctl(TTY, $TIOCGWINSZ, $winsize=) || return($row,$col); ($row, $col, $xpix, $ypix) = unpack('S4', $winsize); return($row,$col); } # timestruct2int - Convert a timestruct value (64 bits) into an integer # of seconds. # sub timestruct2int { my $timestruct = shift; my ($secs,$nsecs,$time); $secs = $nsecs = $time = 0; ($secs,$nsecs) = unpack("LL",$timestruct); $time = $secs + $nsecs * 10**-9; return $time; } # trunc - Returns a truncated string if required. # sub trunc { my $string = shift; my $length = shift; if ($WIDE) { return $string; } else { return substr($string,0,$length); } } # Usage - print usage message and exit. # sub Usage { print STDERR <<END; prusage ver 0.97 USAGE: prusage [-chinuwx] [-p PID] [-s sort] [-t top] [interval] [count] prusage # Default. (-ic 1), fit to screen, 1 secs. prusage -b # Child times report (must be root or owner) prusage -i # I/O stats (default) prusage -u # USR/SYS times prusage -x # Context Switchs prusage -w # Wide output prusage -c # Clear the screen (default) prusage -C # Don't clear the screen prusage -T # Don't fit to screen (print all lines) prusage -p pid # Print this PID only prusage -s sort # Sort on pid,blks,cpu,utime,inblk,vctx,... prusage -t lines # Print top lines only eg, prusage 2 # 2 second samples (first is historical) prusage 2 5 # 5 x 2 second samples prusage -xi 2 # I/O and Context switch reports, 2 secs prusage -biux 10 # multi output, all reports every 10 secs prusage -C 10 # 10 second samples, no clear screen prusage -CT 10 # 10 second samples, all lines prusage -Ct8 10 5 # 5 x 10 second samples, top 8 lines only prusage -p 11321 # PID 11321 only prusage -s pid # sort on PID END exit; }