Showing posts with label AIX Administration. Show all posts
Showing posts with label AIX Administration. Show all posts

AIX Administration



(IBM 9117-570 8-Way LPAR managed by Dual HMCs. EMC DMX Frame 2TB, Sepaton Tape Library. OS: AIX, Applications: Sybase ASE 12.5, Veritas VCS. Function: Backend data warehouse)



AIX Paging Space Management


An very good guide for AIX memory usage is at HERE.


; See real memory size
lsattr -El sys0 | grep realmem

bootinfo -r

svmon -G

; Create Paging
mkps -a -n -s8 rootvg hdisk1
mkps -a -n -s4 rootvg hdisk3


#swapon /dev/paging01

#lsps -a


Display paging:

#lsps -a
Page Space Physical Volume Volume Group Size %Used Active Auto Type
hd6 hdisk0 rootvg 512MB 1 yes yes lv
paging00 hdisk0 rootvg 2560MB 1 yes yes lv


Shrink paging space on the fly:

chps -d decrement_LVs -a y|n paging00


Shrink paging00 by 4 LPs and activate the paging on each subsequent reboot

#chps -d 4 -a y paging00 ; reduce swap space


#swapoff /dev/paging00 ; offline swap space

#rmps paging00 ; to remove paging



;all paging space are defined in /etc/swapspaces file

;to activate all paging space at once
swapon -a

;to deactivate paging03 and remove
swapoff /dev/paging03
rmps paging03

;to reduce the size of paging by two LPs
chps -d2 paging03

;paging summary
mkps
rmps
chps
lsps
swapoff
swapon
chps -s
chps -d
shrinkps
svmon
topas
vmstat
sar
istat
slibclean
ps

Mirrorring Rootdisk

Set Boot list for different options:
===================================================================
bootlist -m normal -o ; to see the bootlist
bootlist -m service -o ; to see the maintenance bootlist
bootlist -m normal -o hdisk0 hdisk4


To mirror rootvg, current bootdisk is hdisk0.
===================================================================
1. Add disk
extendvg -f rootvg hdisk1
chvg -Qn rootvg

2. Mirror data (create LVM structure and sync data)

# mklvcopy hd1 2 hdisk1
# mklvcopy hd2 2 hdisk1
# mklvcopy hd3 2 hdisk1
# mklvcopy hd4 2 hdisk1
# mklvcopy hd5 2 hdisk1
# mklvcopy hd6 2 hdisk1
# mklvcopy hd7 2 hdisk1
# mklvcopy hd8 2 hdisk1
# mklvcopy hd9var 2 hdisk1
# mklvcopy hd10opt 2 hdisk1
# mklvcopy hd11msb 2 hdisk1
# mklvcopy lvtivoli 2 hdisk1

Or use
mirrorvg rootvg ; this will disable qurum

Or use
mirrorvg -s rootvg ; just create LVM structure, no data sync
syncvg -v rootvg ; sync the rootvg data

or use
mirrorvg -S rootvg ; sync data in the background

3.create BLV and boot record

bosboot -ad hdisk1 ; recreate BLV and the boot record on hdisk1
bosboot -ad -u /dev/hdisk1 or

4.Update bootlist
bootlist -m normal hdisk1 hdisk0
bootlist -m service cd0 rmt0 hdisk0 hdisk1


(Actual operations Screen Shot)

0:root@sactgx0005:/home/root # bootlist -m normal -o
hdisk0
0:root@sactgx0005:/home/root # ipl_varyon -i


PVNAME BOOT DEVICE PVID VOLUME GROUP ID
hdisk0 YES 00031523be6762d10000000000000000 00031523
00004c00
hdisk1 NO 00031523d36a9c5e0000000000000000 00031523
00004c00
hdisk2 NO 00031523f3bacf050000000000000000 00031523
f3badb67
hdisk3 NO 00031523f3baf1de0000000000000000 00031523
f3badb67
hdisk4 NO 00031523f3bb5b570000000000000000 00031523
f3bb694c
hdisk5 NO 00031523f3bb82350000000000000000 00031523
f3bb694c
0:root@sactgx0005:/home/root # bosboot -ad hdisk1

bosboot: Boot image is 19527 512 byte blocks.
0:root@sactgx0005:/home/root # bootlist -m normal -o
hdisk0
0:root@sactgx0005:/home/root # ipl_varyon -i


PVNAME BOOT DEVICE PVID VOLUME GROUP ID
hdisk0 YES 00031523be6762d10000000000000000 00031523
00004c00
hdisk1 YES 00031523d36a9c5e0000000000000000 00031523
00004c00
hdisk2 NO 00031523f3bacf050000000000000000 00031523
f3badb67
hdisk3 NO 00031523f3baf1de0000000000000000 00031523
f3badb67
hdisk4 NO 00031523f3bb5b570000000000000000 00031523
f3bb694c
hdisk5 NO 00031523f3bb82350000000000000000 00031523
f3bb694c

Clone rootdisk:
===================================================================



Other Useful Commands
===================================================================
lspv -m hdisk1 ; To see avaliable physical partitions on a disk

bootinfo -b ; to see the current boot device

ipl_varyon -i ; to see which disks are bootable

alt_disk_inst -C hdisk4
alt_disk_inst -C -B -O hdisk4 ; doesn't update the bootlist

bootinfo -K ; see what kernel
bootinfo -y ; see if 64 bit kernel is available


#bosboot -a -L
#lockstat -a

AIX Sybase ASE Performance Tuning

Thanks GOOGLE for making this page top 3 search result of "maxperm sybase ase".

Resolving Poor ASE Performance Due to AIX Default Virtual Memory Paging

ASE delivers its best performance when all of its memory resides in physical memory on the machine.
It ASE’s memory is paged out by the operating system’s virtual memory subsystem then ASE performance
can suffer dramatically.

Poor performance due to memory paging on AIX is often reported as problems with ASE’s checkpoint and
/ or housekeeper tasks. These tasks may be slow, appear hung, or encounter timeslice errors.
Transaction dumps may also be affected as ASE cannot properly execute a checkpoint.
Paging particularly affects these tasks because they often process long lists of memory.
If ASE’s memory has been paged out access to it may require a read from disk, resulting in
poor performance. Any other ASE task that must traverse long lists of memory pages may be similarly affected,
resulting in timeslice errors. Logical lock contention may also increase as tasks
which hold locks take longer to run.

While this is a potential problem on all platforms, certain characteristics of AIX make it more likely
to occur on that platform. The default configuration of the AIX kernel allows the file system cache
to consume up to 80% of the available physical memory pages. Depending on system demand this may result
in ASE memory pages being copied to the swap device. Sybase recommends tuning the AIX virtual memory
subsystem to avoid this scenario.

The AIX values minperm and maxperm loosely control the ratio of page frames used for files versus
those used for computational processes (such as ASE). Tuning these values requires determining
the physical memory load of processes running on the host machine. maxperm should then be set
so that file pages do not interfere with process pages.

For example, consider an AIX host that has 8 Gb of physical memory and hosts two ASE servers.
One ASE server is configured to use 3 Gb of max memory and the other is configured to use 2 Gb of max memory.
Other applications on the host require a total of 1 Gb of memory. Therefore the total memory
requirement is 6 Gb (3 Gb + 2 Gb + 1 Gb). As 75% of this host’s physical memory is needed
for applications (6 Gb out of 8 Gb), maxperm should be set no higher than 25%.

In addition to configuring maxperm, Sybase highly recommends setting strict_maxperm to 1. When strict_maxperm
is set to 0 (the default value), AIX may override the maxperm setting at its discretion. Setting strict_maxperm
to 1 informs AIX that this is a hard limit.



Kernel parameters:


Note: Value for tunable maxperm% must be greater than or equal to the value of tunable maxclient%

=======================
maxclient%

Purpose:
Specifies maximum percentage of RAM that can be used for caching client pages.
Similar to maxperm% but cannot be bigger than maxperm%.
Values:
o Default: 80
o Range: 1 to 100%.
o Type: Dynamic
Diagnosis:
If J2 file pages or NFS pages are causing working storage pages to get paged
out, maxclient can be reduced.
Tuning
Decrease the value of maxclient if paging out to paging space is occurring due
to too many J2 client pages or NFS client pages in memory. Increasing the
value can allow more J2 or NFS client pages to be in memory before page
replacement starts.

Refer To:
Miscellaneous I/O Tuning Parameters

========================

maxperm%

Purpose:
Specifies the point above which the page-stealing algorithm steals only file
pages.
Values:
o Default: total number of memory frames * 0.8
o Range: 1 to 100
o Type: Dynamic
Diagnosis:
Monitor disk I/O with iostat n.
Tuning
This value is expressed as a percentage of the total real-memory page frames
in the system. Reducing this value may reduce or eliminate page replacement of
working storage pages caused by high number of file page accesses. Increasing
this value may help NFS servers that are mostly read-only. For example, if
some files are known to be read repetitively, and I/O rates do not decrease
with time from startup, maxperm may be too low.
Refer To:
Tuning VMM Page Replacement with the vmtune Command

---------------------------------------------------------------------------------------
maxfree

Purpose:
Specifies the number of frames on the free list at which page-stealing is to
stop.
Values:
o Default: 128
o Range: 16 to 204800
o Type: Dynamic
Diagnosis:
Observe free-list-size changes with vmstat n.
Tuning
If vmstat n shows free-list size frequently driven below minfree by
application demands, increase maxfree to reduce calls to replenish the free
list. Generally, keep maxfree - minfree equal to or less than 100. Setting the
value too high causes page replacement to run for a longer period of time.
Value must be at least 8 greater than minfree

-------------------------------------------------------------------------------------

maxpin%

Purpose:
Specifies the maximum percentage of real memory that can be pinned.
Values:
o Default: 80 percent
o Range: 1 to 99
o Type: Dynamic
Diagnosis:
Cannot pin memory, although free memory is available.
Tuning
If this value is changed, the new value should ensure that at least 4 MB of
real memory will be left unpinned for use by the kernel. The maxpin values
must be greater than one and less than 100. Change this parameter only in
extreme situations, such as maximum-load benchmarking.

---------------------------------------------------------------------------------------


minfree

Purpose:
Specifies the minimum number of frames on the free list at which the VMM
starts to steal pages to replenish the free list.
Values:
o Default: maxfree - 8
o Range: 8 to 204800
o Type: Dynamic
Diagnosis:
vmstat n
Tuning
Page replacement occurs when the number of free frames reaches minfree. If
processes are being delayed by page stealing, increase minfree to improve
response time. The difference between minfree and maxfree should always be
equal to or greater than maxpgahead.


-----------------------------------------------------------------------------------------

strict_maxperm

Purpose:
If set to 1, the maxperm value will be a hard limit on how much of RAM can be
used as a persistent file cache.
Values:
o Default: 0 (off)
o Range: 0 or 1.
o Type: Dynamic
Diagnosis:
Excessive page outs to page space caused by too many file pages in RAM.
Tuning
Set to 1 in order to make the maxperm value a hard limit (use in conjunction
with the tuning of the maxperm parameter).
Refer To:
Placing a Hard Limit on Persistent File Cache with strict_maxperm

Hands-on examples:


#vmo –a ; display all

#vmo –D ; reset all to default value


594 vmo -p -o maxclient%=50 ; good for both current and reboot values
594 vmo -r -o maxclient%=50 ; good for reboot values
595 vmo -o maxclient%=50 ; maxperm% must be larger/equal to maxclient %
596 vmo -o maxperm%=50 ; good for now
597 vmo -r -o maxperm%=50 ; good for reboot
602 vmo -r -o strict_maxperm=1 ; good for reboot
603 vmo -o strict_maxperm=1 ; good for now
604 vmo –a




1. To list the current and reboot value, range, unit, type and dependencies of
all tunables parameters managed by the vmo command, type:
vmo -L

2. To turn on and reserve 16MB large pages on a POWER4 system, type:
vmo -r -o lgpg_regions=10 -o lgpg_size=16777216

This command will propose bosboot to the user, and warn that a reboot is
necessary before the change will be effective.
3. To display help on nokilluid, type:
vmo -h nokilluid

4. To turn on v_pinshm after the next reboot, type:
vmo -r -o v_pinshm=1

5. To permanently reset all vmo tunable parameters to default, type:
vmo -p -D

6. To list the reboot value for all virtual Memory Manager tuning parameters,
type:
vmo -r -a

7. To list (spreadsheet format) the current and reboot value, range, unit,
type and dependencies of all tunables parameters managed by the vmo
command, type:
vmo -x


AIX Kernel-VM-Tuning


See memory usage

svmon -G -i 1 2

svmon -P pid

Kernel Tuning in AIX 5L Version 5.2 Performance Tools Guide and Reference.


Performance Overview of the Virtual Memory Manager
(VMM) and Tuning VMM Page Replacement



vmo -a ; display
vmo -d ; set to default value
vmo -o ; display or set a tunable to new value


vmo -p ; When used in combination with -o, -d or -D, makes changes apply to both
current and reboot values, that is, turns on the updating of the
/etc/tunables/nextboot file in addition to the updating of the current value.
These combinations cannot be used on Reboot and Bosboot type parameters becasue
their current value can't be changed.

3:root@sactgx0020:/var/log/nmonlog # vmo -a
memory_frames = 4194304
pinnable_frames = 3947805
maxfree = 288
minfree = 256
minperm% = 20
minperm = 808874
maxperm% = 80
maxperm = 3235503
strict_maxperm = 0
maxpin% = 80
maxpin = 3355444
maxclient% = 80
lrubucket = 131072
defps = 1
nokilluid = 0
numpsblks = 6684672
npskill = 52224
npswarn = 208896
v_pinshm = 0
pta_balance_threshold = n/a
pagecoloring = n/a
framesets = 2
mempools = 1
lgpg_size = 0
lgpg_regions = 0
num_spec_dataseg = 0
spec_dataseg_int = 512
memory_affinity = 1
htabscale = n/a
force_relalias_lite = 0
relalias_percentage = 0
data_stagger_interval = 161
large_page_heap_size = 0
kernel_heap_psize = 4096
soft_min_lgpgs_vmpool = 0
vmm_fork_policy = 0
low_ps_handling = 1
mbuf_heap_psize = 4096
strict_maxclient = 1
cpu_scale_memp = 8
lru_poll_interval = 0
lru_file_repage = 1



1:root@sactgx0010:/home/root # vmo -L
NAME CUR DEF BOOT MIN MAX UNIT TYPE
DEPENDENCIES
--------------------------------------------------------------------------------
memory_frames 3840K 3840K 4KB pages S
--------------------------------------------------------------------------------
pinnable_frames 3644K 3644K 4KB pages S
--------------------------------------------------------------------------------
maxfree 128 128 128 16 200K 4KB pages D
minfree
memory_frames
--------------------------------------------------------------------------------
minfree 120 120 120 8 200K 4KB pages D
maxfree
memory_frames
--------------------------------------------------------------------------------
minperm% 20 20 20 1 100 % memory D
maxperm%
--------------------------------------------------------------------------------
minperm 757803 757803 S
--------------------------------------------------------------------------------
maxperm% 80 80 80 1 100 % memory D
minperm%
maxclient%
--------------------------------------------------------------------------------
maxperm 2960K 2960K S
--------------------------------------------------------------------------------
strict_maxperm 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
maxpin% 80 80 80 1 99 % memory D
pinnable_frames
memory_frames
--------------------------------------------------------------------------------
maxpin 3M 3M S
--------------------------------------------------------------------------------
maxclient% 80 80 80 1 100 % memory D
maxperm%
--------------------------------------------------------------------------------
lrubucket 128K 128K 128K 64K 4KB pages D
--------------------------------------------------------------------------------
defps 1 1 1 0 1 boolean D
--------------------------------------------------------------------------------
nokilluid 0 0 0 0 4G-1 uid D
--------------------------------------------------------------------------------
numpsblks 7808K 7808K 4KB pages S
--------------------------------------------------------------------------------
npskill 61K 61K 61K 1 7M-1 4KB pages D
--------------------------------------------------------------------------------
npswarn 244K 244K 244K 0 7M-1 4KB pages D
--------------------------------------------------------------------------------
v_pinshm 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
pta_balance_threshold n/a 50 50 0 99 % pta segment R
--------------------------------------------------------------------------------
pagecoloring n/a 0 0 0 1 boolean B
--------------------------------------------------------------------------------
framesets 2 2 2 1 10 B
--------------------------------------------------------------------------------
mempools 1 1 1 1 4 B
--------------------------------------------------------------------------------
lgpg_size 0 0 0 0 256M bytes B
lgpg_regions
--------------------------------------------------------------------------------
lgpg_regions 0 0 0 0 B
lgpg_size
--------------------------------------------------------------------------------
num_spec_dataseg 0 0 0 0 B
--------------------------------------------------------------------------------
spec_dataseg_int 512 512 512 0 B
--------------------------------------------------------------------------------
memory_affinity 1 1 1 0 1 boolean B
--------------------------------------------------------------------------------
htabscale n/a -1 -1 -4 0 B
--------------------------------------------------------------------------------
force_relalias_lite 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
relalias_percentage 0 0 0 0 32K-1 D
--------------------------------------------------------------------------------
data_stagger_interval 161 161 161 0 0 4KB pages D
lgpg_size
--------------------------------------------------------------------------------
large_page_heap_size 0 0 0 0 8E-1 bytes B
lgpg_size
--------------------------------------------------------------------------------
kernel_heap_psize 4K 4K 4K 4K 16M bytes B
lgpg_size
--------------------------------------------------------------------------------
soft_min_lgpgs_vmpool 0 0 0 0 90 % D
lgpg_size
--------------------------------------------------------------------------------
vmm_fork_policy 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
low_ps_handling 1 1 1 1 2 D
--------------------------------------------------------------------------------
mbuf_heap_psize 4K 4K 4K 4K 16M bytes B
--------------------------------------------------------------------------------
strict_maxclient 1 1 1 0 1 boolean D
--------------------------------------------------------------------------------
cpu_scale_memp 8 8 8 1 64 B
--------------------------------------------------------------------------------
lru_poll_interval 0 0 0 0 60000 milliseconds D
--------------------------------------------------------------------------------
lru_file_repage 1 1 1 0 1 boolean D
--------------------------------------------------------------------------------

n/a means parameter not supported by the current platform or kernel

Parameter types:
S = Static: cannot be changed
D = Dynamic: can be freely changed
B = Bosboot: can only be changed using bosboot and reboot
R = Reboot: can only be changed during reboot
C = Connect: changes are only effective for future socket connections
M = Mount: changes are only effective for future mountings
I = Incremental: can only be incremented

Value conventions:
K = Kilo: 2^10 G = Giga: 2^30 P = Peta: 2^50
M = Mega: 2^20 T = Tera: 2^40 E = Exa: 2^60

AIX Checkpoints


A general 10-point checklist for new build AIX servers.
Actual check-point and values may vary. This is just a reference.

Quick Checkpoint:
1. Adjust paging size;
2. Configure NIC/etherchannel;
3. Update /etc/security/limits;
4. Check/configure the dump devices;
5. Create a shutdown wrapper;
6. Setup/modify sudo privilege;
7. Install additional packages (SAN, EMC, HDS, etc.)
8. Setup NIS client (if required)
9. Mirror rootvg and verify rootvg mirroring
10. Fine tune kernel memory settings (vmo)


1. Increase paging space size as appropriate:
# mkps –a –n –sXX rootvg

Update the system settings:
# chdev -l aio0 -a minservers='250' -a maxservers='500'
# chdev -l aio0 -a maxreqs=8192 –a autoconfig='available'
# chdev -l sys0 -a cpuguard=enable -a autorestart=true
# chdev -l sys0 -a iostat=true
# chdev -l sys0 -a maxuproc=500

2. Configure NIC adapter and create etherchannel:
Remove and recreate the interface used to build the OS:
# ifconfig enX down detach
# rmdev –dl enX
# cfgmgr -v

Check each NIC adapter's media speed:
# lsdev -Cc adapter | awk '{print $1}' | grep ent | while read i
do
echo "################## $i"
lsattr -El $i -a media_speed
done

Update each NIC adapter's media speed as necessary:
# chdev -l entX -a media_speed=100_Full_Duplex
(For Gigabit Ethernet, use auto_negotiation)

Create the etherchannel interface which should look like:
EtherChannel / Link Aggregation: ent8
Status: Available
Attributes:
adapter_names ent4 EtherChannel Adapters
alt_addr 0x000000000000 Alternate EtherChannel Address
auto_recovery yes Enable automatic recovery after failover
backup_adapter ent6 Adapter used when whole channel fails
hash_mode default Determines how outgoing adapter is chosen
mode standard EtherChannel mode of operation
netaddr 10.150.144.129 Address to ping
noloss_failover yes Enable lossless failover after ping failure
num_retries 2 Times to retry ping before failing
retry_time 5 Wait time (in seconds) between pings
use_alt_addr no Enable Alternate EtherChannel Address
use_jumbo_frame no Enable Gigabit Ethernet Jumbo Frames

Use the etherchannel to set up the server's public network connectivity

3. Update /etc/security/limits:
default:
fsize = -1
core = 2097151
cpu = -1
data = -1
rss = -1
stack = -1
nofiles = 2000

4. Configure the dump device:
# sysdumpdev –K
# sysdumpdev –C
# sysdumpdev -Pp /dev/sysdumpnull
# rmlv hd7
# mklv -y hd7 -a im -t sysdump rootvg 16 <== 4G based on 256 MB PP size # sysdumpdev -Pp /dev/hd7 5. Create the /etc/rc.shutdown script: ##################### Beginning of rc.shutdown ##################### if [ -r /tmp/allowshutdown ];then rm -f /tmp/allowshutdown ps -ef | grep "/opt/VRTSvcs/bin/had" | grep -v grep > /dev/null
if [ $? -eq 0 ];then
echo
echo "WARNING: VCS should be stopped before shutdown is run"
echo
exit 1
fi

echo "Running /etc/rc.shutdown, please wait...."
exec 1>/tmp/shutdown.log 2>&1
date

#
# Add the processes to be shut down in background here
#
# sleep 120

else
echo
banner `hostname`
echo
echo "WARNING: If you really want to shutdown this system"
echo " you must create the file /tmp/allowshutdown"
echo
exit 1
fi
######################## End of rc.shutdown ########################

# chmod 700 /etc/rc.shutdown


6. Setup sudo: edit /etc/sudoers for proper setting


7. Install EMC (optional)
# Install EMC Software and License Key
echo "PP_LPAR_KEY_FIX=1" >> /etc/environment

Install EMC ODM definitions:
# cd /emc_package_home
# installp -agXd. EMC

Install EMC PowerPath:
# installp -agXd. EMCpower

Add EMC PowerPath license key:
# emcpreg -add wxyz-wxyz-wxyz-wxyz-wxyz-wxyz

Install Solution Enabler:
# installp -agXd. EMCsymm


Config sendmail for security

Reboot:
# touch /tmp/allowshutdown
# shsutdown -rF

Add the appropriate netgroups to /etc/passwd

Create the /etc/no_shell file:

##################### Beginning of /etc/no_shell #####################
#!/bin/sh
/bin/cat 1>&2 <> /etc/passwd


8. Setup NIS client and Test NIS:

Add the following entry to /etc/environment:
# echo "YPBIND_MAXWAIT=10" >> /etc/environment

Create ypservers from list:


Change the domain name:
(For PROD/DR servers)
# /usr/sbin/chypdom -B prod.us.db.com
(For DEV/UAT servers)
# /usr/sbin/chypdom -B dev.us.db.com

This command updates /etc/rc.nfs:
if [ -x /usr/bin/domainname ]; then
/usr/bin/domainname
fi

Start ypbind:
# /usr/sbin/mkclient -B -S

This command updates /etc/rc.nfs:
if [ -x /usr/lib/netsvc/yp/ypbind ]; then
start ypbind /usr/lib/netsvc/yp/ypbind -ypsetme
/usr/sbin/ypset
fi

Verify that ypbind is running:
# ps -ef | grep ypbind
# ypwhich
# ypwhich -m
# ypcat passwd

Append the following line to /etc/group for NIS group access:
# echo "+:" >> /etc/group

Update host resolution order (if needed):
/etc/netsvc.conf --> hosts=local,bind - OR -
NSORDER=local,bind;export NSORDER
(Default is bind,nis,local)

9. Mirror rootvg
# mirrorvg –S rootvg
# bosboot -ad /dev/ipldevice
# bootlist -m normal hdisk0 hdisk1


10.Tune VMM (vmo, ioo)
vmo -a
vmo -o
For Sybase ASE servers on AIX 5.xL, to ensure the best performance on Sybase, set the following parameters:
maxperf% = 25% (default is 80%)
strict_maxperf = 1

For details on vmo and database performance issue, please refer to this blog article.

AIX Breaking Root Disk Mirrors for OS Upgrade

Before performing any OS patching, it is common practice to break the root disk mirrors. One mirror is used to perform the patching and the second mirror is saved for fail back purposes. This note describes a procedure to break root disk for AIX OS upgrade. This procedure was tested successfully on AIX 5.3 TL7 but should work on all AIX 5.3+ technology levels.

Note:
1. Do a mksysb backup before patching to provide an alternative method of rebuilding the OS from the NIM server using mksysb copy.
2. The following example assumes that the rootvg contains mirrored copies of LVs on hdisk0 and hdisk1.

###
### Step 1: Save the system configuration and break the root mirrors
###
Capture and save outputs from the following commands:
# lspv
# lsvg
# lsvg -o
# lsvg -l `lsvg -o`
# lsvg -p `lsvg -o`
# bootlist -m normal -o

Export all non-rootvg volume groups
Extract the major and minor number of the disk the system was booted from:

# ls -al /dev/ipldevice
crw------- 2 root system 22, 9 Feb 10 11:15 /dev/ipldevice

Confirm the disk the system was booted from:
# ls -al /dev | grep "22, 9"
brw------- 1 root system 22, 9 Feb 11 15:50 hdisk0
crw------- 2 root system 22, 9 Feb 10 11:15 ipldevice
crw------- 2 root system 22, 9 Feb 10 11:15 rhdisk0

Check the current bootlist:
# bootlist -m normal -o
hdisk0 blv=hd5
hdisk1 blv=hd5

Remove LV mirrored copies from hdisk1:
# unmirrorvg rootvg hdisk1

Remove the boot block from hdisk1:
# chpv -c hdisk1

Remove hdisk1 from rootvg:
# reducevg rootvg hdisk1

If you see the following error:
0516-016 ldeletepv: Cannot delete physical volume with allocated
partitions. Use either migratepv to move the partitions or
reducevg with the -d option to delete the partitions.
0516-884 reducevg: Unable to remove physical volume hdisk1
some LVs are present only in hdisk1 and have not been mirrored.

Migrate them to hdisk0 and then remove hdisk1 from rootvg:
# migratepv hdisk1 hdisk0
# reducevg rootvg hdisk1

Confirm that hdisk1 has been removed from rootvg:
# lsvg -p rootvg
rootvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
hdisk0 active 546 250 107..87..22..00..34


###
### Step 2: Create the alternate boot disk
###
Create the alternate boot disk on hdisk1:
# alt_disk_copy -d hdisk1
Check the volume groups:
# lsvg
rootvg
altinst_rootvg <= altinst_rootvg contains rootvg on hdisk1

# lsvg -o
rootvg <= altinst_rootvg is varied off

The boot list should now point to the alternate disk only:
# bootlist -m normal -o
hdisk1 blv=hd5


###
### Step 3: Boot from the alternate boot disk (hdisk1)
###
Reboot:
# shutdown -rF
Verify that the system was booted from the alternate boot disk (hdisk1):
# ls -al /dev/ipldevice
crw------- 2 root system 22, 1 Feb 20 15:50 /dev/ipldevice

# ls -al /dev | grep "22, 1"
brw------- 1 root system 22, 1 Feb 20 16:07 hdisk1
crw------- 2 root system 22, 1 Feb 20 15:50 ipldevice
crw------- 2 root system 22, 1 Feb 20 15:50 rhdisk1

Check the volume groups:
# lsvg
rootvg
old_rootvg <= old_rootvg contains rootvg on hdisk0

# lsvg -o
rootvg <= old_rootvg is varied off
Change the boot list to point to the original boot disk only:
# bootlist -m normal hdisk0
# bootlist -m normal -o
hdisk0 blv=hd5

###
### Step 4: Boot from the original boot disk (hdisk0)
###
Reboot:
# shutdown -rF
Confirm that the system was booted from the original boot disk (hdisk0):
# ls -al /dev/ipldevice
crw------- 2 root system 22, 9 Feb 10 11:15 /dev/ipldevice
# ls -al /dev | grep "22, 9"
brw------- 1 root system 22, 9 Feb 11 15:50 hdisk0
crw------- 2 root system 22, 9 Feb 10 11:15 ipldevice
crw------- 2 root system 22, 9 Feb 10 11:15 rhdisk0

Proceed with the OS upgrade

###
### Step 5: If the upgrade is successful
###
Remove the alternate boot disk:
# exportvg alinst_rootvg

Remove the boot block from hdisk1:
# chpv -c hdisk1

Add hdisk1 to rootvg:
# extendvg -f rootvg hdisk1

Reestablish root mirroring:
# mirrorvg -S rootvg

Update the boot list:
# bootlist -m normal hdisk0 hdisk1

Rebuild the boot volume:
# bosboot -ad /dev/ipldevice
Import non-rootvg volume groups

###
### Step 6:
### If the upgrade is unsuccessful, boot from the alternate boot disk
### to failback to the OS before the upgrade
###
Update the boot list to point to the alternate boot disk:
# bootlist -m normal hdisk1
# bootlist -m normal -o
hdisk1 blv=hd5

Reboot:
# shutdown -rF
Check the volume groups:
# lsvg
rootvg
old_rootvg <= old_rootvg contains rootvg on hdisk0

# lsvg -o
rootvg <= old_rootvg is varied off

Remove old_rootvg permanently:
# alt_rootvg_op -X old_rootvg - OR -
# alt_disk_install -X old_rootvg

Add hdisk0 to rootvg and reestablish root mirroring
Import non-rootvg volume groups

AIX Boot Related Administration Notes (1)

AIX Boot Related Administration Notes

RAMFS:

RAMFS files are accessed using a prefix of /../, eg. rc.boot uses /../usr/sbin/fsck to find fsck.

Mount root out onto /mnt, or you will be unable to access the RAMFS files.

;To see the last boot device

#bootinfo -b

;To see all boot process related messages

#alog -t boot -o

Multi-user initialization completed

ot: executing "copycore"

Changed: tty

Changed: ipldevice

Changed: IPL_rootvg

mergedev replaced 3 files in the hardfile /dev directory

rc.boot: run time mount of /var

rc.boot: run time mount of /tmp

rc.boot: checking free space in /tmp

rc.boot: executing "strload"

rc.boot: allow 64-bit apps

rc.boot: executing "cfglivedump -c"

rc.boot: executing "cfgmgr"

cfgmgr is running in phase 2

----------------

Time: 0 LEDS: 0x538

Invoking top level program -- "/etc/methods/cfgprobe -c /etc/drivers/coreprobe.e

xt"

Time: 0 LEDS: 0x539

Return code = 0

*** no stdout ****

*** no stderr ****

----------------

stdin

;Each fs has two superblocks at 1 and 31.

;To copy the superblock from block 31 to block 1

# dd count=1 bs=4k skip=31 seek=1 if=/dev/hd4 of=/dev/hd4

;To unlock rootvg

#chvg -u rootvg

E1F1 is the led that indicates that a system is looking for a boot devices list in the NVRAM. Please follow the procedurebelow in order to start your system in maintenance mode and then recreate the boot list:

Booting in Service Mode

This document describes how to boot the system into Service mode (also known as Maintenance mode) to install the machine,restore an operating system backup, or perform maintenance on the rootvg volume group.

The information in this document applies to AIX Versions 3.x, 4.x and 5.x.

Booting PCI-based systems into Service mode

PCI machine-specific information

Accessing rootvg and mounting file systems

Related documentation

Booting PCI-based systems into Service mode

When booting a PowerPC into Service mode, cd0 or rmt0 must be before the hdisk in the bootlist. If not, change the bootlist

at boot time. On some models, you can set the machine to use a default bootlist that includes both cd0 and rmt0. If a bootable CD or tape is in the CD-ROM or tape drive, the machine will boot from this device.

For most of the newer PCI-based models, selecting the default bootlist, with a bootable tape or CD loaded in the machine, causes the system to automatically boot from that device. Generally, the next menu on the screen asks the administrator

to define the system console.

For all machines discussed here, if you are using a graphical terminal, you will use a function key such as F5. If you are using an ASCII terminal, use an equivalent number key such as 5. Use the numbers across the top of the keyboard, not thenumbers on the numeric keypad. On ASCII terminals, the icons may not be displayed on the screen; the number can be pressedbetween the second and third beeps, the second beep being a series of three clicks.

PCI machine-specific information

The following systems all use the F5 or 5 key to read from the default boot list, which is written into the system firmware:

MODEL 7017 7024 7025 7026 7043 7137

-------------- ------- ------- ------- ------- ------- -------

TYPE S70 E20 F30 H10 43P-140 F3L

S7A E30 F40 H50 43P-150

S80 F50 H70 43P-240

B80 43P-260

On these machines, use 5 (on the keyboard, not the keypad) if you are using an ASCII terminal. On a locally attached graphics console, use the F5 function key. The F5 or 5 key must be pressed just after the keyboard icon or message is displayed on the console. If you have either a 7026-M80, 7026-H80 or a 7025-F80, then the 5 key will be the default whether you have an ascii or graphics console.

The following systems use the F1 key to enter System Management Services mode (SMS):

MODEL 6040 7042 7247 7249

------- ------- ------- ------- -------

TYPE 620 850 82x 860

You should be in an Easy-Setup menu. Select the Start Up menu. Clear the current bootlist settings and then select the CD-ROM for choice 1 and hdd (the hard disk) for choice 2. Select OK. Insert the CD-ROM and select the EXIT icon. The machine should now boot from the CD-ROM.

The following systems use the F2 key to enter SMS:

MODEL 6015 6050 6070 7020 7248

------- ------- ------- ------- ------- -------

TYPE 440 830 850 40P 43P

Select Select Boot Device from the initial menu on the screen, and then select Restore Default Settings from the list. Press the Esc key to exit all the menus, and then reboot the machine. The system should boot from your bootable media.

For information on accessing the rootvg volume group, see the next section in this document.

Accessing rootvg and mounting file systems For AIX Versions 4 and 5, choose Start Maintenance Mode for System Recovery , option 3. The next screen will be called Maintenance; select option 1, Access a Root Volume Group. At the next screen, type 0 to continue, and select the appropriate volume group by typing the number next to it. A screen like the following will display.

Example:

Access a Root Volume Group

Type the number for a volume group to display the logical volume information and press Enter.

1) Volume Group 0073656f2608e46a contains these disks:

hdisk0 2063 04-C0-00-4,0

Once a volume group has been selected, information will be displayed about that volume group.

Example:

Volume Group Information

------------------------------------------------------------------------------

Volume Group ID 0073656f2608e46a includes the following logical volumes:

hd6 hd5 hd8 hd4 hd2 hd9var

hd3 hd1

------------------------------------------------------------------------------

Type the number of your choice and press Enter.

1) Access this Volume Group and start a shell

2) Access this Volume Group and start a shell before mounting filesystems

99) Previous Menu

If the logical volumes listed do not include logical volumes like hd4, hd2, hd3, and so on, you may have selected the wrong volume group. Press 99 to back up one screen and select again.

Now you may select one of two options: Access this volume group and start a shell , option 1, or Access this volume group and start a shell before mounting file systems , option 2. Option 2 allows you to perform file system maintenance on /, /usr, /tmp, and /var before mounting them.

NOTE: If you intend to use SMIT or vi, set your terminal type in preparation for editing the file. xxx stands for a terminal

type such as lft, ibm3151, or vt100.

TERM=

export TERM

Errors from these steps may indicate failed or corrupt disks in rootvg. These problems should be corrected. For additional assistance, contact your vendor, your local branch office, or your AIX support center.

Related documentation

For more in-depth coverage of this subject, the following IBM publication is recommended:

AIX Version 4.3 System Management Guide: Operating System and Devices

AIX Version 5.1 System Management Guide: Operating System and Devices

IBM documentation can also be accessed online through the following URL:

http://www.rs6000.ibm.com/resource/aix_resource/Pubs/index.html

Similar documents can be accessed through the following URL:

http://techsupport.services.ibm.com/server/support?view=pSeries

Once you have a valid prompt, you can recreate the boot device list (it's supposed you already know the hdisk number from which AIX starts, let's supposed it's hdisk0

bootlist -m normal hdisk0

If ths action does not work, repeat the procedure and issue

bosboot -ad /dev/ipldevice

Just before the bootlist command

UNIX RPC - Remote Procedure Call


Thanks GOOGLE for making this the #1 page in searching "rpcinfo clnt_create"




Let's jump into the details of RPC!

RPC Key Facts
- rpcbind daemon listens at 111
- /etc/rpc has the program number for rpc programs.
- rpcbind table is maintained - starting the service will update the table.
- rpcinfo -p to see rpcbind table
- uses program number instead of well-known ports
- clients send program number asking for ports
- rpc server check program number against /etc/rpc to know the service
- rpcbind check rpcbind table to get port.
- if service isn't started, no entry in rpcbind table - rpcinfo -p returns nothing
- start service will register port inside rpcbind table
- unregister service by 'rpcinfo -d service' ver command


Background Information
Network Ports

Two different approaches to port assignments
1)Central Authority
- www.iana.org
- well known ports
- /etc/services lists all the well-known services
lrwxrwxrwx /etc/services -> ./inet/services

Central Authority
telnet example:
inetd listens at port 23.
- client sends telnet request
- inetd detects the request and assign different port back to client
- client talks to server on the given port
- inetd keeps listening at port 23 for new telnet request.


Features of RPC

Dynamic Binding - RPC
- client doesn't know the port #
- hosts assign port dynamically
- sun created RPCs
- sun hard code the program number
- service not listed in /etc/services but program# in /etc/rpc
- when start a service, it record its port in rpcbind table, rpcinfo -p to see
- client requests by program# to rpcbind and rpcbind checks prg# and rpcbind table,
then returns port to client
- the rest communication is just like 1).

-------------

Hands-on Examples 1:
host1 spray host2

host1 spray sends (dest port#:111, source Port#,program# 100012) ---> rpcbind @port 111 on host2

host2: rpcbind reads host1 packets' program# 100012 and checks rpcbind table for
(proto,port,service).

on Host2:
1. host2 check program# with /etc/rpc file and knows this is spray.
rpc program number - 100012 for spray (consults /etc/rpc file, records rpc prog#)

2. host2 rpcbind checks program# against rpcbind table - rpcbind table is generated
during the boot by rpcbind. It has the format of:
program#, port#,service
100012,32760,spray
To list current rpcbind table , use 'rpcinfo -p' command

'rpcinfo -p' shows:

- program number
- version number of the RPC program number
- RPC protocol
- port number
- RPC service

#rpcinfo -p hostname ; get rpcbind table from remote host
#rpcinfo -p ; get rpcbind table from local host
program vers proto port service
100000 4 tcp 111 rpcbind
100000 3 tcp 111 rpcbind
100000 2 tcp 111 rpcbind
100000 4 udp 111 rpcbind
100000 3 udp 111 rpcbind
100000 2 udp 111 rpcbind
100024 1 udp 32802 status
100012 1 udp 33563 sprayd
......
3. host2 sends back the port number (33563) for spray back to the client.
4. the client sends another request with the correct port number for spray to host2.
5. host2 inetd receives the requests (by listening at spray's port) and rpc.sprayd
takes over the communcation.



#rpcinfo -d 100012 1 ; unregisters the RPC service with prog#100012


#svcadm disable svc:/network/rpc/spray:udp
#svcadm enable svc:/network/rpc/spray:udp


To start spray services
#svcadm enable svc:/network/rpc/spray:default


---------------------
Hands-on Examples 2:

ksh:gulf1# svcadm enable svc:/network/rpc/spray:default
ksh:gulf1# grep rpcbind /etc/services
sunrpc 111/udp rpcbind
sunrpc 111/tcp rpcbind

ksh:gulf1# rpcinfo -p | grep spray ;spray started,rpcbind has it
100012 1 udp 33563 sprayd

ksh:gulf1# grep gray /etc/services ;not listed as well-known port


ksh:gulf1# grep sprayd /etc/rpc ;check prog#
sprayd 100012 spray

ksh:gulf1# spray hotel1
spray: cannot clnt_create hotel1:netpath: RPC: Program not registered


Start spray on hotel1 server.

ksh:gulf1# spray hotel1
sending 1162 packets of length 86 to hotel1 ...
no packets dropped by hotel1
25270 packets/sec, 2173285 bytes/sec

unregister 'spray - rpcinfo -d spray 1'

ksh:hotel1#rpcinfo -d spray 1
ksh:gulf1# spray hotel1
spray: cannot clnt_create hotel1:netpath: RPC: Program not registered

Restart spray
ksh:gulf1# svcadm disable svc:/network/rpc/spray:default
ksh:gulf1# svcadm enable svc:/network/rpc/spray:default

Confirm spray is listed in rpcbind
ksh:gulf1# rpcinfo -p hotel1 | grep spray
100012 1 udp 32949 sprayd

ksh:gulf1# spray hotel1
sending 1162 packets of length 86 to hotel1 ...
no packets dropped by hotel1
28799 packets/sec, 2476752 bytes/sec


==============
Below is a tcpdump capture of detailed "rpcinfo -p" query between a Linux host and a Sun Solaris server.

ETHER: ----- Ether Header -----
ETHER:
ETHER: Packet 1 arrived at 22:24:53.67045
ETHER: Packet size = 74 bytes
ETHER: Destination = 0:c:29:2e:ed:c1,
ETHER: Source = 0:c:29:e5:d:fb,
ETHER: Ethertype = 0800 (IP)
ETHER:
IP: ----- IP Header -----
IP:
IP: Version = 4
IP: Header length = 20 bytes
IP: Type of service = 0x00
IP: xxx. .... = 0 (precedence)
IP: ...0 .... = normal delay
IP: .... 0... = normal throughput
IP: .... .0.. = normal reliability
IP: .... ..0. = not ECN capable transport
IP: .... ...0 = no ECN congestion experienced
IP: Total length = 60 bytes
IP: Identification = 5703
IP: Flags = 0x4
IP: .1.. .... = do not fragment
IP: ..0. .... = last fragment
IP: Fragment offset = 0 bytes
IP: Time to live = 64 seconds/hops
IP: Protocol = 6 (TCP)
IP: Header checksum = a058
IP: Source address = 192.168.1.104, ipc4.shanjing.com
IP: Destination address = 192.168.1.100, sun1.shanjing.com
IP: No options
IP:
TCP: ----- TCP Header -----
TCP:
TCP: Source port = 750
TCP: Destination port = 111
TCP: Sequence number = 3368322776
TCP: Acknowledgement number = 0
TCP: Data offset = 40 bytes
TCP: Flags = 0x02
TCP: 0... .... = No ECN congestion window reduced
TCP: .0.. .... = No ECN echo
TCP: ..0. .... = No urgent pointer
TCP: ...0 .... = No acknowledgement
TCP: .... 0... = No push
TCP: .... .0.. = No reset
TCP: .... ..1. = Syn
TCP: .... ...0 = No Fin
TCP: Window = 5840
TCP: Checksum = 0xc95d
TCP: Urgent pointer = 0
TCP: Options: (20 bytes)
TCP: - Maximum segment size = 1460 bytes
TCP: - SACK permitted option
TCP: - TS Val = 2134176, TS Echo = 0
TCP: - No operation
TCP: - Window scale = 2
TCP:

ETHER: ----- Ether Header -----
ETHER:
ETHER: Packet 2 arrived at 22:24:53.67077
ETHER: Packet size = 78 bytes
ETHER: Destination = 0:c:29:e5:d:fb,
ETHER: Source = 0:c:29:2e:ed:c1,
ETHER: Ethertype = 0800 (IP)
ETHER:
IP: ----- IP Header -----
IP:
IP: Version = 4
IP: Header length = 20 bytes
IP: Type of service = 0x00
IP: xxx. .... = 0 (precedence)
IP: ...0 .... = normal delay
IP: .... 0... = normal throughput
IP: .... .0.. = normal reliability
IP: .... ..0. = not ECN capable transport
IP: .... ...0 = no ECN congestion experienced
IP: Total length = 64 bytes
IP: Identification = 27277
IP: Flags = 0x4
IP: .1.. .... = do not fragment
IP: ..0. .... = last fragment
IP: Fragment offset = 0 bytes
IP: Time to live = 64 seconds/hops
IP: Protocol = 6 (TCP)
IP: Header checksum = 4c0e
IP: Source address = 192.168.1.100, sun1.shanjing.com
IP: Destination address = 192.168.1.104, ipc4.shanjing.com
IP: No options
IP:
TCP: ----- TCP Header -----
TCP:
TCP: Source port = 111
TCP: Destination port = 750
TCP: Sequence number = 1322806787
TCP: Acknowledgement number = 3368322777
TCP: Data offset = 44 bytes
TCP: Flags = 0x12
TCP: 0... .... = No ECN congestion window reduced
TCP: .0.. .... = No ECN echo
TCP: ..0. .... = No urgent pointer
TCP: ...1 .... = Acknowledgement
TCP: .... 0... = No push
TCP: .... .0.. = No reset
TCP: .... ..1. = Syn
TCP: .... ...0 = No Fin
TCP: Window = 49232
TCP: Checksum = 0xeedf
TCP: Urgent pointer = 0
TCP: Options: (24 bytes)
TCP: - No operation
TCP: - No operation
TCP: - TS Val = 1532405, TS Echo = 2134176
TCP: - Maximum segment size = 1460 bytes
TCP: - No operation
TCP: - Window scale = 0
TCP: - No operation
TCP: - No operation
TCP: - SACK permitted option
TCP:

ETHER: ----- Ether Header -----
ETHER:
ETHER: Packet 3 arrived at 22:24:53.67082
ETHER: Packet size = 66 bytes
ETHER: Destination = 0:c:29:2e:ed:c1,
ETHER: Source = 0:c:29:e5:d:fb,
ETHER: Ethertype = 0800 (IP)
ETHER:
IP: ----- IP Header -----
IP:
IP: Version = 4
IP: Header length = 20 bytes
IP: Type of service = 0x00
IP: xxx. .... = 0 (precedence)
IP: ...0 .... = normal delay
IP: .... 0... = normal throughput
IP: .... .0.. = normal reliability
IP: .... ..0. = not ECN capable transport
IP: .... ...0 = no ECN congestion experienced
IP: Total length = 52 bytes
IP: Identification = 5705
IP: Flags = 0x4
IP: .1.. .... = do not fragment
IP: ..0. .... = last fragment
IP: Fragment offset = 0 bytes
IP: Time to live = 64 seconds/hops
IP: Protocol = 6 (TCP)
IP: Header checksum = a05e
IP: Source address = 192.168.1.104, ipc4.shanjing.com
IP: Destination address = 192.168.1.100, sun1.shanjing.com
IP: No options
IP:
TCP: ----- TCP Header -----
TCP:
TCP: Source port = 750
TCP: Destination port = 111
TCP: Sequence number = 3368322777
TCP: Acknowledgement number = 1322806788
TCP: Data offset = 32 bytes
TCP: Flags = 0x10
TCP: 0... .... = No ECN congestion window reduced
TCP: .0.. .... = No ECN echo
TCP: ..0. .... = No urgent pointer
TCP: ...1 .... = Acknowledgement
TCP: .... 0... = No push
TCP: .... .0.. = No reset
TCP: .... ..0. = No Syn
TCP: .... ...0 = No Fin
TCP: Window = 1460
TCP: Checksum = 0xea46
TCP: Urgent pointer = 0
TCP: Options: (12 bytes)
TCP: - No operation
TCP: - No operation
TCP: - TS Val = 2134177, TS Echo = 1532405
TCP:

ETHER: ----- Ether Header -----
ETHER:
ETHER: Packet 4 arrived at 22:24:53.67095
ETHER: Packet size = 110 bytes
ETHER: Destination = 0:c:29:2e:ed:c1,
ETHER: Source = 0:c:29:e5:d:fb,
ETHER: Ethertype = 0800 (IP)
ETHER:
IP: ----- IP Header -----
IP:
IP: Version = 4
IP: Header length = 20 bytes
IP: Type of service = 0x00
IP: xxx. .... = 0 (precedence)
IP: ...0 .... = normal delay
IP: .... 0... = normal throughput
IP: .... .0.. = normal reliability
IP: .... ..0. = not ECN capable transport
IP: .... ...0 = no ECN congestion experienced
IP: Total length = 96 bytes
IP: Identification = 5707
IP: Flags = 0x4
IP: .1.. .... = do not fragment
IP: ..0. .... = last fragment
IP: Fragment offset = 0 bytes
IP: Time to live = 64 seconds/hops
IP: Protocol = 6 (TCP)
IP: Header checksum = a030
IP: Source address = 192.168.1.104, ipc4.shanjing.com
IP: Destination address = 192.168.1.100, sun1.shanjing.com
IP: No options
IP:
TCP: ----- TCP Header -----
TCP:
TCP: Source port = 750
TCP: Destination port = 111 (Sun RPC)
TCP: Sequence number = 3368322777
TCP: Acknowledgement number = 1322806788
TCP: Data offset = 32 bytes
TCP: Flags = 0x18
TCP: 0... .... = No ECN congestion window reduced
TCP: .0.. .... = No ECN echo
TCP: ..0. .... = No urgent pointer
TCP: ...1 .... = Acknowledgement
TCP: .... 1... = Push
TCP: .... .0.. = No reset
TCP: .... ..0. = No Syn
TCP: .... ...0 = No Fin
TCP: Window = 1460
TCP: Checksum = 0xe454
TCP: Urgent pointer = 0
TCP: Options: (12 bytes)
TCP: - No operation
TCP: - No operation
TCP: - TS Val = 2134177, TS Echo = 1532405
TCP:
RPC: ----- SUN RPC Header -----
RPC:
RPC: Record Mark: last fragment, length = 40
RPC: Transaction id = 450618384
RPC: Type = 0 (Call)
RPC: RPC version = 2
RPC: Program = 100000 (PMAP), version = 2, procedure = 4
RPC: Credentials: Flavor = 0 (None), len = 0 bytes
RPC: Verifier : Flavor = 0 (None), len = 0 bytes
RPC:
PMAP: ----- Portmapper -----
PMAP:
PMAP: Proc = 4 (Dump the mappings)
PMAP:

ETHER: ----- Ether Header -----
ETHER:
ETHER: Packet 5 arrived at 22:24:53.67117
ETHER: Packet size = 66 bytes
ETHER: Destination = 0:c:29:e5:d:fb,
ETHER: Source = 0:c:29:2e:ed:c1,
ETHER: Ethertype = 0800 (IP)
ETHER:
IP: ----- IP Header -----
IP:
IP: Version = 4
IP: Header length = 20 bytes
IP: Type of service = 0x00
IP: xxx. .... = 0 (precedence)
IP: ...0 .... = normal delay
IP: .... 0... = normal throughput
IP: .... .0.. = normal reliability
IP: .... ..0. = not ECN capable transport
IP: .... ...0 = no ECN congestion experienced
IP: Total length = 52 bytes
IP: Identification = 27278
IP: Flags = 0x4
IP: .1.. .... = do not fragment
IP: ..0. .... = last fragment
IP: Fragment offset = 0 bytes
IP: Time to live = 64 seconds/hops
IP: Protocol = 6 (TCP)
IP: Header checksum = 4c19
IP: Source address = 192.168.1.100, sun1.shanjing.com
IP: Destination address = 192.168.1.104, ipc4.shanjing.com
IP: No options
IP:
TCP: ----- TCP Header -----
TCP:
TCP: Source port = 111
TCP: Destination port = 750
TCP: Sequence number = 1322806788
TCP: Acknowledgement number = 3368322821
TCP: Data offset = 32 bytes
TCP: Flags = 0x10
TCP: 0... .... = No ECN congestion window reduced
TCP: .0.. .... = No ECN echo
TCP: ..0. .... = No urgent pointer
TCP: ...1 .... = Acknowledgement
TCP: .... 0... = No push
TCP: .... .0.. = No reset
TCP: .... ..0. = No Syn
TCP: .... ...0 = No Fin
TCP: Window = 49188
TCP: Checksum = 0x2faa
TCP: Urgent pointer = 0
TCP: Options: (12 bytes)
TCP: - No operation
TCP: - No operation
TCP: - TS Val = 1532405, TS Echo = 2134177
TCP:

ETHER: ----- Ether Header -----
ETHER:
ETHER: Packet 6 arrived at 22:24:53.67211
ETHER: Packet size = 66 bytes
ETHER: Destination = 0:c:29:2e:ed:c1,
ETHER: Source = 0:c:29:e5:d:fb,
ETHER: Ethertype = 0800 (IP)
ETHER:
IP: ----- IP Header -----
IP:
IP: Version = 4
IP: Header length = 20 bytes
IP: Type of service = 0x00
IP: xxx. .... = 0 (precedence)
IP: ...0 .... = normal delay
IP: .... 0... = normal throughput
IP: .... .0.. = normal reliability
IP: .... ..0. = not ECN capable transport
IP: .... ...0 = no ECN congestion experienced
IP: Total length = 52 bytes
IP: Identification = 5709
IP: Flags = 0x4
IP: .1.. .... = do not fragment
IP: ..0. .... = last fragment
IP: Fragment offset = 0 bytes
IP: Time to live = 64 seconds/hops
IP: Protocol = 6 (TCP)
IP: Header checksum = a05a
IP: Source address = 192.168.1.104, ipc4.shanjing.com
IP: Destination address = 192.168.1.100, sun1.shanjing.com
IP: No options
IP:
TCP: ----- TCP Header -----
TCP:
TCP: Source port = 750
TCP: Destination port = 111
TCP: Sequence number = 3368322821
TCP: Acknowledgement number = 1322807580
TCP: Data offset = 32 bytes
TCP: Flags = 0x10
TCP: 0... .... = No ECN congestion window reduced
TCP: .0.. .... = No ECN echo
TCP: ..0. .... = No urgent pointer
TCP: ...1 .... = Acknowledgement
TCP: .... 0... = No push
TCP: .... .0.. = No reset
TCP: .... ..0. = No Syn
TCP: .... ...0 = No Fin
TCP: Window = 1856
TCP: Checksum = 0xe574
TCP: Urgent pointer = 0
TCP: Options: (12 bytes)
TCP: - No operation
TCP: - No operation
TCP: - TS Val = 2134179, TS Echo = 1532405
TCP:

ETHER: ----- Ether Header -----
ETHER:
ETHER: Packet 7 arrived at 22:24:53.67212
ETHER: Packet size = 858 bytes
ETHER: Destination = 0:c:29:e5:d:fb,
ETHER: Source = 0:c:29:2e:ed:c1,
ETHER: Ethertype = 0800 (IP)
ETHER:
IP: ----- IP Header -----
IP:
IP: Version = 4
IP: Header length = 20 bytes
IP: Type of service = 0x00
IP: xxx. .... = 0 (precedence)
IP: ...0 .... = normal delay
IP: .... 0... = normal throughput
IP: .... .0.. = normal reliability
IP: .... ..0. = not ECN capable transport
IP: .... ...0 = no ECN congestion experienced
IP: Total length = 844 bytes
IP: Identification = 27279
IP: Flags = 0x4
IP: .1.. .... = do not fragment
IP: ..0. .... = last fragment
IP: Fragment offset = 0 bytes
IP: Time to live = 64 seconds/hops
IP: Protocol = 6 (TCP)
IP: Header checksum = 4900
IP: Source address = 192.168.1.100, sun1.shanjing.com
IP: Destination address = 192.168.1.104, ipc4.shanjing.com
IP: No options
IP:
TCP: ----- TCP Header -----
TCP:
TCP: Source port = 111
TCP: Destination port = 750 (Sun RPC)
TCP: Sequence number = 1322806788
TCP: Acknowledgement number = 3368322821
TCP: Data offset = 32 bytes
TCP: Flags = 0x18
TCP: 0... .... = No ECN congestion window reduced
TCP: .0.. .... = No ECN echo
TCP: ..0. .... = No urgent pointer
TCP: ...1 .... = Acknowledgement
TCP: .... 1... = Push
TCP: .... .0.. = No reset
TCP: .... ..0. = No Syn
TCP: .... ...0 = No Fin
TCP: Window = 49232
TCP: Checksum = 0xf9c4
TCP: Urgent pointer = 0
TCP: Options: (12 bytes)
TCP: - No operation
TCP: - No operation
TCP: - TS Val = 1532405, TS Echo = 2134177
TCP:
RPC: ----- SUN RPC Header -----
RPC:
RPC: Record Mark: last fragment, length = 788
RPC: Transaction id = 450618384
RPC: Type = 1 (Reply)
RPC: This is a reply to frame 4
RPC: Status = 0 (Accepted)
RPC: Verifier : Flavor = 0 (None), len = 0 bytes
RPC: Accept status = 0 (Success)
RPC:
PMAP: ----- Portmapper -----
PMAP:
PMAP: Proc = 4 (Dump the mappings)
PMAP: Program Version Protocol Port
PMAP: 100000 4 6 111 PMAP
PMAP: 100000 3 6 111 PMAP
PMAP: 100000 2 6 111 PMAP
PMAP: 100000 4 17 111 PMAP
PMAP: 100000 3 17 111 PMAP
PMAP: 100000 2 17 111 PMAP
PMAP: 100024 1 17 32772 STATMON2
PMAP: 100024 1 6 32771 STATMON2
PMAP: 100133 1 17 32772 ?
PMAP: 100133 1 6 32771 ?
PMAP: 1073741824 1 6 32772 transient
PMAP: 100021 1 17 4045 NLM
PMAP: 100021 2 17 4045 NLM
PMAP: 100021 3 17 4045 NLM
PMAP: 100021 4 17 4045 NLM
PMAP: 100021 1 6 4045 NLM
PMAP: 100021 2 6 4045 NLM
PMAP: 100021 3 6 4045 NLM
PMAP: 100021 4 6 4045 NLM
PMAP: 100001 2 17 32773 RSTAT
PMAP: 100001 3 17 32773 RSTAT
PMAP: 100001 4 17 32773 RSTAT
PMAP: 100068 2 17 32774 CMSD
PMAP: 100068 3 17 32774 CMSD
PMAP: 100068 4 17 32774 CMSD
PMAP: 100068 5 17 32774 CMSD
PMAP: 100083 1 6 32775 ?
PMAP: 100002 2 6 32776 RUSERS
PMAP: 100002 3 6 32776 RUSERS
PMAP: 100002 2 17 32775 RUSERS
PMAP: 100002 3 17 32775 RUSERS
PMAP: 100011 1 17 32776 RQUOTA
PMAP: 300598 1 17 32779 ?
PMAP: 300598 1 6 32777 ?
PMAP: 805306368 1 17 32779 ?
PMAP: 805306368 1 6 32777 ?
PMAP: 100249 1 17 32780 ?
PMAP: 100249 1 6 32778 ?
PMAP: 38 maps
PMAP:

ETHER: ----- Ether Header -----
ETHER:
ETHER: Packet 8 arrived at 22:24:53.68476
ETHER: Packet size = 66 bytes
ETHER: Destination = 0:c:29:2e:ed:c1,
ETHER: Source = 0:c:29:e5:d:fb,
ETHER: Ethertype = 0800 (IP)
ETHER:
IP: ----- IP Header -----
IP:
IP: Version = 4
IP: Header length = 20 bytes
IP: Type of service = 0x00
IP: xxx. .... = 0 (precedence)
IP: ...0 .... = normal delay
IP: .... 0... = normal throughput
IP: .... .0.. = normal reliability
IP: .... ..0. = not ECN capable transport
IP: .... ...0 = no ECN congestion experienced
IP: Total length = 52 bytes
IP: Identification = 5711
IP: Flags = 0x4
IP: .1.. .... = do not fragment
IP: ..0. .... = last fragment
IP: Fragment offset = 0 bytes
IP: Time to live = 64 seconds/hops
IP: Protocol = 6 (TCP)
IP: Header checksum = a058
IP: Source address = 192.168.1.104, ipc4.shanjing.com
IP: Destination address = 192.168.1.100, sun1.shanjing.com
IP: No options
IP:
TCP: ----- TCP Header -----
TCP:
TCP: Source port = 750
TCP: Destination port = 111
TCP: Sequence number = 3368322821
TCP: Acknowledgement number = 1322807580
TCP: Data offset = 32 bytes
TCP: Flags = 0x11
TCP: 0... .... = No ECN congestion window reduced
TCP: .0.. .... = No ECN echo
TCP: ..0. .... = No urgent pointer
TCP: ...1 .... = Acknowledgement
TCP: .... 0... = No push
TCP: .... .0.. = No reset
TCP: .... ..0. = No Syn
TCP: .... ...1 = Fin
TCP: Window = 1856
TCP: Checksum = 0xe562
TCP: Urgent pointer = 0
TCP: Options: (12 bytes)
TCP: - No operation
TCP: - No operation
TCP: - TS Val = 2134196, TS Echo = 1532405
TCP:

ETHER: ----- Ether Header -----
ETHER:
ETHER: Packet 9 arrived at 22:24:53.68489
ETHER: Packet size = 66 bytes
ETHER: Destination = 0:c:29:e5:d:fb,
ETHER: Source = 0:c:29:2e:ed:c1,
ETHER: Ethertype = 0800 (IP)
ETHER:
IP: ----- IP Header -----
IP:
IP: Version = 4
IP: Header length = 20 bytes
IP: Type of service = 0x00
IP: xxx. .... = 0 (precedence)
IP: ...0 .... = normal delay
IP: .... 0... = normal throughput
IP: .... .0.. = normal reliability
IP: .... ..0. = not ECN capable transport
IP: .... ...0 = no ECN congestion experienced
IP: Total length = 52 bytes
IP: Identification = 27280
IP: Flags = 0x4
IP: .1.. .... = do not fragment
IP: ..0. .... = last fragment
IP: Fragment offset = 0 bytes
IP: Time to live = 64 seconds/hops
IP: Protocol = 6 (TCP)
IP: Header checksum = 4c17
IP: Source address = 192.168.1.100, sun1.shanjing.com
IP: Destination address = 192.168.1.104, ipc4.shanjing.com
IP: No options
IP:
TCP: ----- TCP Header -----
TCP:
TCP: Source port = 111
TCP: Destination port = 750
TCP: Sequence number = 1322807580
TCP: Acknowledgement number = 3368322822
TCP: Data offset = 32 bytes
TCP: Flags = 0x10
TCP: 0... .... = No ECN congestion window reduced
TCP: .0.. .... = No ECN echo
TCP: ..0. .... = No urgent pointer
TCP: ...1 .... = Acknowledgement
TCP: .... 0... = No push
TCP: .... .0.. = No reset
TCP: .... ..0. = No Syn
TCP: .... ...0 = No Fin
TCP: Window = 49232
TCP: Checksum = 0x2c51
TCP: Urgent pointer = 0
TCP: Options: (12 bytes)
TCP: - No operation
TCP: - No operation
TCP: - TS Val = 1532406, TS Echo = 2134196
TCP:

ETHER: ----- Ether Header -----
ETHER:
ETHER: Packet 10 arrived at 22:24:53.68522
ETHER: Packet size = 66 bytes
ETHER: Destination = 0:c:29:e5:d:fb,
ETHER: Source = 0:c:29:2e:ed:c1,
ETHER: Ethertype = 0800 (IP)
ETHER:
IP: ----- IP Header -----
IP:
IP: Version = 4
IP: Header length = 20 bytes
IP: Type of service = 0x00
IP: xxx. .... = 0 (precedence)
IP: ...0 .... = normal delay
IP: .... 0... = normal throughput
IP: .... .0.. = normal reliability
IP: .... ..0. = not ECN capable transport
IP: .... ...0 = no ECN congestion experienced
IP: Total length = 52 bytes
IP: Identification = 27281
IP: Flags = 0x4
IP: .1.. .... = do not fragment
IP: ..0. .... = last fragment
IP: Fragment offset = 0 bytes
IP: Time to live = 64 seconds/hops
IP: Protocol = 6 (TCP)
IP: Header checksum = 4c16
IP: Source address = 192.168.1.100, sun1.shanjing.com
IP: Destination address = 192.168.1.104, ipc4.shanjing.com
IP: No options
IP:
TCP: ----- TCP Header -----
TCP:
TCP: Source port = 111
TCP: Destination port = 750
TCP: Sequence number = 1322807580
TCP: Acknowledgement number = 3368322822
TCP: Data offset = 32 bytes
TCP: Flags = 0x11
TCP: 0... .... = No ECN congestion window reduced
TCP: .0.. .... = No ECN echo
TCP: ..0. .... = No urgent pointer
TCP: ...1 .... = Acknowledgement
TCP: .... 0... = No push
TCP: .... .0.. = No reset
TCP: .... ..0. = No Syn
TCP: .... ...1 = Fin
TCP: Window = 49232
TCP: Checksum = 0x2c50
TCP: Urgent pointer = 0
TCP: Options: (12 bytes)
TCP: - No operation
TCP: - No operation
TCP: - TS Val = 1532406, TS Echo = 2134196
TCP:

ETHER: ----- Ether Header -----
ETHER:
ETHER: Packet 11 arrived at 22:24:53.68535
ETHER: Packet size = 66 bytes
ETHER: Destination = 0:c:29:2e:ed:c1,
ETHER: Source = 0:c:29:e5:d:fb,
ETHER: Ethertype = 0800 (IP)
ETHER:
IP: ----- IP Header -----
IP:
IP: Version = 4
IP: Header length = 20 bytes
IP: Type of service = 0x00
IP: xxx. .... = 0 (precedence)
IP: ...0 .... = normal delay
IP: .... 0... = normal throughput
IP: .... .0.. = normal reliability
IP: .... ..0. = not ECN capable transport
IP: .... ...0 = no ECN congestion experienced
IP: Total length = 52 bytes
IP: Identification = 2
IP: Flags = 0x4
IP: .1.. .... = do not fragment
IP: ..0. .... = last fragment
IP: Fragment offset = 0 bytes
IP: Time to live = 64 seconds/hops
IP: Protocol = 6 (TCP)
IP: Header checksum = b6a5
IP: Source address = 192.168.1.104, ipc4.shanjing.com
IP: Destination address = 192.168.1.100, sun1.shanjing.com
IP: No options
IP:
TCP: ----- TCP Header -----
TCP:
TCP: Source port = 750
TCP: Destination port = 111
TCP: Sequence number = 3368322822
TCP: Acknowledgement number = 1322807581
TCP: Data offset = 32 bytes
TCP: Flags = 0x10
TCP: 0... .... = No ECN congestion window reduced
TCP: .0.. .... = No ECN echo
TCP: ..0. .... = No urgent pointer
TCP: ...1 .... = Acknowledgement
TCP: .... 0... = No push
TCP: .... .0.. = No reset
TCP: .... ..0. = No Syn
TCP: .... ...0 = No Fin
TCP: Window = 1856
TCP: Checksum = 0xe55e
TCP: Urgent pointer = 0
TCP: Options: (12 bytes)
TCP: - No operation
TCP: - No operation
TCP: - TS Val = 2134198, TS Echo = 1532406
TCP: