Resolving Poor ASE Performance Due to AIX Default Virtual Memory Paging
ASE delivers its best performance when all of its memory resides in physical memory on the machine.
It ASE’s memory is paged out by the operating system’s virtual memory subsystem then ASE performance
can suffer dramatically.
Poor performance due to memory paging on AIX is often reported as problems with ASE’s checkpoint and
/ or housekeeper tasks. These tasks may be slow, appear hung, or encounter timeslice errors.
Transaction dumps may also be affected as ASE cannot properly execute a checkpoint.
Paging particularly affects these tasks because they often process long lists of memory.
If ASE’s memory has been paged out access to it may require a read from disk, resulting in
poor performance. Any other ASE task that must traverse long lists of memory pages may be similarly affected,
resulting in timeslice errors. Logical lock contention may also increase as tasks
which hold locks take longer to run.
While this is a potential problem on all platforms, certain characteristics of AIX make it more likely
to occur on that platform. The default configuration of the AIX kernel allows the file system cache
to consume up to 80% of the available physical memory pages. Depending on system demand this may result
in ASE memory pages being copied to the swap device. Sybase recommends tuning the AIX virtual memory
subsystem to avoid this scenario.
The AIX values minperm and maxperm loosely control the ratio of page frames used for files versus
those used for computational processes (such as ASE). Tuning these values requires determining
the physical memory load of processes running on the host machine. maxperm should then be set
so that file pages do not interfere with process pages.
For example, consider an AIX host that has 8 Gb of physical memory and hosts two ASE servers.
One ASE server is configured to use 3 Gb of max memory and the other is configured to use 2 Gb of max memory.
Other applications on the host require a total of 1 Gb of memory. Therefore the total memory
requirement is 6 Gb (3 Gb + 2 Gb + 1 Gb). As 75% of this host’s physical memory is needed
for applications (6 Gb out of 8 Gb), maxperm should be set no higher than 25%.
In addition to configuring maxperm, Sybase highly recommends setting strict_maxperm to 1. When strict_maxperm
is set to 0 (the default value), AIX may override the maxperm setting at its discretion. Setting strict_maxperm
to 1 informs AIX that this is a hard limit.
Kernel parameters:
Note: Value for tunable maxperm% must be greater than or equal to the value of tunable maxclient%
=======================
maxclient%
Purpose:
Specifies maximum percentage of RAM that can be used for caching client pages.
Similar to maxperm% but cannot be bigger than maxperm%.
Values:
o Default: 80
o Range: 1 to 100%.
o Type: Dynamic
Diagnosis:
If J2 file pages or NFS pages are causing working storage pages to get paged
out, maxclient can be reduced.
Tuning
Decrease the value of maxclient if paging out to paging space is occurring due
to too many J2 client pages or NFS client pages in memory. Increasing the
value can allow more J2 or NFS client pages to be in memory before page
replacement starts.
Refer To:
Miscellaneous I/O Tuning Parameters
========================
maxperm%
Purpose:
Specifies the point above which the page-stealing algorithm steals only file
pages.
Values:
o Default: total number of memory frames * 0.8
o Range: 1 to 100
o Type: Dynamic
Diagnosis:
Monitor disk I/O with iostat n.
Tuning
This value is expressed as a percentage of the total real-memory page frames
in the system. Reducing this value may reduce or eliminate page replacement of
working storage pages caused by high number of file page accesses. Increasing
this value may help NFS servers that are mostly read-only. For example, if
some files are known to be read repetitively, and I/O rates do not decrease
with time from startup, maxperm may be too low.
Refer To:
Tuning VMM Page Replacement with the vmtune Command
---------------------------------------------------------------------------------------
maxfree
Purpose:
Specifies the number of frames on the free list at which page-stealing is to
stop.
Values:
o Default: 128
o Range: 16 to 204800
o Type: Dynamic
Diagnosis:
Observe free-list-size changes with vmstat n.
Tuning
If vmstat n shows free-list size frequently driven below minfree by
application demands, increase maxfree to reduce calls to replenish the free
list. Generally, keep maxfree - minfree equal to or less than 100. Setting the
value too high causes page replacement to run for a longer period of time.
Value must be at least 8 greater than minfree
-------------------------------------------------------------------------------------
maxpin%
Purpose:
Specifies the maximum percentage of real memory that can be pinned.
Values:
o Default: 80 percent
o Range: 1 to 99
o Type: Dynamic
Diagnosis:
Cannot pin memory, although free memory is available.
Tuning
If this value is changed, the new value should ensure that at least 4 MB of
real memory will be left unpinned for use by the kernel. The maxpin values
must be greater than one and less than 100. Change this parameter only in
extreme situations, such as maximum-load benchmarking.
---------------------------------------------------------------------------------------
minfree
Purpose:
Specifies the minimum number of frames on the free list at which the VMM
starts to steal pages to replenish the free list.
Values:
o Default: maxfree - 8
o Range: 8 to 204800
o Type: Dynamic
Diagnosis:
vmstat n
Tuning
Page replacement occurs when the number of free frames reaches minfree. If
processes are being delayed by page stealing, increase minfree to improve
response time. The difference between minfree and maxfree should always be
equal to or greater than maxpgahead.
-----------------------------------------------------------------------------------------
strict_maxperm
Purpose:
If set to 1, the maxperm value will be a hard limit on how much of RAM can be
used as a persistent file cache.
Values:
o Default: 0 (off)
o Range: 0 or 1.
o Type: Dynamic
Diagnosis:
Excessive page outs to page space caused by too many file pages in RAM.
Tuning
Set to 1 in order to make the maxperm value a hard limit (use in conjunction
with the tuning of the maxperm parameter).
Refer To:
Placing a Hard Limit on Persistent File Cache with strict_maxperm
Hands-on examples:
#vmo –a ; display all
#vmo –D ; reset all to default value
594 vmo -p -o maxclient%=50 ; good for both current and reboot values
594 vmo -r -o maxclient%=50 ; good for reboot values
595 vmo -o maxclient%=50 ; maxperm% must be larger/equal to maxclient %
596 vmo -o maxperm%=50 ; good for now
597 vmo -r -o maxperm%=50 ; good for reboot
602 vmo -r -o strict_maxperm=1 ; good for reboot
603 vmo -o strict_maxperm=1 ; good for now
604 vmo –a
1. To list the current and reboot value, range, unit, type and dependencies of
all tunables parameters managed by the vmo command, type:
vmo -L
2. To turn on and reserve 16MB large pages on a POWER4 system, type:
vmo -r -o lgpg_regions=10 -o lgpg_size=16777216
This command will propose bosboot to the user, and warn that a reboot is
necessary before the change will be effective.
3. To display help on nokilluid, type:
vmo -h nokilluid
4. To turn on v_pinshm after the next reboot, type:
vmo -r -o v_pinshm=1
5. To permanently reset all vmo tunable parameters to default, type:
vmo -p -D
6. To list the reboot value for all virtual Memory Manager tuning parameters,
type:
vmo -r -a
7. To list (spreadsheet format) the current and reboot value, range, unit,
type and dependencies of all tunables parameters managed by the vmo
command, type:
vmo -x
AIX Kernel-VM-Tuning
See memory usage
svmon -G -i 1 2
svmon -P pid
Kernel Tuning in AIX 5L Version 5.2 Performance Tools Guide and Reference.
Performance Overview of the Virtual Memory Manager
(VMM) and Tuning VMM Page Replacement
vmo -a ; display
vmo -d ; set to default value
vmo -o ; display or set a tunable to new value
vmo -p ; When used in combination with -o, -d or -D, makes changes apply to both
current and reboot values, that is, turns on the updating of the
/etc/tunables/nextboot file in addition to the updating of the current value.
These combinations cannot be used on Reboot and Bosboot type parameters becasue
their current value can't be changed.
3:root@sactgx0020:/var/log/nmonlog # vmo -a
memory_frames = 4194304
pinnable_frames = 3947805
maxfree = 288
minfree = 256
minperm% = 20
minperm = 808874
maxperm% = 80
maxperm = 3235503
strict_maxperm = 0
maxpin% = 80
maxpin = 3355444
maxclient% = 80
lrubucket = 131072
defps = 1
nokilluid = 0
numpsblks = 6684672
npskill = 52224
npswarn = 208896
v_pinshm = 0
pta_balance_threshold = n/a
pagecoloring = n/a
framesets = 2
mempools = 1
lgpg_size = 0
lgpg_regions = 0
num_spec_dataseg = 0
spec_dataseg_int = 512
memory_affinity = 1
htabscale = n/a
force_relalias_lite = 0
relalias_percentage = 0
data_stagger_interval = 161
large_page_heap_size = 0
kernel_heap_psize = 4096
soft_min_lgpgs_vmpool = 0
vmm_fork_policy = 0
low_ps_handling = 1
mbuf_heap_psize = 4096
strict_maxclient = 1
cpu_scale_memp = 8
lru_poll_interval = 0
lru_file_repage = 1
1:root@sactgx0010:/home/root # vmo -L
NAME CUR DEF BOOT MIN MAX UNIT TYPE
DEPENDENCIES
--------------------------------------------------------------------------------
memory_frames 3840K 3840K 4KB pages S
--------------------------------------------------------------------------------
pinnable_frames 3644K 3644K 4KB pages S
--------------------------------------------------------------------------------
maxfree 128 128 128 16 200K 4KB pages D
minfree
memory_frames
--------------------------------------------------------------------------------
minfree 120 120 120 8 200K 4KB pages D
maxfree
memory_frames
--------------------------------------------------------------------------------
minperm% 20 20 20 1 100 % memory D
maxperm%
--------------------------------------------------------------------------------
minperm 757803 757803 S
--------------------------------------------------------------------------------
maxperm% 80 80 80 1 100 % memory D
minperm%
maxclient%
--------------------------------------------------------------------------------
maxperm 2960K 2960K S
--------------------------------------------------------------------------------
strict_maxperm 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
maxpin% 80 80 80 1 99 % memory D
pinnable_frames
memory_frames
--------------------------------------------------------------------------------
maxpin 3M 3M S
--------------------------------------------------------------------------------
maxclient% 80 80 80 1 100 % memory D
maxperm%
--------------------------------------------------------------------------------
lrubucket 128K 128K 128K 64K 4KB pages D
--------------------------------------------------------------------------------
defps 1 1 1 0 1 boolean D
--------------------------------------------------------------------------------
nokilluid 0 0 0 0 4G-1 uid D
--------------------------------------------------------------------------------
numpsblks 7808K 7808K 4KB pages S
--------------------------------------------------------------------------------
npskill 61K 61K 61K 1 7M-1 4KB pages D
--------------------------------------------------------------------------------
npswarn 244K 244K 244K 0 7M-1 4KB pages D
--------------------------------------------------------------------------------
v_pinshm 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
pta_balance_threshold n/a 50 50 0 99 % pta segment R
--------------------------------------------------------------------------------
pagecoloring n/a 0 0 0 1 boolean B
--------------------------------------------------------------------------------
framesets 2 2 2 1 10 B
--------------------------------------------------------------------------------
mempools 1 1 1 1 4 B
--------------------------------------------------------------------------------
lgpg_size 0 0 0 0 256M bytes B
lgpg_regions
--------------------------------------------------------------------------------
lgpg_regions 0 0 0 0 B
lgpg_size
--------------------------------------------------------------------------------
num_spec_dataseg 0 0 0 0 B
--------------------------------------------------------------------------------
spec_dataseg_int 512 512 512 0 B
--------------------------------------------------------------------------------
memory_affinity 1 1 1 0 1 boolean B
--------------------------------------------------------------------------------
htabscale n/a -1 -1 -4 0 B
--------------------------------------------------------------------------------
force_relalias_lite 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
relalias_percentage 0 0 0 0 32K-1 D
--------------------------------------------------------------------------------
data_stagger_interval 161 161 161 0 0 4KB pages D
lgpg_size
--------------------------------------------------------------------------------
large_page_heap_size 0 0 0 0 8E-1 bytes B
lgpg_size
--------------------------------------------------------------------------------
kernel_heap_psize 4K 4K 4K 4K 16M bytes B
lgpg_size
--------------------------------------------------------------------------------
soft_min_lgpgs_vmpool 0 0 0 0 90 % D
lgpg_size
--------------------------------------------------------------------------------
vmm_fork_policy 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
low_ps_handling 1 1 1 1 2 D
--------------------------------------------------------------------------------
mbuf_heap_psize 4K 4K 4K 4K 16M bytes B
--------------------------------------------------------------------------------
strict_maxclient 1 1 1 0 1 boolean D
--------------------------------------------------------------------------------
cpu_scale_memp 8 8 8 1 64 B
--------------------------------------------------------------------------------
lru_poll_interval 0 0 0 0 60000 milliseconds D
--------------------------------------------------------------------------------
lru_file_repage 1 1 1 0 1 boolean D
--------------------------------------------------------------------------------
n/a means parameter not supported by the current platform or kernel
Parameter types:
S = Static: cannot be changed
D = Dynamic: can be freely changed
B = Bosboot: can only be changed using bosboot and reboot
R = Reboot: can only be changed during reboot
C = Connect: changes are only effective for future socket connections
M = Mount: changes are only effective for future mountings
I = Incremental: can only be incremented
Value conventions:
K = Kilo: 2^10 G = Giga: 2^30 P = Peta: 2^50
M = Mega: 2^20 T = Tera: 2^40 E = Exa: 2^60
