Shan's "Fix IT in 1 Minute!" UNIX Admin Blog

Always look on the bright side of Life!

2011-05-14T21:00:00.000-07:00

The best chapter of book 'Learning Python'

"Once you finish the quiz, you’ve officially reached the end of this book. Now that you
know Python inside and out, your next step, should you choose to take it, is to explore
the libraries, techniques, and tools available in the application domains in which you
work. Because Python is so widely used, you’ll find ample resources for using it in
almost any application you can think of—from GUIs, the Web, and databases to numeric
programming, robotics, and system administration.

This is where Python starts to become truly fun, but this is also where this book’s story
ends, and others’ begin. For pointers on where to turn after this book, see the list of
recommended follow-up texts in the Preface. Good luck with your journey. And of course,
“Always look on the bright side of Life!”

-Mark Lutz, the author of "Learning Python"

Configuring Python Matplotlib plot to GUI/X and Configuring TK for Python

2011-03-30T14:44:00.000-07:00

To configure Python Matplotlib to display to GUI/X. You need to configure Matplotlib backend to be 'TKAgg' (default is Agg).

Problem:
Most Python compilers delivered with major Linux distributions don't have tcl/tk configured during the build. To test, try 'import Tkinter' in your Python. The underlying shared object is _tkinter.so. Tkinter is required by Matplotlib to enable TKAgg backend.

If your Python is not configured for Tk, you will need to rebuild Python compiler with tcl/tk and possibly rebuild matplotlib.

The key of configuring tcl/tk for Python compiler is to include tcl/tk header files while building Python compiler.

If you don't have root on the system, the best way is to download tcl/tk. Otherwise, you can just rpm tcl/tk devel RPMs. In either cases, you should install the correct version of header files as your system tcl/tk binary/libs.

Below are the detail steps to build everything from source code, it takes about 1 hour.

1. Build tcl 8.4.13
Download at:
http://www.linuxfromscratch.org/blfs/view/6.2.0/general/tcl.html
use --prefix to install at a non-standard location, e.g.
./configure --prefix=/home/sjing/tools

2. Build tk

Download at: http://www.linuxfromscratch.org/blfs/view/6.2.0/general/tk.html
use ./configure --prefix to install at a non-standard location, e.g.
./configure --prefix=/home/sjing/tools

3. Build Python with tkinter and Unicode4 (check your current system python compiler's ucs setting, see Reference 1 below. This will ensure your new Python compiler is compatible with your system's)
./configure --enable-unicode=ucs4 --prefix=/home/sjing/tools

To include tcl/tk in Python, you need to compile Python with tcl/tk header files in the -I path.

If you download tcl/tk from the above sites (step 1, 2), please edit Makefile and add tcl.h/tk.h headers path in CPPFLAGS macro

for example: assume /home/sjing/tools/include has tk.h andtcl.h from step 1 and 2.

CPPFLAGS= -I. -I$(srcdir)/Include -I/home/sjing/tools/include

-bash-3.2$ rpm -qa | grep ^tk
tk-devel-8.4.13-5.el5_1.1
tk-8.4.13-5.el5_1.1
tk-devel-8.4.13-5.el5_1.1
tk-8.4.13-5.el5_1.1
-bash-3.2$ rpm -qa | grep ^tcl
tcl-8.4.13-4.el5
tcl-devel-8.4.13-4.el5
tcl-8.4.13-4.el5
tclx-8.4.0-5.fc6
tcl-devel-8.4.13-4.el5

Once make can find the header files, build it and see the process of
building _tkinter.so file.

-------------
make

building '_tkinter' extension
gcc -pthread -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -fno-strict-aliasing -DWITH_APPINIT=1 -I/usr/X11/include -I. -I/home/sjing/downloads/Python-2.4.3/./Include -I/home/sjing/tools/include -I/usr/local/include -I/home/sjing/downloads/Python-2.4.3/Include -I/home/sjing/downloads/Python-2.4.3 -c /home/sjing/downloads/Python-2.4.3/Modules/_tkinter.c -o build/temp.linux-x86_64-2.4/_tkinter.o
gcc -pthread -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -fno-strict-aliasing -DWITH_APPINIT=1 -I/usr/X11/include -I. -I/home/sjing/downloads/Python-2.4.3/./Include -I/home/sjing/tools/include -I/usr/local/include -I/home/sjing/downloads/Python-2.4.3/Include -I/home/sjing/downloads/Python-2.4.3 -c /home/sjing/downloads/Python-2.4.3/Modules/tkappinit.c -o build/temp.linux-x86_64-2.4/tkappinit.o
gcc -pthread -shared build/temp.linux-x86_64-2.4/_tkinter.o build/temp.linux-x86_64-2.4/tkappinit.o -L/usr/X11/lib -L/home/sjing/tools/lib -L/usr/local/lib -ltk8.4 -ltcl8.4 -lX11 -o build/lib.linux-x86_64-2.4/_tkinter.so
--------------

4. Build matplotlib
Edit setupext.py and add tcl/tk location in 'basdir' dictionary.
(x84_64 bit uses 'linux2' key)
50 basedir = {
...
57 'linux2' : ['/usr/local', '/usr','/home/sjing/tools'],
...
78 }

To confirm Tkinter will be built with matplotlib.
Make sure you use the Python compiler you just built with tcl/tk by adding the path of Python in front of your old python path.

-bash-3.2$ export PATH=/home/sjing/tools/bin:$PATH
( or /home/sjing/tools/bin/python setup.py build )

-bash-3.2$ python setup.py build
basedirlist is: ['/usr/local', '/usr', '/home/sjing/tools']
============================================================================
BUILDING MATPLOTLIB
matplotlib: 1.0.1
python: 2.4.3 (#1, Mar 29 2011, 14:57:37) [GCC 4.1.2
20080704 (Red Hat 4.1.2-48)]
platform: linux2

REQUIRED DEPENDENCIES
numpy: 1.2.0
freetype2: 9.10.3

OPTIONAL BACKEND DEPENDENCIES
libpng: 1.2.10
Tkinter: Tkinter: 39220, Tk: 8.4, Tcl: 8.4

Install matplotlib module:
-bash-3.2$ python setup.py install --prefix=/home/sjing/tools

5. Download matplotlib examples at http://matplotlib.sourceforge.net/examples/index.html

run the examples that use tk
animation example code: animate_decay_tk_blit.py
mplot3d example code: 2dcollections3d_demo.py

6. As long as your newly build and system python compilers are the same version and build with the same paramters (ucs), you should be able to run all your existing python codes/modules with the new compiler. But the best way is your System Adminsitrator update the system wide Python compile with tcl/tk following this guide.

Reference 1:

To check your Python is built with ucs2 or ucs4
1. ucs4
>>> import sys
>>> print sys.maxunicode
1114111

2. ucs2
>>> import sys
>>> print sys.maxunicode
65535

Reference 2:

matplotlibrc (lines defining backend and interactive mode)
--------------------------------------
backend : TKAgg
backend_fallback: True
interactive : False
------------------

Memcache and Binary Protocol

2011-03-28T11:23:00.000-07:00

A very good Memcache Story explains memcache basic.

Article about binary protocol.

bash login profiles for all scenarios

2011-02-15T10:08:00.001-08:00

When invoked as bash (not sh) the start-up files that bash reads for a
login shell are, in order:

i) /etc/profile

and then the first of any of the following that exists:

ii) ~/.bash_profile OR
iii) ~/.bash_login OR
iv) ~/.profile


Non-login interactive bash shells (when changing or starting new shells
during a session) read only ~/.bashrc.

Non-login non-interactive bash shells (ie-when launched from inside a
script) check the startup file in $BASH_ENV, or $ENV if that doesn't
exist.

An easy way to make sure all three invocations of bash reference the same
start-up file is:

i) use .bashrc as your customized startup file

ii) create a .bash_profile containing only:

 export BASH_ENV=~/.bashrc
 if [ -f ~/.bashrc ]; then source ~/.bashrc; fi



This way, after reading /etc/profile a login shell will read the first
file found, ~/.bash_profile, which in turn reads ~/.bashrc

Non-login interactive shells will read ~/.bashrc as usual

Non-login non-interactive shells will read the ~/.bashrc assigned in
$BASH_ENV.


Reference 1: BASH's sourcing profiles order in execution

[sjing@prodpmcf151 ~]$ bash -cx 'env'
+ '[' -f /etc/bashrc ']'
+ source /etc/bashrc
++ '[' 17934 -gt 99 ']'
+++ id -gn
+++ id -un
++ '[' techdept = sjing ']'
++ umask 022
++ '[' '' ']'
++ shopt -q login_shell
++ for i in '/etc/profile.d/*.sh'
++ '[' -r /etc/profile.d/colorls.sh ']'
++ '[' '' ']'
++ . /etc/profile.d/colorls.sh
++ for i in '/etc/profile.d/*.sh'
++ '[' -r /etc/profile.d/glib2.sh ']'
++ '[' '' ']'
++ . /etc/profile.d/glib2.sh
++ for i in '/etc/profile.d/*.sh'
++ '[' -r /etc/profile.d/gnome-ssh-askpass.sh ']'
++ '[' '' ']'
++ . /etc/profile.d/gnome-ssh-askpass.sh
++ for i in '/etc/profile.d/*.sh'
++ '[' -r /etc/profile.d/krb5-workstation.sh ']'
++ '[' '' ']'
++ . /etc/profile.d/krb5-workstation.sh
++ for i in '/etc/profile.d/*.sh'
++ '[' -r /etc/profile.d/lang.sh ']'
++ '[' '' ']'
++ . /etc/profile.d/lang.sh
++ for i in '/etc/profile.d/*.sh'
++ '[' -r /etc/profile.d/less.sh ']'
++ '[' '' ']'
++ . /etc/profile.d/less.sh
++ for i in '/etc/profile.d/*.sh'
++ '[' -r /etc/profile.d/vim.sh ']'
++ '[' '' ']'
++ . /etc/profile.d/vim.sh
++ for i in '/etc/profile.d/*.sh'
++ '[' -r /etc/profile.d/which-2.sh ']'
++ '[' '' ']'
++ . /etc/profile.d/which-2.sh
++ unset i
++ unset pathmunge
+ '[' '!' 1 ']'
+ env

Reference 2: UNIX/LINUX UNICODE FAQ
Reference 3: Useful 'find' Page

Blog Index

2010-10-03T10:52:00.001-07:00

Gauge Your Unix Skill Level!

Python Debugging Tips for pdb and IPython/ipdb

Solaris Administrations

AIX Administrations

Linux Administrations

Perl, Shell Scriptings

Web (Apache, Weblogic, Tomcat)

TCP/IP

Unix Internal and System Programming (C, Perl, Python)

Vim,mintty,vimshell, ubuntu, etc.

Installing Ubuntu 9.1 in vmware virtual host

Kerberos, NIS, DNS, NFS, etc.

Veritas Volume Manager and Veritas Cluster Server (VCS)

Open Source Projects

Hadoop/MapReduce (Cloud Computing)
Job Scheduler

Fresh Catches of the Day

vim Conque Shell (run bash, ipython, pdb inside vim!)

Python: Check Python Syntax with -m Switch

2010-08-27T09:52:00.000-07:00

Use Python -m switch and py_compile module to check Python syntax.

Python code with error at line 11


  1 #!/usr/bin/python
  2
  3 import os
  4 import time
  5
  6 prcfile="/home/sjing/prod/man/python/batch_cf/tmpfile"
  7 while True:
  8     fsize=int(os.path.getsize(prcfile))
  9     print "file %s size is : %d" % (prcfile,fsize)
 10     time.sleep(10)
 11     if fsize == int(os.path.getsize(prcfile))
 12         print "transaction is done. Exit\n"
 13         break


-bash-3.2$ python -m py_compile test.py
  File "test.py", line 11
    if fsize == int(os.path.getsize(prcfile))
                                             ^
SyntaxError: invalid syntax


python -v will disply module initialization message:

-bash-3.2$ python -v -m py_compile test.py
# installing zipimport hook
import zipimport # builtin
# installed zipimport hook
# /appl/mtg/dev/vendor/python/lib/python2.4/site-packages/site.pyc matches /appl/mtg/dev/vendor/python/lib/python2.4/site-packages/site.py
import site # precompiled from /appl/mtg/dev/vendor/python/lib/python2.4/site-packages/site.pyc
import imp # builtin
# /usr/lib64/python2.4/os.pyc matches /usr/lib64/python2.4/os.py
import os # precompiled from /usr/lib64/python2.4/os.pyc
import posix # builtin
# /usr/lib64/python2.4/posixpath.pyc matches /usr/lib64/python2.4/posixpath.py
import posixpath # precompiled from /usr/lib64/python2.4/posixpath.pyc
# /usr/lib64/python2.4/stat.pyc matches /usr/lib64/python2.4/stat.py
import stat # precompiled from /usr/lib64/python2.4/stat.pyc
import errno # builtin
# /usr/lib64/python2.4/UserDict.pyc matches /usr/lib64/python2.4/UserDict.py
import UserDict # precompiled from /usr/lib64/python2.4/UserDict.pyc
# /usr/lib64/python2.4/copy_reg.pyc matches /usr/lib64/python2.4/copy_reg.py
import copy_reg # precompiled from /usr/lib64/python2.4/copy_reg.pyc
# /usr/lib64/python2.4/types.pyc matches /usr/lib64/python2.4/types.py
import types # precompiled from /usr/lib64/python2.4/types.pyc
# /usr/lib64/python2.4/site.pyc matches /usr/lib64/python2.4/site.py
import site # precompiled from /usr/lib64/python2.4/site.pyc
import japanese # directory /usr/lib64/python2.4/site-packages/japanese
# /usr/lib64/python2.4/site-packages/japanese/__init__.pyc matches /usr/lib64/python2.4/site-packages/japanese/__init__.py
import japanese # precompiled from /usr/lib64/python2.4/site-packages/japanese/__init__.pyc
import japanese.aliases # directory /usr/lib64/python2.4/site-packages/japanese/aliases
# /usr/lib64/python2.4/site-packages/japanese/aliases/__init__.pyc matches /usr/lib64/python2.4/site-packages/japanese/aliases/__init__.py
import japanese.aliases # precompiled from /usr/lib64/python2.4/site-packages/japanese/aliases/__init__.pyc
import encodings # directory /usr/lib64/python2.4/encodings
# /usr/lib64/python2.4/encodings/__init__.pyc matches /usr/lib64/python2.4/encodings/__init__.py
import encodings # precompiled from /usr/lib64/python2.4/encodings/__init__.pyc
# /usr/lib64/python2.4/codecs.pyc matches /usr/lib64/python2.4/codecs.py
import codecs # precompiled from /usr/lib64/python2.4/codecs.pyc
import _codecs # builtin
# /usr/lib64/python2.4/encodings/aliases.pyc matches /usr/lib64/python2.4/encodings/aliases.py
import encodings.aliases # precompiled from /usr/lib64/python2.4/encodings/aliases.pyc
# zipimport: found 10 names in /appl/mtg/dev/vendor/python/lib64/python2.4/site-packages/python_sybase-0.39-py2.4-linux-x86_64.egg
# zipimport: found 50 names in /appl/mtg/dev/vendor/python/lib64/python2.4/site-packages/simplejson-2.0.9-py2.4-linux-x86_64.egg
# zipimport: found 77 names in /appl/mtg/dev/vendor/html5lib/lib/python/html5lib-0.90-py2.4.egg
# /usr/lib64/python2.4/new.pyc matches /usr/lib64/python2.4/new.py
import new # precompiled from /usr/lib64/python2.4/new.pyc
# /usr/lib64/python2.4/warnings.pyc matches /usr/lib64/python2.4/warnings.py
import warnings # precompiled from /usr/lib64/python2.4/warnings.pyc
# /usr/lib64/python2.4/linecache.pyc matches /usr/lib64/python2.4/linecache.py
import linecache # precompiled from /usr/lib64/python2.4/linecache.pyc
# /usr/lib64/python2.4/encodings/ascii.pyc matches /usr/lib64/python2.4/encodings/ascii.py
import encodings.ascii # precompiled from /usr/lib64/python2.4/encodings/ascii.pyc
Python 2.4.3 (#1, Jun 11 2009, 14:09:37)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
import marshal # builtin
# /usr/lib64/python2.4/traceback.pyc matches /usr/lib64/python2.4/traceback.py
import traceback # precompiled from /usr/lib64/python2.4/traceback.pyc
  File "test.py", line 11
    if fsize == int(os.path.getsize(prcfile))
                                             ^
SyntaxError: invalid syntax
# clear __builtin__._
# clear sys.path
# clear sys.argv
# clear sys.ps1
# clear sys.ps2
# clear sys.exitfunc
# clear sys.exc_type
# clear sys.exc_value
# clear sys.exc_traceback
# clear sys.last_type
# clear sys.last_value
# clear sys.last_traceback
# clear sys.path_hooks
# clear sys.path_importer_cache
# clear sys.meta_path
# restore sys.stdin
# restore sys.stdout
# restore sys.stderr
# cleanup __main__
# cleanup[1] japanese
# cleanup[1] errno
# cleanup[1] imp
# cleanup[1] _codecs
# cleanup[1] new
# cleanup[1] zipimport
# cleanup[1] warnings
# cleanup[1] site
# cleanup[1] signal
# cleanup[1] traceback
# cleanup[1] linecache
# cleanup[1] posix
# cleanup[1] marshal
# cleanup[1] japanese.aliases
# cleanup[1] encodings
# cleanup[1] encodings.ascii
# cleanup[1] codecs
# cleanup[1] types
# cleanup[1] encodings.aliases
# cleanup[1] exceptions
# cleanup[2] copy_reg
# cleanup[2] posixpath
# cleanup[2] os.path
# cleanup[2] stat
# cleanup[2] UserDict
# cleanup[2] os
# cleanup sys
# cleanup __builtin__
# cleanup ints: 810 unfreed ints in 21 out of 26 blocks
# cleanup floats


Correct the code's error and check again

  1 #!/usr/bin/python
  2
  3 import os
  4 import time
  5
  6 prcfile="/home/sjing/prod/man/python/batch_cf/tmpfile"
  7 while True:
  8     fsize=int(os.path.getsize(prcfile))
  9     print "file %s size is : %d" % (prcfile,fsize)
 10     time.sleep(10)
 11     if fsize == int(os.path.getsize(prcfile)):
 12         print "transaction is done. Exit\n"
 13         break


-bash-3.2$ python -m py_compile test.py
-bash-3.2$

No error found.

Blog Index

2010-06-27T18:17:00.000-07:00

Gauge Your Unix Skill Level!

Python Debugging Tips for pdb and IPython/ipdb

Solaris Administrations

AIX Administrations

Linux Administrations

Perl, Shell Scriptings

Web (Apache, Weblogic, Tomcat)

TCP/IP

Unix Internal and System Programming (C, Perl, Python)

Vim,mintty,vimshell, ubuntu, etc.

Installing Ubuntu 9.1 in vmware virtual host

Kerberos, NIS, DNS, NFS, etc.

Veritas Volume Manager and Veritas Cluster Server (VCS)

Open Source Projects

Hadoop/MapReduce (Cloud Computing)
Job Scheduler

Fresh Catches of the Day

vim Conque Shell (run bash, ipython, pdb inside vim!)

Debugging Python in ipython and ipdb

2010-06-27T17:23:00.000-07:00

In this blog, I'm going to demonstrate some very useful features of ipython, especially when debugging python using ipdb; examining objects and step into pdb/ipdb from Python code, etc.

You should know how to use pdb to debug python, if not, please visit my other blog article.

I'll briefly describe how to exam objects and name space in iPython and then demonstrate how to debug Python in iPython/ipdb.

Note: a quick link to ipython. (make sure you come back here after reading the external page)

(*** Very Important ***)
If your ipython is 0.10 or earlier version, you need to do the following to fix a bug in checkline() method in IPython/Debugger.py.

shan@ub1:~/code$ python
Python 2.6.4 (r264:75706, Dec 7 2009, 18:45:15)
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import IPython
>>> IPython.Debugger.__file__
'/usr/local/lib/python2.6/dist-packages/IPython/Debugger.py'

wget the updated Debugger.py from here and replace the above Debugger.py with the new one. That's it.

Let's start with examining object and name space in ipython:

1. Start ipdb from ipython: (a very useful way to debug python code)

First, let's start ipython

shan@ub1:~/code$ ipython
Python 2.6.4 (r264:75706, Dec  7 2009, 18:45:15)
Type "copyright", "credits" or "license" for more information.

IPython 0.10 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.

In [1]:

The first few lines already show the most important tricks of ipython. The problem is most people rarely read it!

Now, let's import test code and see some of the features python IDLE doesn't offer.

shan@ub1:~/code$ ipython
Python 2.6.4 (r264:75706, Dec  7 2009, 18:45:15)
Type "copyright", "credits" or "license" for more information.

IPython 0.10 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.

In [1]: import test2

In [2]: import sys

In [3]: import os

In [4]: who
os      sys     test2

In [5]: whos
Variable   Type      Data/Info
------------------------------
os         module    
sys        module    
test2      module    

In [6]: test2?
Type:           module
Base Class:     
String Form:    
Namespace:      Interactive
File:           /home/shan/code/test2.py
Docstring:



In [7]: test2??
Type:           module
Base Class:     
String Form:    
Namespace:      Interactive
File:           /home/shan/code/test2.py
Source:
#/usr/bin/env python

class Employee:
def __init__(self, name, title=None, phone=5551212):
self.name = name
self.title = title
self.phone = phone

def employee():
shan = Employee("Shan Jing","UNIX ADMIN","2720748")
import pdb;pdb.set_trace()
joe = Employee("Joe Smith","Application Developer","7207837")
print (shan.name, shan.title, shan.phone)
print (joe.name, joe.title, joe.phone)


if __name__ == '__main__':
employee()

In [8]:

In the above, we typed 7 commands (3 imports, who, whos, test2? and test2??):

From the output, I'm sure you already know what each command does. But just in case you need to know the detail about these commands, type '%cmd?' in ipython. (see below). By the way, this was printed in the last line of the ipython's startup message - "object? -> Details about 'object'. ?? prints more details about the object."

In [9]: %who?
Type:           Magic function
Base Class:     
String Form:    >
Namespace:      IPython internal
File:           /usr/local/lib/python2.6/dist-packages/IPython/Magic.py
Definition:     %who(self, parameter_s='')
Docstring:
 Print all interactive variables, with some minimal formatting.

 If any arguments are given, only variables whose type matches one of
 these are printed.  For example:

   %who function str

 will only list functions and strings, excluding all other types of
 variables.  To find the proper type names, simply use type(var) at a
 command line to see how python prints type names.  For example:

   In [1]: type('hello')
   Out[1]: 

 indicates that the type name for strings is 'str'.

 %who always excludes executed names loaded through your configuration
 file and things which are internal to IPython.

 This is deliberate, as typically you may load many modules and the
 purpose of %who is to show you only what you've manually defined.

whos gives extra information about an object.

Next, let's see how to use ipython to debug our code.

If you are already inside ipython, to start pdb/ipdb, all you need to do is run the code with -d option:

In [14]: run -d test2.py
*** Blank or comment
*** Blank or comment
Breakpoint 1 at /home/shan/code/test2.py:3
NOTE: Enter 'c' at the ipdb>  prompt to start your script.
> (1)()

ipdb> c
> /home/shan/code/test2.py(3)()
2
1---> 3 class Employee:
4     def __init__(self, name, title=None, phone=5551212):

ipdb> l
1 #/usr/bin/env python

2
1---> 3 class Employee:
4     def __init__(self, name, title=None, phone=5551212):
5        self.name = name
6        self.title = title
7        self.phone = phone
8
9 def employee():
10     shan = Employee("Shan Jing","UNIX ADMIN","2720748")
11     import pdb;pdb.set_trace()
ipdb> h

Documented commands (type help ):
========================================
EOF    bt         cont      enable  jump  pdef   r        tbreak   w
a      c          continue  exit    l     pdoc   restart  u        whatis
alias  cl         d         h       list  pinfo  return   unalias  where
args   clear      debug     help    n     pp     run      unt
b      commands   disable   ignore  next  q      s        until
break  condition  down      j       p     quit   step     up

Miscellaneous help topics:
==========================
exec  pdb

Undocumented commands:
======================
retval  rv

As we can see, 'run -d code' brings us into ipdb. The rest is same as pdb except ipdb offers good features like command line completion, etc.

2. Set ipdb breakpoints from Python source code

First, you can always set pdb breakpoints as I described in a previous blog. After you pdb.set_trace(), when you run codes from ipython, it will open pdb instead of ipdb. The differences between pdb and ipdb is ipdb offers some features that ipython offers while pdb doesn't.

Now, if you want to set breakpoint and opens ipdb inside ipython when your code reaches the breakpoint, all you need to do is:

insert the following code at your breakpoint:

from IPython.Debugger import Tracer; breakpoint1 = Tracer()

Challenge: Write a vim key map to map F1 key in insert mode to insert a breakpoint in your source code without the need of typing the above import code. Also, map F6 key to open a shell in a split window then start ipython/ipdb to debug the code you are editting. Hint: vim's % variable and ConqueShell or Vimshell.

Answer: please refer to my previous blog Debugging Python with pdb.

Debugging Python in pdb and setup breakpoints from Python code

2010-06-27T15:51:00.000-07:00

This blog describes some techniques/tips on how to debug Python using the Python debugger pdb.

My "Vim Python IDE"
The following screen shot shows VIM configured as an almost full function Python IDE - there are 3 windows in VIM. I edit source code in the upper left window and open the lower left window for ipython, which shows imported objects in current namespace; finally, I have pdb debugging the source code at the right side window. All these can be done with a single key stroke inside VIM! (click the image to enlarge and click again to maximize it)

First, I'll show how to use basic Python pdb commands to debug Python code; then I'll show how to setup pdb breakpoints from the source code.

Then, I'll list vim key mapping that inserts breakpoints by just pressing F1 in vim and use F5 key to debug Python code without leaving Vim editor. Use this technique, you can debug Python code and travel back-and-forth between vim and pdb quickly.

At the end, I'll list additional vim key mappings (Thanks ConqueShell!!) that enable a single key stroke to trigger a pdb/ipython/ipdb in a split window inside Vim. This will turn your Vim into a full function Python IDE, you can edit and debug source code, evaluate attributes, objects in one vim screen!

Assume I'm working on the follow code and want to debug it:

shan@ub1:~/code$ nl test.py
1  #/usr/bin/env python

2  class Employee:
3      def __init__(self, name, title=None, phone=5551212):
4         self.name = name
5         self.title = title
6         self.phone = phone

7  def employee():
8      shan = Employee("Shan Jing","UNIX ADMIN","2720748")
9      joe = Employee("Joe Smith","Application Developer","7207837")
10      print (shan.name, shan.title, shan.phone)
11      print (joe.name, joe.title, joe.phone)


12  if __name__ == '__main__':
13      employee()

1. Use pdb to debug python (from Python IDLE):

Basic procedures:
a. start python IDLE
b. import pdb module
c. import python-code
d. type pdb.run('module.method()') and you will stop at (Pdb) prompt:

For exmaple:


shan@ub1:~/code$ python
Python 2.6.4 (r264:75706, Dec  7 2009, 18:45:15)
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pdb
>>> import test
>>> pdb.run('test.employee()')
> (1)()->None
(Pdb)

Now, type pdb command 's' to step into your code.


(Pdb) s
--Call--
> /home/shan/code/test.py(9)employee()

The line
>/home/shan/code/test.py(9)employee()
means the we stop at code /home/shan/code/test.py line 9 which is employee() method/function.

The "--Call--" indicates we step into a function/method, if you don't want to step into a function but want to execute it, use the 'n' command instead (explained later)

type pdb command 'l' to list source code. If you want to see more lines of code, you can type 'l range' eg 'l 12,22' to display line 12 to 22.


(Pdb) l
4         def __init__(self, name, title=None, phone=5551212):
5            self.name = name
6            self.title = title
7            self.phone = phone
8
9  -> def employee():
10         shan = Employee("Shan Jing","UNIX ADMIN","2720748")
11         joe = Employee("Joe Smith","Application Developer","7207837")
12         print (shan.name, shan.title, shan.phone)
13         print (joe.name, joe.title, joe.phone)
14

Please note, from the above output, the following line shows the NEXT line of code Python is about to execute.


'9  -> def employee():'

When using 's' (step command), it's good to know two other related commands - 'n' and 'c'.

Command 'n' (next) will execute the command one line at a time without stepping into functions.

Command 'c' (continue) will continue to execute until the end of current function.

Another useful pdb command is 'w' (where), it shows how the python compiler gets to the current line. It's very useful if your current position is nested.

Finally, to inspect attribute's value, use p (print) command. Or just type attribute's name.

Here's a complete run:


shan@ub1:~/code$ python
Python 2.6.4 (r264:75706, Dec  7 2009, 18:45:15)
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pdb
>>> import test
>>> pdb.run('test.employee()')
> (1)()
(Pdb) s
--Call--
> /home/shan/code/test.py(9)employee()
-> def employee():
(Pdb) l
4         def __init__(self, name, title=None, phone=5551212):
5            self.name = name
6            self.title = title
7            self.phone = phone
8
9  -> def employee():
10         shan = Employee("Shan Jing","UNIX ADMIN","2720748")
11         joe = Employee("Joe Smith","Application Developer","7207837")
12         print (shan.name, shan.title, shan.phone)
13         print (joe.name, joe.title, joe.phone)
14
(Pdb) s
> /home/shan/code/test.py(10)employee()
-> shan = Employee("Shan Jing","UNIX ADMIN","2720748")
(Pdb) s
--Call--
> /home/shan/code/test.py(4)__init__()
-> def __init__(self, name, title=None, phone=5551212):
(Pdb) s
> /home/shan/code/test.py(5)__init__()
-> self.name = name
(Pdb) s
> /home/shan/code/test.py(6)__init__()
-> self.title = title
(Pdb) s
> /home/shan/code/test.py(7)__init__()
-> self.phone = phone
(Pdb) s
--Return--
> /home/shan/code/test.py(7)__init__()->None
-> self.phone = phone
(Pdb) s
> /home/shan/code/test.py(11)employee()
-> joe = Employee("Joe Smith","Application Developer","7207837")
(Pdb) shan.title
'UNIX ADMIN'
(Pdb) print shan.phone
2720748
(Pdb) w
/usr/lib/python2.6/bdb.py(368)run()
-> exec cmd in globals, locals
(1)()
> /home/shan/code/test.py(11)employee()
-> joe = Employee("Joe Smith","Application Developer","7207837")
(Pdb) c
('Shan Jing', 'UNIX ADMIN', '2720748')
('Joe Smith', 'Application Developer', '7207837')

2. Setting breakpoints from Python code

Sometimes, you don't want to go thru the entire code line by line to debug. All you need is to jump into certain point in your code and debug that section only, you can do that by setting up a breakpoint in Python code.

Just add 'import pdb;pdb.set_trace()' at the point where you want to mark as breakpoint then run your code as normal, it will stop at the breakpoint and gives you pdb prompt.

See below line 9:


1  #/usr/bin/env python

2  class Employee:
3      def __init__(self, name, title=None, phone=5551212):
4         self.name = name
5         self.title = title
6         self.phone = phone

7  def employee():
8      shan = Employee("Shan Jing","UNIX ADMIN","2720748")
9      import pdb;pdb.set_trace()
10      joe = Employee("Joe Smith","Application Developer","7207837")
11      print (shan.name, shan.title, shan.phone)
12      print (joe.name, joe.title, joe.phone)


13  if __name__ == '__main__':
14      employee()


shan@ub1:~/code$ python test2.py
> /home/shan/code/test2.py(12)employee()
-> joe = Employee("Joe Smith","Application Developer","7207837")
(Pdb) l
7            self.phone = phone
8
9     def employee():
10         shan = Employee("Shan Jing","UNIX ADMIN","2720748")
11         import pdb;pdb.set_trace()
12  ->     joe = Employee("Joe Smith","Application Developer","7207837")
13         print (shan.name, shan.title, shan.phone)
14         print (joe.name, joe.title, joe.phone)
15
16
17     if __name__ == '__main__':
(Pdb) w
/home/shan/code/test2.py(18)()
-> employee()
> /home/shan/code/test2.py(12)employee()
-> joe = Employee("Joe Smith","Application Developer","7207837")
(Pdb) shan.title
'UNIX ADMIN'
(Pdb) c
('Shan Jing', 'UNIX ADMIN', '2720748')
('Joe Smith', 'Application Developer', '7207837')
shan@ub1:~/code$

Assume you are editing your code in Vim editor, wouldn't it be nice if just hit one key to set a breakpoint and hit another key to debug the code without leaving Vim?

Put the following lines in ~/.vimrc


"pdb setting : insert pdb breakpoints
imap <F1> import pdb;pdb.set_trace()
au BufRead *.py nmap <F5> :!python %

Now, at Vim's insert mode, find the place you want to insert a breakpoint, hit F1 key. To debug your code, (save it first) and hit F5 key, you will be stopped at the breakpoint in a shell, once done debugging just type 'exit' to close pdb and go back to vim continue your coding.

If you have conqueshell installed in vim, using the following Vim mappings, you will be able to open ipython and debug your code from a new buffer in a split window on the same screen on vim. Basically, you turn Vim into a dynamic Python/iPython IDE with a full function debugger!


" ConqueShell Mappings
nmap <F1> :ConqueTermSplit bash
nmap <F2> :ConqueTermVSplit bash
nmap <F3> :ConqueTermSplit ipython
nmap <F4> :ConqueTermVSplit ipython
nmap <F6> :execute 'ConqueTermSplit ipython '.expand('%:p')
nmap <F7> :execute 'ConqueTermVSplit ipython '.expand('%:p')
let g:ConqueTerm_EscKey = ''

I have posted my .vimrc in another blog.

If you want to use ipython (which is a much better IDE) and ipdb to debug Python, please read my next blog - Debugging Python with ipython and ipdb.

.vimrc for Python

2010-06-14T21:40:00.000-07:00


"My .vimrc file (Shan Jing)
"
" Part 1. Python Vim settings
" Part 2. Misc Settings
" Part 3. Global Format Settings

" <F1> opens bash in a horizonal split window
" <F2> opens bash in a vertical split window
" <F3> opens ipython in a horizonal split window
" <F4> opens ipython in a vertical split window
" <F5> debug/run current python file in a shell
" <c-,> opens a shell in a buffer/work with tabnew
" <c-w>t<c-w>H swap windows shapes H->V
" <c-w>t<c-w>K swap windows shapes V->H
" :map or :map! to show current mappings
" :nmap - Display normal mode maps
" :imap - Display insert mode maps
" :vmap - Display visual and select mode maps
" :smap - Display select mode maps
" :xmap - Display visual mode maps
" :cmap - Display command-line mode maps
" :omap - Display operator pending mode maps


"1. Python vim settings
set tabstop=4       " numbers of spaces of tab character
set shiftwidth=4    " numbers of spaces to (auto)indent
set smarttab
set expandtab       " tabs are converted to spaces,use only when  required
set autoindent
set smartindent


if has("autocmd")
    " clear au, note: au = autocmd
    au!
    " Restore cursor position
    au BufReadPost * if line("'\"") > 0|if line("'\"") <=  line("$")|exe("norm '\"")|else|exe "norm $"|endif|endif

"omnicompletion
filetype plugin on
set ofu=syntaxcomplete#Complete
    au FileType python set omnifunc=pythoncomplete#Complete
    au FileType javascript set omnifunc=javascriptcomplete#CompleteJS
    au FileType html set omnifunc=htmlcomplete#CompleteTags
    au FileType css set omnifunc=csscomplete#CompleteCSS
    au FileType xml set omnifunc=xmlcomplete#CompleteTags
    au FileType php set omnifunc=phpcomplete#CompletePHP
    au FileType c set omnifunc=ccomplete#Complete

"http://vim.wikia.com/wiki/Python_-_check_syntax_and_run_script
"nmap <buffer> <F5> :wmwG:r!python %<cr>`.
   au BufRead *.py set makeprg=python\ -c\ \"import\ py_compile,sys;\  sys.stderr=sys.stdout;\ py_compile.compile(r'%')\"
   au BufRead *.py set efm=%C\ %.%#,%A\ \ File\ \"%f\"\\,\ line\  %l%.%#,%Z%[%^\ ]%\\@=%m
   au BufRead *.py nmap <F5> :!python %<cr>
   au BufRead *.py set nowrap
   au BufRead *.py set go+=b
   au BufRead *.py set tabstop=4
"Python settings: auto indent the next line after these key words:
   au BufRead *.py set smartindent cinwords=def,if,elif,else,for,while,try,except,finall

   " Automatically chmod +x Python, Perl and Shell scripts
   au BufWritePost   *.py             !chmod +x %
   au BufWritePost   *.sh             !chmod +x %
   au BufWritePost   *.pl             !chmod +x %

   "remove all training space when editing
   "au BufWritePre *.py :%s/\s\+$//e
endif

"3. Other global setttings
"3a. A Python Calculator
"3b. ConqueShell Mappings
"3c. vim 7+ tabs settings
"3d. color scheme settings
"3e. general coding settings for c,cpp,php,java, etc
"3f. global format settings


"A Python Calculator for python aware vim
"usage : command mode: Calc x*y
:command! -nargs=+ Calc :py print <args>
:py from math import *

" ConqueShell Mappings
nmap <F1> :ConqueTermSplit bash<cr>
nmap <F2> :ConqueTermVSplit bash<cr>
nmap <F3> :ConqueTermSplit ipython<cr>
nmap <F4> :ConqueTermVSplit ipython<cr>
nmap <F6> :ConqueTerm bash -o vi<cr>
let g:ConqueTerm_EscKey = '<c-k>'

"map <F9> :previous<cr>  " map F9 to open previous buffer
"map <F10> :next<cr>      " map F10 to open next buffer

" turn off highlighted search
map <silent> <c-h> :silent noh<cr>

"editting and sourcing .vimrc
map ,e :vsp ~/.vimrc<cr>      " edit my .vimrc file
map ,u :source ~/.vimrc<cr>   " update vim setting

"insert the directory name of the current buffer
imap <2> <c-r>=expand('%:p:h')<cr>

"For vim 7.0+ tabs limit (default is 10)
set tabpagemax=15
"open a new tab
map ,t :tabnew<cr>



"command mode mappings
"insert the directory name of the current buffer at command mode
"e.g., :!ls <2> will do ls dir at current cursor
cmap <F2> <c-r>=expand('%:p:h')<cr>


"color scheme setting
if has("gui_running")
    " See ~/.gvimrc
    set guifont=Monospace\ 10  " use this font
    set lines=50       " height = 50 lines
    set columns=100        " width = 100 columns
    set background=light   " adapt colors for background
    set selectmode=mouse,key,cmd
    set keymodel=
else
    colorscheme elflord    " use this color scheme
    set background=dark        " adapt colors for background
endif

if has("autocmd")
    " Filetypes (au = autocmd)
    au FileType helpfile set nonumber      " no line numbers when  viewing help
    au FileType helpfile nnoremap <buffer><cr> <c-]>    " Enter selects subject
    au FileType helpfile nnoremap <buffer><bs> <c-t>    " Backspace to go back

    " When using mutt, text width=72
    au FileType mail,tex set textwidth=72
    au FileType cpp,c,java,sh,pl,php,asp  set autoindent
    au FileType cpp,c,java,sh,pl,php,asp  set smartindent
    au FileType cpp,c,java,sh,pl,php,asp  set cindent
    "au BufRead mutt*[0-9] set tw=72


    " File formats
    au BufNewFile,BufRead  *.pls    set syntax=dosini
    au BufNewFile,BufRead  modprobe.conf    set syntax=modconf
    "remove all training space when editing
    au BufWritePre *.py :%s/\s\+$//e
endif

"Global format settings
set nocompatible    " use vim defaults
set ls=2            " allways show status line
set scrolloff=3     " keep 3 lines when scrolling
set showcmd         " display incomplete commands
set hlsearch        " highlight searches
set incsearch       " do incremental searching
set ruler           " show the cursor position all the time
set visualbell t_vb=    " turn off error beep/flash
set novisualbell    " turn off visual bell
set nobackup        " do not keep a backup file
set number          " show line numbers
set ignorecase      " ignore case when searching
"set noignorecase   " don't ignore case
set title           " show title in console title bar
set ttyfast         " smoother changes
"set ttyscroll=0        " turn off scrolling, didn't work well with  PuTTY
set modeline        " last lines in document sets vim mode
set modelines=3     " number lines checked for modelines
set shortmess=atI   " Abbreviate messages
set nostartofline   " don't jump to first character when paging
set whichwrap=b,s,h,l,<,>,[,]   " move freely between files


syntax on

Solaris 10 Performance Tuning - Analyzing CPU

2010-05-17T15:16:00.000-07:00


Solaris 10 Performance Tuning – Analyzing CPU

Tools :
psrinfo
uptime
vmstat
sar -u
sar -q  (cpu queue, same as vmstat kthr:r)
mpstat
dtrace



$ psrinfo -vp
The physical processor has 8 virtual processors (32-39)
  SPARC64-VII (portid 1056 impl 0x7 ver 0x91 clock 2400 MHz)
The physical processor has 8 virtual processors (40-47)
  SPARC64-VII (portid 1064 impl 0x7 ver 0x91 clock 2400 MHz)
The physical processor has 8 virtual processors (48-55)
  SPARC64-VII (portid 1072 impl 0x7 ver 0x91 clock 2400 MHz)
The physical processor has 8 virtual processors (56-63)
  SPARC64-VII (portid 1080 impl 0x7 ver 0x91 clock 2400 MHz)



$ uptime
  1:04pm  up 114 day(s), 14:15,  6 users,  load average: 4.41, 4.39, 3.76

the intervals are 1,5,15 minutes



$ vmstat 5
 kthr      memory            page            disk          faults      cpu
 r b w   swap    free     re  mf pi po fr de sr s0 s1 sd sd   in   sy     cs    us sy id
 0 0 0 107752976 11913152 74 296 901 2 2  0  0  2  2 -0 -349 5960  727992 5910  9  2 89
 0 0 0 106811152 8983688  8  31  2   0 0  0  0  0  0  0  2   8818  615705 13764 13 2 85
 0 0 0 106701904 8884808  0  14  0   0 0  0  0  0  0  0  2   14250 584633 24692 12 3 85
 0 0 0 106535552 8730784  9  87  0   0 0  0  0  9  9  0  0   9247  379414 14642 12 2 87
 0 0 0 106403648 8603144  0  10  0   0 0  0  0  0  0  0  0   6668  550100 9664  12 2 87
 0 0 0 106296328 8504712  0  5   0   0 0  0  0  0  0  0  0   10490 471906 17677 12 2 85
 0 0 0 106277472 8510232  1  3   6   0 0  0  0  0  0  0  0   12011 535986 20477 11 2 87
 0 0 0 106277472 8527712  3  37  0   0 0  0  0  0  0  0  0   13685 668057 23689 9  2 89


The first line is the summary since boot. vmstat reads its statistic from kstat,
which maintains CPU utilizztion statistics for each CPU.

cpu:id : for percent idle.
kthr:r : the total number of threads on the ready to run queues, CPU saturation.

faults:in : number of interrupts/sec
faults:sys: number of system calls/sec
faults:cs : number of context switches/sec

cpu:us  : percent user time/user-mode threads
cpu:sy  : percent system time
cpu:id  : percent idle


sar : turn on sar if it's not on.

$ svcs -a | grep sar
online         Jan_22   svc:/system/sar:default
$ svcs -l svc:/system/sar:default
fmri         svc:/system/sar:default
name         system activity reporting package
enabled      true
state        online
next_state   none
state_time   Fri Jan 22 22:08:22 2010
logfile      /var/svc/log/system-sar:default.log
restarter    svc:/system/svc/restarter:default
dependency   require_all/none svc:/system/filesystem/minimal (online)



$ sar 5 10

SunOS betapm 5.10 Generic_142900-03 sun4u    05/17/2010

13:07:48    %usr    %sys    %wio   %idle
13:07:53      13       4       0      84
13:07:58      11       4       0      85
13:08:03       9       3       0      88
13:08:08       9       3       0      88
13:08:13      11       4       0      85
13:08:18      11       3       0      86
13:08:23      10       3       0      86
13:08:28       9       3       0      88
13:08:33       9       3       0      88
13:08:38      11       3       0      86

Average       10       3       0      86


%usr vs %sys : 70% vs 30%

%wio is always 0 in solaris 10 (disabled)


$ sar -q 2 10

SunOS betapm 5.10 Generic_142900-03 sun4u    05/17/2010

13:11:34 runq-sz %runocc swpq-sz %swpocc
13:11:36     1.0      50     0.0       0
13:11:38     0.0       0     0.0       0
13:11:40     0.0       0     0.0       0
13:11:42     0.0       0     0.0       0
13:11:44     0.0       0     0.0       0
13:11:46     0.0       0     0.0       0
13:11:48     0.0       0     0.0       0
13:11:50     1.0      50     0.0       0
13:11:52     0.0       0     0.0       0
13:11:54     0.0       0     0.0       0

Average      1.0      10     0.0       0


runq-sz: run queue size , same as vmstat kthr:r

%runocc: identify short bursts of run queue activity occurred

swpq-sz: number of swapped-out threads

%swpocc: swapped out occupancy, percentage of time there were swapped out threads


$ mpstat 3

CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
 32    0   0   18   460  126  257    0    3   10    0  2858    0   0   0  99
 33    0   0   14   131   34  180    1    7   49    0   740    5   1   0  94
 34    0   0    5   134   45  161    1    6    3    0   879    0   0   0 100
 35    0   0   60    78   33   93    0    7   54    0   457    0   0   0 100
 36    0   0    3   199   91  204    0    4    2    0  1125    0   0   0 100
 37    0   0  105    49   13   55    0    3   17    0   164    0   0   0 100
 38    0   0    2   124   56  126    0    2    1    0  1716    0   0   0  99
 39   30   0   21     5    0    7    0    2    4    0    13    0   0   0 100
 40    0   0    1     6    2    3    0    1    0    0    50    0   0   0 100
 41    0   0    0     3    0    5    0    3    5    0     0    0   0   0 100
 42    0   0   10    22   10    0    0    0    0    0     0    0   0   0 100
 43    0   0    1     7    1   10    0    5   12    0     1    0   0   0 100
 44    0   0    0     2    0    0    0    0    0    0     0    0   0   0 100
 45    0   0    1     6    0    8    0    4    7    0     2    0   0   0 100
 46    0   0   29    68   30    0    6    0  510    0 795891   72  28   0   0
 47    0   0    2     5    1    2    0    1    4    0     0    0   0   0 100
 48    0   0   26   346  153  331    1    6    8    0  4467    1   0   0  99
 49    0   0    0     3    0    7    0    4    8    0     0    0   0   0 100
 50    0   0   53   129   92   22    0    0    3    0     8    0   0   0 100
 51    0   0   65   131   42  142    0    7   61    0   568    1   0   0  99
 52    0   0   23   878  671  382    0    4   22    0  3765    1   2   0  97
 53    0   0   43    43   32   15    0    3    9    0     0    0   0   0 100
 54    0   0    0    22    6   27    0    0    0    0    39    0   0   0 100
 55    0   0   91   237  108  241    0    6   11    0   213    0   0   0 100
 56    0   0   38   303  143  231    0    5    2    0  3360    1   1   0  99
 57    0   0    8    59   14   81    0    2   19    0   352    3   0   0  97
 58    0   0    1     5    1    3    0    1    0    0     7    0   0   0 100
 59    0   0    9    61   18   57    0    3   13    0    62    0   1   0  99
 60    0   0   23   212   96  183    1    6    2    0  1159    0   0   0  99
 61    0   0   25   134   38  169    0    5   81    0   921    5   0   0  95
 62    0   0    0    19    3   27    0    0    2    0    27    0   0   0 100
 63    0   0    6    62   22   62    0    6    7    0   271    0   0   0 100


csw : the number of context switch (both voluntary and involuntary);
occurs when a thread perfomrs a blocking system call e.g. performing I/O when
the thread sleeps until the I/O event has completed.

icsw: the number of involuntary context switches; indicates if there were more
threads ready to run than available phyiscal processors.

migr: migrations of threads between processors; the number of times  the OS
scheduler moves ready-to-run threads to an idle prcessor. OS normally keeps
the threads on the last processor it ran unless the process is busy, then the
thread migrates. Such migration is bad for performance.

intr: the number of interrupts taken on CPU.

ithr: the number of interrupts that are converted to real threads, as result of
inbound network packets, blocking for a mutex, or a synchronization event.


smtx: the number of mutex contentions in the kernel, high shows high %sys time.

srw : the number of reader-writer lock contention events in the kernel.
Excessive reader/write lock contention results in nonscaling performance and
systems that are unable to use all the available CPU resources (symptom is idle time).


$ mpstat -ap 5 20
SET minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl sze
  0  183   1 1186   874  207 1104    7  158  156    0  4229    0   1   0  99  16
  1  112 104 2329  5085 2222 4805  215  710 1958  187 723869   17   4   0  80  16
SET minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl sze
  0  107   0 1505   995  323 1128    5   78  368    0  3755    2   1   0  97  16
  1    0   0  374  3011 1364 2399    7   68   35    0 38155    0   0   0  99  16
SET minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl sze
  0    4   0  244  1010  322 1170    4   74  304    0  3549    2   0   0  98  16
  1    0   0  472  2907 1313 2303    5   62   25    0 45064    0   0   0  99  16
SET minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl sze
  0    1   0  290   990  323 1156    4   79  322    0  3577    2   1   0  97  16
  1    0   0  265  2600 1119 2080    6   51   37    0 48610    1   0   0  99  16
SET minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl sze
  0   74   0  715  1074  323 1275    6   90  287    0  3816    3   1   0  96  16
  1    0   0  325  2695 1203 2077    6   67   31    0 59557    1   0   0  99  16


$prstat
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
 29302 sybase     11G   11G sleep   59    0 195:35:53 0.1% dataserver/259
 29347 sybase     11G   11G sleep   59    0 151:18:42 0.1% dataserver/259
 29348 sybase     11G   11G sleep   59    0 153:15:41 0.0% dataserver/259
 29340 sybase     11G   11G sleep   59    0 147:23:56 0.0% dataserver/259
 29341 sybase     11G   11G sleep   59    0 153:08:31 0.0% dataserver/259
 29342 sybase     11G   11G sleep   59    0 149:55:02 0.0% dataserver/259
 29346 sybase     11G   11G sleep   59    0 149:56:52 0.0% dataserver/259
 29334 sybase     11G   11G sleep   59    0 153:15:53 0.0% dataserver/259
 29349 sybase     11G   11G sleep   59    0 182:37:54 0.0% dataserver/259
 29343 sybase     11G   11G sleep   59    0 148:44:13 0.0% dataserver/259
 29345 sybase     11G   11G sleep   59    0 152:04:00 0.0% dataserver/259
 29333 sybase     11G   11G sleep   59    0 154:10:37 0.0% dataserver/259
 29332 sybase     11G   11G sleep   59    0 146:54:46 0.0% dataserver/259
 29335 sybase     11G   11G sleep   59    0 156:39:37 0.0% dataserver/259
 29344 sybase     11G   11G sleep   59    0 144:58:21 0.0% dataserver/259
  1540 sybase     14M   11M sleep   59    0   0:00:00 0.0% xpserver/1
 29320 sybase     10M 9152K sleep   59    0   0:00:15 0.0% backupserver/1
 28892 sybase   4296K 2136K sleep   59    0   0:00:06 0.0% SYBASE/1
 29304 sybase     10M 6872K sleep   59    0   0:00:00 0.0% backupserver/1
 28891 sybase   3440K 1880K sleep   59    0   0:00:06 0.0% lmgrd/1
Total: 20 processes, 3890 lwps, load averages: 1.02, 1.73, 2.12

Quick note for prstat:
CPU field is thread->pr_pctcpu
load average is the same as 'uptime'
When average load average exceeds the number of CPUs in the system (psrinfo),
the system is very likely over loaded.

prstat -mL  ; turn on microstates '-m' and show per-thread '-L'
   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
 14853 sjing     21  72 0.7 0.0 0.0 0.0 6.5 0.0  73  73 78K   0 prstat/1
 12047 betapm    34 0.8 0.0 0.0 0.0 0.0  65 0.1 773  21 12K   0 MathKernel/1
 29302 sybase    22 4.5 0.0 0.0 0.0 0.0  74 0.0 227  71 .3M   9 dataserver/1
 29349 sybase    20 3.7 0.0 0.0 0.0 0.0  76 0.0 234  75 .2M   7 dataserver/1
 29335 sybase    17 3.4 0.0 0.0 0.0 0.0  79 0.0 240  67 .2M   7 dataserver/1
 29333 sybase    17 3.4 0.0 0.0 0.0 0.0  79 0.0 242  67 .2M   8 dataserver/1
 29341 sybase    17 3.2 0.0 0.0 0.0 0.0  80 0.0 244  66 .2M   9 dataserver/1
 29348 sybase    17 3.2 0.0 0.0 0.0 0.0  80 0.0 243  67 .2M   8 dataserver/1
 29334 sybase    17 3.2 0.0 0.0 0.0 0.0  80 0.0 242  67 .2M   7 dataserver/1
 29345 sybase    17 3.2 0.0 0.0 0.0 0.0  80 0.0 242  66 .2M   7 dataserver/1
 29347 sybase    17 3.1 0.0 0.0 0.0 0.0  80 0.0 243  67 .2M   8 dataserver/1
 29342 sybase    17 3.2 0.0 0.0 0.0 0.0  80 0.0 245  66 .2M  11 dataserver/1
 29346 sybase    17 3.2 0.0 0.0 0.0 0.0  80 0.0 244  67 .2M   8 dataserver/1
 29343 sybase    17 3.0 0.0 0.0 0.0 0.0  80 0.0 244  67 .2M   8 dataserver/1
 29340 sybase    17 3.1 0.0 0.0 0.0 0.0  80 0.0 245  66 .2M   8 dataserver/1
Total: 141 processes, 4479 lwps, load averages: 0.87, 1.24, 1.83
   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
 29302 sybase    10 4.5 0.0 0.0 0.0 0.0  85 0.0 367   9 .3M 126 dataserver/1
 29347 sybase   9.9 4.0 0.0 0.0 0.0 0.0  86 0.0 373   8 .2M 127 dataserver/1
 12047 betapm   9.1 0.4 0.0 0.0 0.0 0.0  90 0.1 536   6  8K   0 MathKernel/1
 29348 sybase   1.0 0.4 0.0 0.0 0.0 0.0  99 0.0 303   1 29K  12 dataserver/1
 29340 sybase   0.8 0.4 0.0 0.0 0.0 0.0  99 0.0 302   2 26K   9 dataserver/1
 14853 sjing    0.3 0.8 0.0 0.0 0.0 0.0  99 0.0  17   1 727   0 prstat/1
 14831 dbop     0.0 0.9 0.0 0.0 0.0 0.0  99 0.1 826   2  1K   0 find/1
 29341 sybase   0.6 0.3 0.0 0.0 0.0 0.0  99 0.0 297   1 19K   7 dataserver/1
 29342 sybase   0.6 0.2 0.0 0.0 0.0 0.0  99 0.0 298   0 17K   4 dataserver/1
 29349 sybase   0.5 0.2 0.0 0.0 0.0 0.0  99 0.0 298   1 16K   5 dataserver/1
 12063 betapm   0.2 0.4 0.0 0.0 0.0 0.0  99 0.0 505   1  6K   0 MathAddon/1
 29346 sybase   0.4 0.2 0.0 0.0 0.0 0.0  99 0.0 297   1 14K   4 dataserver/1
 29333 sybase   0.4 0.2 0.0 0.0 0.0 0.0  99 0.0 297   0 11K   3 dataserver/1
 29334 sybase   0.4 0.2 0.0 0.0 0.0 0.0  99 0.0 299   0 12K   3 dataserver/1
 29335 sybase   0.4 0.2 0.0 0.0 0.0 0.0  99 0.0 297   1  9K   2 dataserver/1
Total: 141 processes, 4479 lwps, load averages: 0.86, 1.23, 1.82

LAT : (latency) is the amount of time spent waiting for CPU.

Gauge UNIX Skill Level

2010-04-02T13:38:00.001-07:00

Welcome to Shan's Gauge UNIX Skill Level Test.

Here you will find some challenging Unix questions I put together from my daily tasks and readings. Some are the questions I answered at it-toolbox forum. You may rate yourself from 1 to 10 first, then start to look at these interesting questions and check your rating again. All answers are discussed in my blog.

The skill levels
3-5 : easy, user level, e.g. Java developer's UNIX knowledge or junior SA
6-7 : medium level, Mid to Senior Level UNIX SAs
8-9 : hard, creative, smart Senior SAs with strong computer science background or a guy who can teach UNIX courses at school of continuing eduction
10 : reserved for industry experts who can write books for the subject

Have fun!

__BEGIN__

------------------------------------------------------------------------------

0.(hard)Which directory's "." and ".." have the same i-node number (st_ino) ?

1.(medium) Why the UNIX command 'cd' must be built into a shell?

1a. What's wrong with the following shell commands? List at least two ways to fix it?
export pipe=""; ls $pipe wc -l

1c. (medium) This one is for Linux Administrators - How do you adjust the Linux kernel value for tcp_fin_timeout, tcp_keepalive_time? Can you do it with a few lines of C code?

2.(creative) Use shell utilities (Perl/sed/awk not allowed) and curl/wget to count the frequency of each word of this webpage.

For example,
curl http://webpage your_commands

the output should be:
577 n
499 a
476 data
350 span
336 class
324 br
295 b
266 div
210 post
203 li
199 http
194 href
192 com
165 if
136 the
118 style
115 color
113 title
108 quot
106 blogspot

2.5 (medium) Use shell utility to reverse text file upside down, e.g.
line1
line2
line3
...
line99

->
line99
line98
...
line1

3.(medium)Use C, Python or Perl (pick 2) to implement a pipe. e.g. "my_pipe cat file1 wc -l" = "cat file1 | wc -l"

4.(medium) Write a regular expression to match a valid IP address. ( 999.999.999.999 isn't a valid one)

5.(hard)Why only root can set sticky bit for files?

6.(web- creative) Write a curl command to login one of major sites with your id/password without using a browser. sites can be ebay, yahoo, youtube, etc.

7.(creative)How do you copy a terminal to another one? Say tty0 to tty2 and see tty0's real time typing/output from tty2 ? (very useful , you want to know!)

8.(Enterprise, medium) Write down the commands you use to extend or reduce the size of a mounted, live production vxfs filesystem (sitting on top of a mirrored volume) without using vxresize or veritas command line interface. Of course, you can't use the gui tool. (Hint, there's a difference in action sequence for extending and reducing the size)

9.(Web, medium) Describe apache AJP connector, how to you configure it? Where the connector is defined inside J2EE container, for example tomcat.

10.(Enterprise, medium) List 3 different commands you use on AIX to find out the amount of real memory? Describe the general UNIX/AIX performance tuning practice on a dedicated backend database server running AIX 5.3+, Sybase 12.5 with EMC SRDF. (Hint, only from UNIX SA's prospect, DBAs will do their parts)

11.(medium)Write a sed command to wrap each word in a text file with parenthesis. e.g. abc def -> (abc) (def)

12.(Web, medium) Describe Apache DSO vs static modules, what's the difference and advantage of DSO? How to compile/configure apache and DSO? How do you find out if your running apache instance is DSO enabled?

13.(hard) Write sed command to implement gnu grep command's -A1 and -B1 function, e.g. 'grep -A1 -B1'.

14.(medium)Write a grep command to match "the" but not "them", "theory". etc.

15.(Web, medium) What are tomcat workers? Describe what's inside workers.properties file

16.(medium) Describe the procedure and write a sample code using Perl to access mysql database, do query and update records. Do it with Python, Java?

17.(web- medium) Write a curl command to download a range of files from a site.

18.(creative)What commands you will use to spy someone's typing and record all his/her terminal session in a file?

19.(easy)At UNIX shell prompt, how do you find out which tty you are using?

20.(easy)How do you set your terminal not to display what you type?

21.(easy)What are ksh/bash "$$" and "$#" variables?

22.(easy)What's the difference between shell variables "$*" and "$@"?

23.(medium)Will this cd command "pwd; (cd/usr/local/bin); pwd" changes the current path? Why?

24.(hard) Write a sed command to add commas to numeric strings, e.g. changing "1234567" to "1,234,567"

25.(medium)Write a regular expression and use one utility (sed, perl, etc.) to look into a text file and remove all duplicated letters in any word, e.g. change abbcdee -> abcde ... zzfwxxy ->zfwxy.

26.(medium)Write a sed command to count the number of lines in several files that don't begin with a "#:"

27.(medium) Write a grep command to display 5 lines before and after the match.

28.(medium)What does UNIX command "apropos" do?

29.(medium)At UNIX prompt, how to find out all the commands related to sending signal to processes?

30.(hard)What does pid "-1" represent in "kill" command?

31.(medium)How to find out the list of available signals for kill command?

32.(medium)What does sticky bit is set for file? for directory?

33.(hard)Why only root can set sticky bit for files?

34.(hard)Can a user write into a UNIX directory?

35.(hard)From Unix internal prospect, describe a directory? How many hard links a leaf directory has?

36.(medium)Who can create a new link that points to a directory?

37.(hard)List and describe objects that a UNIX filesystem normally has.

38.(hard)In "ls -l file1" command, what information is NOT included in the inode?

39.(hard)Which directory's "." and ".." have the same i-node number (st_ino) ?

40.(medium)How do you stop a running process?

41.(medium)Which 'ls' option shows files' st_ctime, st_atime and mtime?

42.(hard)Give examples of Unix functions/commands that change each of a file's 3 time values.

43.(hard)Can you see an i-node's last access time?

44.(medium)Explain what's RPC (remote procedure call)? How it works. List one major service relies on rpc.

45.(medium)Explain the procedures to setup NIS (pick 2 from AIX,Solaris, Linux) and autofs so that users can login from any host inside NIS domain and automounte his/her home directory.

46.(hard)Describe the difference between multithreaded and multiprocess in UNIX.

47.(hard)Describe UNIX internal's "the lowest-available-file-descriptor" principle.

48.(hard)Describe 3 ways to attach STDIN to a file. Use C, Python or Perl to implement it.

49.(Web, medium) Describe the AJP connector and mod_jk module, function, configuration.

50.(Web, medium) Describe load balancing tomcat with mod_proxy, configuration and testing.

51.(Enterprise, creative) Create a solution that utilizes enterprise products Veritas VCS and Solaris 10 Zone to create virtual Solaris servers (v1, v2, v3...) that can have HA features on two physical servers, for example, two SUN M5Ks. Assume your environment has EMC SRDF, extend that design to remote replication and Disaster Recovery (DR) implementation on a remote site.

52.(Enterprise, Web, hard) Describe a 3-tier web environment, all the products you use, the detail configuration and administration operation of each product (for example, what's .war file, where do you define url mapping in tomcat? what's apache DSO?), pick 2 environments out of the 3 - Linux, Solaris and AIX.

53.(Enterprise, Web, hard) Describe the local High Availability implementation for each tier in the above 3-tier environment (the technology/products, the implementation)

54.(Enterprise, Web, hard) Describe your DR/ remote replication design (the technology and implementation) for the above 3-tier environment at both infrastructure and application levels.(Hint, there's no budget issue and you can pick any enterprise or open source products you need)

55.(Enterprise,medium) A company has over 2000 IT staffs around the world (app developers in India, SAs in Singapore, DBAs in Japan etc), as UNIX engineer, implement a solution to allow each user sign in her/his UNIX accounts and being able to have her/his home directories mounted automatically from any (Linux or Solaris) workstations around the world. Your team is also responsible backing up/restore all the users home directories.

56.(Enterprise, hard) For the above implementation, add security layers for authentication and account management.

57. (medium) List 7 types of items that a UNIX filesystem has. What are Perl's file test operators for each item?

58. (medium) What's the difference between Perl's stat and lstat function?

--------------------------------------------------------------------------------------------------------
ANSWERS: ( If answers are not here, search my blog to find out.)

Which directory's "." and ".." have the same i-node number (st_ino)?
ls -ila / will show the answer. Why? Answer.

Why the UNIX "cd" command must be built into the shells?
Because the shell forks a child process to run chdir function to change directory, once the child process terminates, the child's working directory can not affect the shell's. Therefore, the shell's working directory can never be changed by its child process. For this reason, the chdir function has to be called directly from the shell. This is why cd is built into the shells.

export pipe=""; ls $pipe wc -l
The shell take care of pipes and I/O redirection BEFORE variable substitution, therefore, it nver recognizes the pipe symbol inside pipe. The result is that the three arguments , wc and -l are passed to "ls" command as arguments.

Solutions:
a. eval export pipe=""; ls $pipe wc -l
(eval will have shell scan the command sequence twice before executing it. If the variables contain any characters that must be seen by the shell directly on the command line, then eval can be useful.)

b. Just type "ls wc -l".

sysctl -p the following configuration file:

# Decrease the time default value for tcp_fin_timeout connection.
net.ipv4.tcp_fin_timeout = 30

# Decrease the time default value for tcp_keepalive_time connection
net.ipv4.tcp_keepalive_time = 1800

# Turn off tcp_window_scaling
net.ipv4.tcp_window_scaling = 0

# Turn off the tcp_sack
net.ipv4.tcp_sack = 0

# Turn off tcp_timestamps
net.ipv4.tcp_timestamps = 0

See Hack #71 in this book.

Use shell utilities (Perl/Sed/awk not allowed) to count the frequency of each English word of this webpage.
$ curl 'http://shanit.blogspot.com/2009/10/unix-handy-tools-for-system.html' tr -sc a-zA-Z '\012' sort uniq -c sort -nr

Output:

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Dload  Upload   Total   Spent    Left  Speed
100 83662    0 83662    0     0   305k      0 --:--:-- --:--:-- --:--:--  533k

497 n
425 span
409 data
306 a
241 b
238 class
230 br
217 div
185 style
181 post
142 font
140 color

To reverse file vertically, e.g. the last line becomes the first line.
nl input_file sort -nr sed 's/^.\{7\}//'

Will this cd command "pwd; (cd/usr/loca/bin); pwd" changes the current path? Why?
No. Commands inside () are executed in a subshell.

Write a sed command to wrap each word in a text file with parenthesis. e.g. abc def -> (abc) (def)
sed 's/[A-Za-z][A-Za-z]*/(&)/g' <inputfile> output

Use sed implement 'grep -A1 -B1' (note: grep -A -B options are NOT available on Solaris, you may sed or perl, awk instead. Sure, how do you implement it with Perl, awk? :-))
sed -n -e '/regexp/{=;x;1!p;g;$!N;p;D;}' -e h

Write a sed command to add commas to numeric strings, e.g. changing "1234567" to "1,234,567"
sed -e :a -e 's/$.*[0-9]$$[0-9]\{3\}$/\1,\2/;ta'

Write a regular expression and use one utility (sed, perl, etc.) to look into a text file and remove all duplicated letters in any word, e.g. change abbcdee -> abcde ... zzfwxxy ->zfwxy.

2 ways to do it:
a. [shan@ipc4 ~]$ sed -n 's@$[a-zA-Z]$\1*@\1@pg' < input > output
b. echo "aabccdee" tr -s [[:alpha:]]

Use curl to sign in a major site such as yahoo! youtube or ebay. Answer.

List 3 different commands you use on AIX to find out the amount of real memory? Describe the general UNIX/AIX performance tuning practice.
Answers.

Write a sed command to count the number of lines in several files that don't begin with a "#:"
sed 's/^#.*//' file1 file2 file3 grep -v '^$' wc -l

Write a grep command to match "the" not "thexyz..." such as "them", "theory".

What are ksh/bash "$$" and "$#" variables ?
$ is the pid of current shell
$# is the number of the positional parameters to the command

What's the difference between shell variables "$*" and "$@"?
$* expands to all positional parameters passed to the command
$@ expands to all positional parameters passed to the command, but individually quoted when "$@" is used.

What does UNIX command "apropos" do?
man -k keyword

Usage: find out all the commands related to sending signal to processes:

[root@ipc4 ~]# apropos kill grep signal
kill (2) - send signal to a process
killpg (2) - send signal to a process group
killpg (3) - send signal to all members of a process group
pkill [pgrep] (1) - look up or signal processes based on name and other attributes
pthread_kill [pthread_sigmask] (3thr) - handling of signals in threads
skill (1) - send a signal or report process status
snice [skill] (1) - send a signal or report process status
tkill (2) - send a signal to a single process

What's the default signal sent by "kill" command ?

What's this command does? "kill -9 -1"

How to find out the list of available signals for kill command?

[root@ipc4 ~]# kill -l
1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL
5) SIGTRAP 6) SIGABRT 7) SIGBUS 8) SIGFPE
9) SIGKILL 10) SIGUSR1 11) SIGSEGV 12) SIGUSR2
13) SIGPIPE 14) SIGALRM 15) SIGTERM 17) SIGCHLD
18) SIGCONT 19) SIGSTOP 20) SIGTSTP 21) SIGTTIN
22) SIGTTOU 23) SIGURG 24) SIGXCPU 25) SIGXFSZ
26) SIGVTALRM 27) SIGPROF 28) SIGWINCH 29) SIGIO
30) SIGPWR 31) SIGSYS 34) SIGRTMIN 35) SIGRTMIN+1
36) SIGRTMIN+2 37) SIGRTMIN+3 38) SIGRTMIN+4 39) SIGRTMIN+5
40) SIGRTMIN+6 41) SIGRTMIN+7 42) SIGRTMIN+8 43) SIGRTMIN+9
44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13
48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13
52) SIGRTMAX-12 53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9
56) SIGRTMAX-8 57) SIGRTMAX-7 58) SIGRTMAX-6 59) SIGRTMAX-5
60) SIGRTMAX-4 61) SIGRTMAX-3 62) SIGRTMAX-2 63) SIGRTMAX-1
64) SIGRTMAX

or, we can see the linux programmer's manual

[root@ipc4 ~]# apropos signal grep list
signal (7) - list of available signals

[root@ipc4 ~]# man 7 signal col -b grep SIG head -20
SIGNAL(7) Linux Programmerâs Manual SIGNAL(7)
SIGHUP 1 Term Hangup detected on controlling terminal
SIGINT 2 Term Interrupt from keyboard
SIGQUIT 3 Core Quit from keyboard
SIGILL 4 Core Illegal Instruction
SIGABRT 6 Core Abort signal from abort(3)
SIGFPE 8 Core Floating point exception
SIGKILL 9 Term Kill signal
SIGSEGV 11 Core Invalid memory reference
SIGPIPE 13 Term Broken pipe: write to pipe with no readers
SIGALRM 14 Term Timer signal from alarm(2)
SIGTERM 15 Term Termination signal
SIGUSR1 30,10,16 Term User-defined signal 1
SIGUSR2 31,12,17 Term User-defined signal 2
SIGCHLD 20,17,18 Ign Child stopped or terminated
SIGCONT 19,18,25 Continue if stopped
SIGSTOP 17,19,23 Stop Stop process
SIGTSTP 18,20,24 Stop Stop typed at tty
SIGTTIN 21,21,26 Stop tty input for background process
SIGTTOU 22,22,27 Stop tty output for background process

What does sticky bit is set for file? for directory?

A sticky bit is set for an executable file, then a copy of the binary (after the first time it's executed) is saved in the swap area when the process terminated. For editors or other common programs, sticky bit is set.
Only the superuser can set the sticky bit (S_ISVTX) of a regular file. This is to prevent malicious users from setting the sticky bit and trying to fill up the swap area.

When a sticky bit is set for a directory (-t on the directories group access mode), a file in the directory can be removed or renamed only if the user has write permission for the directory and either owns the file, owns the directory or is the superuser. /tmp has sticky bit set for directory.

Can a user write into a UNIX directory? NO. Only the kernel can write to a directory.

A directory is just a file containing directory entries(filenames and associate i-node numbers). Adding, deleting, or modifying these directory entries can affect all three times associated with that directory. With permissions, we can create new files in the direcotry and remove files from it, but we can't write to the directory itself.

What is a directory? How many hard links a leaf directory has? Who can create a new link that points to a directory?

A filesystem normally has boot block, super block, i-node list, directory and data blocks.

The i-node contains all the information about the file: the file type, the file's access permission bits, the size of the file, pointers to the data blocks for the file, and so on. Most of the information in the stat structure is obtained from the i-node. Only two items are stored in the directory entry: the filename and the i-node number.

When renaming a file without changing filesystems, the actual content of the file need not be moved - all that needs to be done is to have a new directory entry point to the existing i-node and have the old directory entry removed.

Since the i-node number in the directory entry points to an i-node in the same filesystem, we cannot have a directory entry pint to an-inode in a different filesystem. This is why hardlink can't cross filesystems.

With a symbolic link, the actual contents of the file (the data blocks) contains the name of the file that the symbolic link points to.

Each i-node has a link count that contains the number of directory entries that point to the i-node. When the link count goes to 0 can the file be deleted. This is why "unlinking a file" does not always mean "deleting the blocks associated with the file." They is why the function that removes a directory is called unlink not delete.

Which directory's "." and ".." have the same i-node number (st_ino) ? /

How do you stop (not kill) a running process?
control-Z
bg - brings it to background
fg - brings it to foreground
% list jobs

Which 'ls' option shows files' st_ctime, st_atime and mtime? Give examples of Unix functions/commands that change each of a file's 3 time values.

Field	Description	Example	`ls`(1) option
`st_atime`	last-access time of file data	`read`	`-u`
`st_mtime`	last-modification time of file data	`write`	default
`st_ctime`	last-change time of i-node status	`chmod, chown`	`-c`

ls -u : atime
ls -t : mtime(default)
ls -c : ctime

st_mtime is the modification time - when the contents of the file were last changed.
st_ctime is when the i-node of the file was last modified such as chmod, chown, link, etc.
Because all the information in the i-node is stored separately from the actual contents of the file,we need the changed-status time.

Can you see an i-node's last access time?
No. The system doesn't maintain the last-access time for an i-node. This is why functions like 'access' and 'stat' don't change any of the three times.

UNIX filesystem's 7 types of items and the Perl's file test operators for each item:


Item                    Perl's File Test
====                    ================
file                    -f
directory               -d
symbolic link           -l
socket                  -S
named pipe(fifo)        -p
block-special file      -b
character-special file  -c

What's the difference between Perl's stat and lstat function?
stat returns fs information for files and directories while lstat returns information for symbolic links.

Figure - i-node structure

All tomcat/apache questions are discussed at my blog's Web/Apache Section.

All AIX/Solaris related questions are discussed at my blog's Solaris and AIX Sections.

All TCP/IP related questions are discussed at my blog's TCP/IP Sections.

My thoughts about "UNIX Administrator" vs "Enterprise Administrator in UNIX environments" -
In my personal opinion, to gauge a UNIX SA's skill level, the general UNIX and computer science background are more important than the knowledge of certain enterprise products. For over 10 years, I have been working with all kinds of enterprise technologies such as Veritas VCS, VXFS, EMC, AIX, Solaris, etc. I have 12 - 15 certificates for enterprise products; but I just don't feel it is very hard to learn and administrate these enterprise products at senior levels (compare to mastering vim, sed or regular expressions); moreover, you don't need a computer science degree to create a Veritas filesystem or setup a cluster.

It is very common that a good portion of "UNIX" administrators who know how to jumpstart Solaris, attach EMC luns or setup NFS in AIX or Solaris still cut and paste/use bash shell's arrow keys to recall commands in UNIX shell; still manually update text file without knowing how to use sed/regular expression. These folks are "Enterprise System Administrators" not UNIX administrators. They happen to work in UNIX environment instead of Windows. Please keep mind that people will gauge UNIX skill levels by just watching a few things you do in UNIX such as how you use shell utilities, use vi/vim, etc. If you are weak in these areas, please fix them asap because, at least, these tools can increase your productivity.

On the other hand, people don't have experience on those enterprise products is not because they can't understand them; it is because they don't have access to the enterprise environment.

Finally, I rate what I know about UNIX as a drop of water in the UNIX knowledge sea. I'm very happy to work inside UNIX/Linux environment because I can always learn something new everyday.

Web Admin - Apache Tomcat Weblogic Java

2010-04-02T13:34:00.000-07:00

Scripting (Perl, Shell, Python)

2010-04-02T13:23:00.000-07:00

Linux Administration (RHEL Ubuntu)

2010-04-02T12:53:00.000-07:00

Linux RPM and Open-Source Development Tools

2010-04-02T12:41:00.000-07:00

rpm -Querying/Verifying, -

-q selection-options -l

rpm -q --whatprovides /usr/bin/gcc ; rpm -ql gcc

-U (--upgrade) -F (--freshen : only upgrade the installed packages)

rpm --showrc

shows the values rpm will use for all of the options are currently set in rpmrc and macros configuration file(s).

rpm -ivh {rpm-file} Install the package
Examples:
rpm -ivh mozilla-mail-1.7.5-17.i586.rpm
rpm -ivh --test mozilla-mail-1.7.5-17.i586.rpm

rpm -Uvh {rpm-file} Upgrade package
Examples:
rpm -Uvh mozilla-mail-1.7.6-12.i586.rpm
rpm -Uvh --test mozilla-mail-1.7.6-12.i586.rpm

rpm -ev {package} Erase/remove/ an installed package
Examples:
rpm -ev mozilla-mail
rpm -ev --nodeps {package} Erase/remove/ an installed package without checking for dependencies
rpm -ev --nodeps mozilla-mail

rpm -qa Display list all installed packages
rpm -qa
rpm -qa | less

rpm -qi {package} Display installed information along with package version and short description
rpm -qi mozilla-mail

rpm -qf {/path/to/file} Find out what package a file belongs to i.e. find what package owns the file
rpm -qf /etc/passwd
rpm -qf /bin/bash

rpm -qc {pacakge-name} Display list of configuration file(s) for a package
rpm -qc httpd

rpm -qcf {/path/to/file} Display list of configuration files for a command
rpm -qcf /usr/X11R6/bin/xeyes

rpm -qa --last Display list of all recently installed RPMs
rpm -qa --last
rpm -qa --last | less

rpm -qpR {.rpm-file}
rpm -qR {package} Find out what dependencies a rpm file has
rpm -qpR mediawiki-1.4rc1-4.i586.rpm
rpm -qR bash

Compile Open Source Code:

Static Libraries

also known as archives, have their names end with .a . /usr/lib/libss.a

.a is created and maintained by ar utility.

To create static library:

ar crv lib1.a prog1.o prog2.o

[root@localhost lib]# ar -t libss.a
ss_err.o
std_rqs.o
invocation.o
help.o
execute_cmd.o
listen.o
parse.o
error.o
prompt.o
request_tbl.o
list_rqs.o
pager.o
requests.o
data.o
get_readline.o

[root@localhost lib]# cd
[root@localhost ~]# vi prog1.c
[root@localhost ~]# vi prog2.c
[root@localhost ~]# vi prog1.c
[root@localhost ~]# gcc -c prog1.c prog2.c
[root@localhost ~]# ls *.o
prog1.o prog2.o
[root@localhost ~]# vi lib.h
[root@localhost ~]# vi prog2.c
[root@localhost ~]# gcc -c prog1.c prog2.c
[root@localhost ~]# vi program.c
[root@localhost ~]# gcc -c program.c
[root@localhost ~]# gcc -o program program.o prog1.o
[root@localhost ~]# ./program
Shan: you won 134513872

[root@localhost ~]# ar crv libshan.a prog1.o prog2.o
a - prog1.o
a - prog2.o

[root@localhost ~]# ar -t libshan.a
prog1.o
prog2.o

extra member from archive
[root@localhost tmp]# ar x libshan.a
[root@localhost tmp]# ls
libshan.a prog1.o prog2.o

Use nm to see which functions are included in an object file, library or executable program(if compiled with symbols).
[root@localhost tmp]# nm program.o
U exit
00000000 T main
U prog1
[root@localhost tmp]# cat ../program.c
#include "lib.h"
int main ()
{
prog1("1000000");
exit (0);
}

[root@localhost ~]# ldd program
libc.so.6 => /lib/tls/libc.so.6 (0x0018a000)
/lib/ld-linux.so.2 (0x00171000)

Probem with Static libraries: when many running processes need to use the same function (provided by static lib),
the system ends up with many copies of the same code/functions in memory.

Shared Library solves this problem. When a program uses a shred lib, it doesn't contain the function code but
it references to shared code that will be available at run time. When the resulting program is loaded into
memory to be executed, the function references are resolved and calls are made to the shared library, which will
be loaded into the memory if needed.

In this way, the system can just have a single copy of a shared library to be used by many applications at once.
An additional benefit is that the shared library can be updated independently of the programs that rely on it
because symbolic links from the file /lib/libc.so.6 to the actual library revision libc-2.3.4.so is used.

rwxrwxrwx 1 root root 13 Oct 30 12:04 /lib/tls/libc.so.6 -> libc-2.3.4.so

RMP cheat page.

Linux Security 1

2010-04-02T12:40:00.000-07:00

Network Security

# Controls IP packet forwarding
net.ipv4.ip_forward = 0

# Disable response to broadcasts.
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1

# enable route verification on all interfaces
net.ipv4.conf.all.rp_filter = 1

NFS Security

Only run NFS as needed, apply latest patches.
When creating your /etc/exports file, be certain to use limited access flags when possible such as readonly or
nosuid. By using fully qualified hostnames, you are guaranteed that only the host you want to be able to
access the filesystem can access it.

If you are exporting file-systems using NFS, be sure to configure /etc/exports with the most restrictive access
possible. This means not using wild cards, not allowing root write access, and exporting read-only wherever
possible.

Device Security

Device files /dev/null, /dev/tty & /dev/console should be world writeable but NEVER executable. Most other
device files should be unreadable and unwriteable by regular users.

Script Security

Never write setuid/setgid shell scripts (can break out). Instead, write a compiled program in a language
like "C". Scripts should ALWAYS have full pathnames.

Program Security

Always get your programs from a known source. Verify that it hasn't been hampered with via checksum. If
you are compiling your own program, make sure you know that the compiler hasn't been tampered with as well.

General Security Measures

Create minimal writable filesystems (esp. system files/directories!). Generally, users should only be able
to write in their own directories, and /tmp. In addition, there will be directories for a specific group to
write in. This way you control how each user can access specific areas of the system.

root# find / -type f $ -perm -04000 -o -perm -02000 $

root# find / -perm -2 ! -type l -ls

root# find / $ -nouser -o -nogroup $ -print

root# find /home -name .rhosts -print

Set file system limits instead of allowing unlimited as is the default. You can control the per-user limits using the resource-limits PAM module and /etc/pam.d/limits.conf. For example, limits for group users might look like this:

@users hard core 0
@users hard nproc 50
@users hard rss 5000

You can also use the /etc/login.defs configuration file to set the same limits.

Linux - Performance Tuning 1

2010-04-02T12:34:00.000-07:00

Linux - Performance Tuning 1

Useful top and pstree commands to show process/threads

#show process tree with pid -p owned by user -u
$pstree -aG -p -u username

#same as above but use ASCII chars to draw the tree. -A
$pstree -aA -p -u username

#top in batch mode -b , show 3 iternations -n and delay 2.5sec -d for user -u
$top -b -n 3 -d 2.5 -u username

#same as above but display threads instead of just process -H
$top -H -b -n 3 -d 2.5 -u sjing

$top -p pid

#Linux : check if Processor (hardware) is 64 bit lm (long mode)
$grep lm /proc/cpuinfo

(inside top, type "A" will enter multi-windows mode)

vmstat -d -p partition

vmstat -m

Probably sysctl is the best tool for tuning linux systems. It can configure a lot of system parameters while the linux kernel
is running. It handles this, by reading and writing to kernel variables through procfs files.

sysctl - configure kernel parameters at runtime. The parameters available are those listed under /proc/sys/.

sysctl -a

sysctl -w variable=value. For example, sysctl -w net.ipv6.conf.all.forwarding=1

sysctl -w kernel.shmmax=63554432

sysctl -p

root@ubuntu:~# sysctl -a
kernel.sched_min_granularity_ns = 8000000
kernel.sched_latency_ns = 40000000
kernel.sched_wakeup_granularity_ns = 10000000
kernel.sched_shares_ratelimit = 500000
kernel.sched_shares_thresh = 4
kernel.sched_child_runs_first = 1
kernel.sched_features = 113916
kernel.sched_migration_cost = 500000
kernel.sched_nr_migrate = 32
kernel.timer_migration = 1
kernel.sched_rt_period_us = 1000000
kernel.sched_rt_runtime_us = 950000
kernel.sched_compat_yield = 0
kernel.panic = 0
...

sysctl displays kernel setting in the following categories:

kernel
vm
fs
debug
dev
net
crypto

A very good guide to "top" utility is at here.

[root@localhost ~]# top -b -n 1
top - 12:50:59 up 1:11, 5 users, load average: 0.22, 0.24, 0.17
Tasks: 129 total, 2 running, 126 sleeping, 0 stopped, 1 zombie
Cpu(s): 1.9% us, 2.8% sy, 0.1% ni, 89.0% id, 3.7% wa, 0.1% hi, 2.4% si
Mem: 514516k total, 321012k used, 193504k free, 36688k buffers
Swap: 1048568k total, 0k used, 1048568k free, 173132k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7438 root 16 0 2552 968 740 S 3.9 0.2 0:04.36 top
8662 root 15 0 3784 856 648 R 3.9 0.2 0:00.03 top
1 root 16 0 2372 548 468 S 0.0 0.1 0:04.49 init
2 root RT 0 0 0 0 S 0.0 0.0 0:00.71 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
4 root RT 0 0 0 0 S 0.0 0.0 0:00.62 migration/1
5 root 34 19 0 0 0 S 0.0 0.0 0:00.02 ksoftirqd/1
6 root 5 -10 0 0 0 S 0.0 0.0 0:00.78 events/0
7 root 5 -10 0 0 0 S 0.0 0.0 0:00.52 events/1
8 root 5 -10 0 0 0 S 0.0 0.0 0:00.11 khelper
9 root 15 -10 0 0 0 S 0.0 0.0 0:00.00 kacpid
90 root 5 -10 0 0 0 S 0.0 0.0 0:00.55 kblockd/0
91 root 5 -10 0 0 0 S 0.0 0.0 0:00.51 kblockd/1
92 root 15 0 0 0 0 S 0.0 0.0 0:00.00 khubd
101 root 20 0 0 0 0 S 0.0 0.0 0:00.00 pdflush
102 root 15 0 0 0 0 S 0.0 0.0 0:01.38 pdflush
104 root 7 -10 0 0 0 S 0.0 0.0 0:00.00 aio/0
105 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 aio/1
103 root 25 0 0 0 0 S 0.0 0.0 0:00.00 kswapd0
179 root 25 0 0 0 0 S 0.0 0.0 0:00.00 kseriod
250 root 19 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_0
267 root 6 -10 0 0 0 S 0.0 0.0 0:00.00 kmirrord/0

Field Description
PID : Process ID
USER : Effective User ID
PR : Dynamic priority
NI : Nice value, also known as base priority
VIRT : Virtual Size of the task. This includes the size of process's executable binary, the data area and all the loaded shared libraries.
RES : The size of RAM currently consumed by the task. Swapped out portion of the task is not included.
SHR : Some memory areas could be shared between two or more task, this field reflects that shared areas. The example of shared area are shared library and SysV shared memory.
S : Task status
%CPU : The percentage of CPU time dedicated to run the task since the last top's screen update.
%MEM : The percentage of RAM currently consumed by the task.
TIME+ : The total CPU time the task has been used since it started. "+" sign means it is displayed with hundreth of a second granularity. By default, TIME/TIME+ doesn't account the CPU time used by the task's dead children.
COMMAND : Showing program names.

$ top -p 4360,4358

top + "A" to see multiple windows mode

1:Def - 13:16:02 up 1:37, 4 users, load average: 0.06, 0.04, 0.03
Tasks: 122 total, 1 running, 120 sleeping, 0 stopped, 1 zombie
Cpu(s): 0.7% us, 1.4% sy, 0.0% ni, 96.5% id, 0.0% wa, 0.0% hi, 1.4% si
Mem: 514516k total, 320228k used, 194288k free, 37336k buffers
Swap: 1048568k total, 0k used, 1048568k free, 174564k cached

1 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9234 root 16 0 3020 964 740 R 2.6 0.2 0:00.04 top
3408 root 16 0 9768 7188 1628 S 1.3 1.4 0:46.53 hald
8601 shan 15 0 7344 2228 1812 S 1.3 0.4 0:00.51 sshd
1 root 16 0 2372 548 468 S 0.0 0.1 0:04.50 init
2 root RT 0 0 0 0 S 0.0 0.0 0:00.76 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
4 root RT 0 0 0 0 S 0.0 0.0 0:00.66 migration/1
5 root 34 19 0 0 0 S 0.0 0.0 0:00.02 ksoftirqd/1
6 root 5 -10 0 0 0 S 0.0 0.0 0:00.80 events/0
7 root 5 -10 0 0 0 S 0.0 0.0 0:00.53 events/1
8 root 5 -10 0 0 0 S 0.0 0.0 0:00.11 khelper
9 root 15 -10 0 0 0 S 0.0 0.0 0:00.00 kacpid
2 PID PPID TIME+ %CPU %MEM PR NI S VIRT SWAP RES UID COMMAND
9234 8630 0:00.04 2.6 0.2 16 0 R 3020 2056 964 0 top
8630 8629 0:00.17 0.0 0.3 16 0 S 5820 4396 1424 0 bash
8629 8628 0:00.01 0.0 0.2 19 0 S 5036 3900 1136 0 su
8628 8602 0:00.02 0.0 0.0 17 0 S 2680 2428 252 0 sesh
8602 8601 0:00.05 0.0 0.3 16 0 S 5836 4460 1376 500 bash
8601 8599 0:00.51 1.3 0.4 15 0 S 7344 5116 2228 500 sshd
8599 3241 0:00.07 0.0 0.4 17 0 S 7164 5028 2136 0 sshd
8537 1 0:00.00 0.0 0.2 16 0 S 3328 2336 992 0 dhclient
7232 1 0:00.01 0.0 0.2 16 0 S 3004 2004 1000 0 dhclient
6932 6931 0:00.21 0.0 0.3 16 0 S 4828 3400 1428 0 bash
6931 6930 0:00.01 0.0 0.2 19 0 S 5228 4092 1136 0 su
6930 6893 0:00.02 0.0 0.0 16 0 S 2392 2136 256 0 sesh
3 PID %MEM VIRT SWAP RES CODE DATA SHR nFLT nDRT S PR NI %CPU COMMAND
3937 4.2 155m 133m 21m 1520 16m 5232 0 0 S 16 0 0.0 X
4838 3.5 40640 21m 17m 588 19m 9920 23 0 S 15 0 0.0 nautilus
6891 2.4 36388 23m 12m 252 15m 8228 0 0 S 15 0 0.0 gnome-terminal
4834 2.2 22848 11m 11m 436 4332 8220 8 0 S 15 0 0.0 gnome-panel
5294 2.0 21696 11m 9.8m 28 2872 7544 9 0 S 15 0 0.0 mixer_applet2
5292 1.9 21208 11m 9576 64 3036 7472 3 0 S 16 0 0.0 wnck-applet
4555 1.8 20696 10m 9472 120 2776 6980 11 0 S 15 0 0.0 gnome-session
5296 1.5 20496 12m 7932 92 2132 6680 6 0 S 16 0 0.0 clock-applet
4620 1.5 12280 4700 7580 44 7272 1668 1 0 S 16 0 0.0 gconfd-2
4868 1.4 40444 32m 7336 104 22m 6268 4 0 S 16 0 0.0 eggcups
3408 1.4 9768 2580 7188 196 6712 1628 2 0 S 16 0 1.3 hald
4696 1.4 14616 7436 7180 428 2304 6024 6 0 S 15 0 0.0 metacity
4627 1.4 18552 11m 7008 140 1272 5924 7 0 S 15 0 0.0 gnome-settings-
4 PID PPID UID USER RUSER TTY TIME+ %CPU %MEM S COMMAND
3338 1 43 xfs xfs ? 0:00.17 0.0 0.3 S xfs
3290 1 51 smmsp smmsp ? 0:00.00 0.0 0.5 S sendmail
4838 1 500 shan shan ? 0:18.78 0.0 3.5 S nautilus
6891 1 500 shan shan ? 0:06.95 0.0 2.4 S gnome-terminal
4834 1 500 shan shan ? 0:11.20 0.0 2.2 S gnome-panel
5294 1 500 shan shan ? 0:02.36 0.0 2.0 S mixer_applet2
5292 1 500 shan shan ? 0:10.48 0.0 1.9 S wnck-applet
4555 3893 500 shan shan ? 0:03.31 0.0 1.8 S gnome-session
5296 1 500 shan shan ? 0:00.87 0.0 1.5 S clock-applet
4620 1 500 shan shan ? 0:10.79 0.0 1.5 S gconfd-2
4868 1 500 shan shan ? 0:01.52 0.0 1.4 S eggcups
4696 1 500 shan shan ? 0:04.49 0.0 1.4 S metacity
4627 1 500 shan shan ? 0:00.48 0.0 1.4 S gnome-settings-

vmstat -d reports disk

Procs
r: The number of processes waiting for run time.
b: The number of processes in uninterruptible sleep.

Memory
swpd: the amount of virtual memory used.
free: the amount of idle memory.
buff: the amount of memory used as buffers.
cache: the amount of memory used as cache.
inact: the amount of inactive memory. (-a option)
active: the amount of active memory. (-a option)

Swap
si: Amount of memory swapped in from disk (/s).
so: Amount of memory swapped to disk (/s).

IO
bi: Blocks received from a block device (blocks/s).
bo: Blocks sent to a block device (blocks/s).

System
in: The number of interrupts per second, including the clock.
cs: The number of context switches per second.

CPU
These are percentages of total CPU time.
us: Time spent running non-kernel code. (user time, including nice time)
sy: Time spent running kernel code. (system time)
id: Time spent idle. Prior to Linux 2.5.41, this includes IO-wait time.
wa: Time spent waiting for IO. Prior to Linux 2.5.41, included in idle.

st: Time stolen from a virtual machine. Prior to Linux 2.6.11, unknown.

Page Fault:

Technically, page fault happens when the task access a non existant page in its address space.
A page fault is said as "major" if kernel needs to access the disk to make the page available.
On the contrary, soft minor page fault means the kernel only need to allocate pages in RAM without
reading anything from disk.

For illustration, consider the size of program ABC is 8 kB and assume the page size is 4 kB.
When the program is fully loaded to RAM, there will be 2 times major page fault (2 * 4 kB).

The program itself allocates another 8 kB for temporary data storage in RAM. Thus, there will be 2 minor page fault.

A high number of nFLT could mean:

1. The task is aggressively load some portions of its executable or library from the disk.

2. The task is accessing a page that is swapped ou

3. It is normal if you see a high number of major page fault when a program is run for first time.
On the next invocations, buffer is utilized so likely you will see "0" or low number of nFLT.
But, if a program is continously triggerring major page fault, big chance your program needs larger RAM size than currently installed.

Dirty Pages:

The number of dirty pages since they are written back to the disk.

Maybe you wonder, what is dirty page? First, a little bac ground. As you know, Linux employ caching mechanism, so everything that is read
from disk is also cached in RAM. The advantage of this action is, subsequent read to the same disk block can be served from RAM thus reading
completes faster.

But it also costs something. If the buffer's content is modified, it needs to be synchronized. Thus, sooner or la this modified buffer (dirty page)
must be written back. The failure on the synchronization might cause data inconsistency on related disk.

On mostly idle to fairly loaded system, nDRT is usually below 10 (this is just a raw prediction)or mostly zero. If it is constantly bigger than that:

1. The task is aggresively write something to file(s). It is so often that disk I/O can't keep up with it

2. The disk suffers I/O congestion, thus even the task only modifies small portion of file(s), it must wait a bit longer to be synchronized.
Congestion happens when many processes access the disk at a time but cache hit is low.

These days, (1) unlikely happens because I/O speed is getting faster and less CPU demanding (thanks to DMA). So (2) has bigger probability.

Note: On 2.6.x, this field is always zero without unknown reason.

The configuration file consists of records that look identical to the output of sysctl -a.

Here's an example configuration file:

# Controls IP packet forwarding
net.ipv4.ip_forward = 0

# Controls source route verification
net.ipv4.conf.default.rp_filter = 1

# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0

# Controls whether core dumps will append the PID to the core filename.
# Useful for debugging multi-threaded applications.
kernel.core_uses_pid = 1

Network Performance Tuning:

# Decrease the time default value for tcp_fin_timeout connection.
net.ipv4.tcp_fin_timeout = 30

# Decrease the time default value for tcp_keepalive_time connection
net.ipv4.tcp_keepalive_time = 1800

# Turn off tcp_window_scaling
net.ipv4.tcp_window_scaling = 0

# Turn off the tcp_sack
net.ipv4.tcp_sack = 0

# Turn off tcp_timestamps
net.ipv4.tcp_timestamps = 0

NFS Performance Tuning:

# Increase transport socket buffers to improve performance of nfs (and
networking in general)

# 'rmem' is 'read memory', 'wmem' is 'write memory'.

net.core.rmem_max = 262143

net.core.rmem_default = 262143

net.core.wmem_max = 262143

net.core.wmem_default = 262143

net.ipv4.tcp_rmem = 4096 87380 8388608

net.ipv4.tcp_wmem = 4096 87380 8388608

# These are for both security and performance

net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1

Linux ext3 Quick Guide

2010-04-02T12:28:00.000-07:00

Related Topics:
i-node structure is explained at here.
A very good introduction to ext2 is at here.

Ext3 is compatible to ext2 filesystems; actually you can look at it as an ext2 filesystem with a journal file.

The journaling capability means no more waiting for fsck's or worrying about metadata corruption.
What is most noticeable is that you can switch back and forth between ext2 and ext3 on a partition without any problem:
it is just a matter of giving the mount command the right filesystem type.

ext3 journaling options

-o data=writeback
-o data=ordered (default)
-o data=journal

These journaling modes are the following: writing filesystem data and/or filesystem metadata

journal - logs all filesystem data and metadata changes. The slowest of the three ext3 journaling modes,
this journaling mode minimizes the chance of losing the changes you have made to any file in an ext3 filesystem.

ordered - only logs changes to filesystem metadata, but flushes file data updates to disk before making changes
to associated filesystem metadata. This is the default ext3 journaling mode.

writeback - only logs changes to filesystem metadata but relies on the standard filesystem write process to write file
data changes to disk. This is the fastest ext3 journaling mode.

The differences between these journaling modes are both subtle and profound. Using the "journal" mode requires that an ext3
filesystem write every change to a filesystem twice - once to the journal, and then again to the filesystem itself. This can
reduce the overall performance of your filesystem, but is the mode most beloved by users, because it minimizes the chances of
losing changes to your files since both metatdata and data updates are recorded in the ext3 journal and can be replayed when
a system reboots.

Using the "ordered" mode, only filesystem metadata changes are logged, which reduces redundancy between writing to the filesystem
and to the journal and is therefore faster. Though the changes to file data are not logged, they must be done before associated
filesystem metadata changes are made by the ext3 journaling daemon, which can slightly reduce the performance of your system.
However, using this journaling mode guarantees that files in the filesystem will never be out of sync with any related changes to
filesystem metadata.

Using the "writeback" mode is faster than the other two ext3 journaling modes because it only logs changes to filesystem metadata and
does not wait for associated changes to file data to be written before updating things like file size and directory information.
Because updates to file data are done asynchronously to journaled changes to filesystem metadata, files in the filesystem may exhibit
metadata inconsistencies such as owning data blocks to which updated data was not yet written when the system went down.
This isn't fatal, but can be disappointing to users.

Commands:

e2fsck
E2fsck will run the journal automatically, and if the filesystem is otherwise clean, it skip doing a full filesystem check.

By default, exhaustive fs check happens every twentieth mount or every 180 days, whichever comes first.

You can view the filesystem check interval (as well as lots of other interesting information) by typing tune2fs -l /dev/sdXX.

[root@localhost geyes]# tune2fs -l /dev/sdb1
tune2fs 1.35 (28-Feb-2004)
Filesystem volume name:
Last mounted on:
Filesystem UUID: 22e4830c-7f65-43c5-9060-b017b402ecb6
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode filetype sparse_super
Default mount options: (none)
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 32768
Block count: 131056
Reserved block count: 6552
Free blocks: 121221
Free inodes: 32733
First block: 1
Block size: 1024
Fragment size: 1024
Reserved GDT blocks: 256
Blocks per group: 8192
Fragments per group: 8192
Inodes per group: 2048
Inode blocks per group: 256
Filesystem created: Tue Mar 30 22:08:47 2010
Last mount time: Tue Mar 30 22:10:17 2010
Last write time: Tue Mar 30 23:21:51 2010
Mount count: 0
Maximum mount count: 36
Last checked: Tue Mar 30 23:21:51 2010
Check interval: 15552000 (6 months)
Next check after: Mon Sep 27 00:21:51 2010
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 128
Journal inode: 8
Default directory hash: tea
Directory Hash Seed: 33370d2c-2965-45ca-8751-9ba4cc28ecf0
Journal backup: inode blocks

mke2fs -j /dev/hdax

fsck.ext2 -f /dev/hdaX

[root@localhost geyes]# e2fsck /dev/sdb1
e2fsck 1.35 (28-Feb-2004)
/dev/sdb1 is mounted.

WARNING!!! Running e2fsck on a mounted filesystem may cause
SEVERE filesystem damage.

Do you really want to continue (y/n)? yes

/dev/sdb1: recovering journal
/dev/sdb1: clean, 34/32768 files, 9835/131056 blocks

You can convert my ext2 partition to ext3 by two simple steps:
1. creating a journal on it. e.g. "tune2fs -j /dev/hdaX "
2. mount -t ext3 /dev/hdax /mnt/mount-point

With mke2fs -j /dev/hdaX you can format a partition as ext3 (as always it will be also usable as ext2 partion).

This can be done on an unmounted or on a mounted filesystem.

(on mounted partition)
If you create the journal on a mounted filesystem you will see a .journal file.

Don't try to delete this and don't back this up or restore it from backup!

(on unmounted partition)
If you run tune2fs -j on an unmounted partition an unvisible journal file will be created.

'df' command says partition is full, while 'du' reports free space
Some user process keeping a deleted file open.
When this happens, the space is not visible via 'du', since the file is no longer visible in the directory tree.
However, the space is still used by the file until it is deallocated, and that can only happen once the last process
which has the file open either closes its file descriptor to the file, or the process exits.
You can use the lsof program to try to find which process is keeping an open file.
Usually it's some log file, or some large data base file which gets rotated out, but some older process are still keeping the log file open.

If you have ext2 compiled into the kernel and ext3 as a module and your root filesystem is ext2/3, then the kernel will always mount the root fs as ext2 and
not as ext3 since at the time when the root filesystem is mounted, the kernel does not have access to the modules, since they are located on the root filesystem.
(This is a chicken and egg problem!)

If you have this setup, you might first want to consider whether it makes sense to compile ext3 as a module. If you want to use ext3 on your root filesystem,
the ext3 filesystem module will always be loaded and can not be unloaded, so it might as well be compiled-in. Furthermore, modules waste a tiny amount of memory
(on average 2k per module), and take up an extra entry in the TLB cache --- a slight, and perhaps not measurable disadvantage, but given that there is no real advantage
to compiling ext3 as a module, why bother?

If you do want to compile ext3 as a module and use it as your root filesystem, it can be done, but what you must do is do is boot into an initial ramdisk (initrd)
image as your root image. This initrd image will contain the necessary modules (scsi, ext3, etc.) so you can mount your "real" root filesystem and then use pivot_root
to replace the initrd root filesystem with the "real" root filesystem.

Most distributions do the pivot_root trick automatically, but they differ in how the tool which builds the initrd image needs to be called.

* On a SuSE system you have to put "jbd ext3" (in this order!) into the YaST setting INITRD_MODULES. Then do a mk_initrd and it should work.
* On a Red Hat system it seems that it is sufficiant to just do a mkinitrd.
* On a Debian system (woody and above) you must run the command:
mkinitrd -o boot/initrd.img-2.4.18-386 /lib/modules/2.4.18-386
where 2.4.18-386 must be substituted with the version number of the debian kernel which you are currently using. Note: this requires that you have the initrd-tools
package installed, and if you currently are not using an initrd setup, your lilo or grub configuration files must be modified to tell the booting kernel to use
the initrd image.

How do I convert my ext3 partition back to ext2?
Actually there is only little need to do so, because in most cases it is sufficient to mount the partition explicitely as ext2.
But if you really need to convert your partion back to ext2 just do the following on an unmounted partition:

tune2fs -O ^has_journal /dev/hdaX

To be on the safe side you should force a fsck run on this partition afterwards:

fsck.ext2 -f /dev/hdaX

After this procedure you can safely delete the .journal file if there was any.

If a system shutdown hard, even with journaling is it at all necessary to run e2fsck?

It's best to just always run e2fsck. [...]
E2fsck will run the journal automatically, and if the filesystem is otherwise clean, it skip doing a full filesystem check.
If the filesystem is not clean (because during the previous run the kernel noticed some filesystem inconsistencies), e2fsck will automatically do a full check
if it is necessary.
If you have multiple disks, fsck will run multiple e2fsck processes in parallel, thus speeding up your boot sequence than if you let the kernel replay the journal
for each filesystem when it tries to mount it, since then the journal replays will be done sequentially, instead of in parallel.

What is the largest possible size of an ext3 filesystem and of files on ext3?

Ext3 can support files up to 1TB. With a 2.4 kernel the filesystem size is limited by the maximal block device size, which is 2TB.

In 2.6 the maximum (32-bit CPU) limit is of block devices is 16TB, but ext3 supports only up to 4TB.

How do I convert the journal file from version 1 to version 2?
Just type:

mount /dev/hdaX /mnt -o journal=update

to convert your old (ext3 v0.0.3* and earlier) filesystem to the new journal format.

How do I convert my root filesystem from version 1 to version 2?
Just issue a "lilo -R linux rw rootflags=journal=update" and reboot.

I updated ext3 today. Got all of my mounts converted. Now on boot, I see: "EXT3-fs: mounted filesystem with ordered data mode". Is this normal?
Nigel Metheringham answered this on the ext3-users mailing list as follows:

That's fine. The EXT3-fs message is just telling you it mounted the fs OK. It's also telling you what form of journaling you are using.

ext3 has 2 formats of journal:

* version 1 - default and only possibility for ext3 releases 0.0.3* and earlier
* version 2 - default for filesystems created with 0.0.4 and later

Version 2 journals support additional semantics required to allow metadata journaling, and provide 2 new forms of journaling - ordered and writeback.
Both V1 & V2 journals support data journaling (where everything that would go to disk is journaled).

Examples:

[root@localhost sysconfig]# lvmdiskscan
/dev/cdrom [ 68.80 MB]
/dev/sda1 [ 101.94 MB]
/dev/sda2 [ 7.90 GB] LVM physical volume
/dev/sdb [ 512.00 MB]
2 disks
1 partition
0 LVM physical volume whole disks
1 LVM physical volume
[root@localhost sysconfig]# e2fsck /dev/sdb
e2fsck 1.35 (28-Feb-2004)
Couldn't find ext2 superblock, trying backup blocks...
e2fsck: Bad magic number in super-block while trying to open /dev/sdb

The superblock could not be read or does not describe a correct ext2
filesystem. If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193

...
[root@localhost sysconfig]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): m
Command action
a toggle a bootable flag
b edit bsd disklabel
c toggle the dos compatibility flag
d delete a partition
l list known partition types
m print this menu
n add a new partition
o create a new empty DOS partition table
p print the partition table
q quit without saving changes
s create a new empty Sun disklabel
t change a partition's system id
u change display/entry units
v verify the partition table
w write table to disk and exit
x extra functionality (experts only)

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-512, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-512, default 512): 128M

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (129-512, default 129):
Using default value 129
Last cylinder or +size or +sizeM or +sizeK (129-512, default 512): 128M
Value out of range.
Last cylinder or +size or +sizeM or +sizeK (129-512, default 512): +128M

Command (m for help): p

Disk /dev/sdb: 536 MB, 536870912 bytes
64 heads, 32 sectors/track, 512 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 1 128 131056 83 Linux
/dev/sdb2 129 251 125952 83 Linux

Command (m for help): v
782367 unallocated sectors

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@localhost sysconfig]# lvmdiskscan
/dev/cdrom [ 68.80 MB]
/dev/sda1 [ 101.94 MB]
/dev/sda2 [ 7.90 GB] LVM physical volume
/dev/sdb1 [ 127.98 MB]
/dev/sdb2 [ 123.00 MB]
1 disk
3 partitions
0 LVM physical volume whole disks
1 LVM physical volume
[root@localhost sysconfig]# pvscan
PV /dev/sda2 VG VolGroup00 lvm2 [7.88 GB / 32.00 MB free]
Total: 1 [7.88 GB] / in use: 1 [7.88 GB] / in no VG: 0 [0 ]

[root@localhost sysconfig]# mke2fs -j /dev/sdb1
mke2fs 1.35 (28-Feb-2004)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
32768 inodes, 131056 blocks
6552 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67371008
16 block groups
8192 blocks per group, 8192 fragments per group
2048 inodes per group
Superblock backups stored on blocks:
8193, 24577, 40961, 57345, 73729

Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 36 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.

Linux NFS QuickGuide

2010-04-02T12:22:00.000-07:00

Linux NFS QuickGuide

NFS Server Daemons

portmap enables NFS clients to discover the NFS services available on a given NFS server

nfsd provides all nfs servide except file locking and quota management

mountd processes nfs client mount requests

statd implements nfs lock recovery when an nfs server system crashes

lockd starts the kernel's nfs lock manager

rquotad provides file system quota information nfs exports to NFS clients using file system quotas

Daemon start sequence : portmap, nfsd, mountd, statd, rquotad

files:

/exports : lists nfs shares and permissions, mount options, etc

/var/lib/nfs/rmtab : lists each NFS export that is mounted by an NFS client.
each time rpc.mounted receives a mount request, it adds
an entry to /var/lib/nfs/rmtab file.

/var/lib/nfs/etab : contains currently exported filesystems

[root@localhost usr]# cat /etc/exports
#
/usr/share 192.168.200.0/255.255.255.0(rw,insecure,sync,anonuid=600,anongid=600)

share options:

secure : requires client request to originate fr om secure port
secure_lock : requires that clients requsting lock authentication before activating the lock

async : allows the server to cache disk write to improve performance

subtree_check : if only part of a filesystem is exported, this checks to make sure the file is
in the exported protion of the fs

root_squash : maps all requests from a user ID of 0 to uid of anonymous user -2 in linux

anonuid = uid : set the UID of annomous account to uid

To start nfs service:
1. #service nfs start
2. #/etc/rc.d/init.d/nfs start

By default 8 copies of nfsd started, to enable the server to process multiple requests, change this value
/etc/sysconfig/nfs RPCNFSDCOUNT=8

[root@localhost sysconfig]# vi /etc/sysconfig/nfs
[root@localhost sysconfig]# service nfs restart
Shutting down NFS mountd: [ OK ]
Shutting down NFS daemon: [ OK ]
Shutting down NFS quotas: [ OK ]
Shutting down NFS services: [ OK ]
Starting NFS services: [ OK ]
Starting NFS quotas: [ OK ]
Starting NFS daemon: [ OK ]
Starting NFS mountd: [ OK ]

[root@localhost sysconfig]# ps -ef grep nfsd grep -v grep
root 5847 1 0 20:34 ? 00:00:00 [nfsd]
root 5848 1 0 20:34 ? 00:00:00 [nfsd]
root 5849 1 0 20:34 ? 00:00:00 [nfsd]
root 5850 1 0 20:34 ? 00:00:00 [nfsd]
root 5851 1 0 20:34 ? 00:00:00 [nfsd]
root 5852 1 0 20:34 ? 00:00:00 [nfsd]
root 5853 1 0 20:34 ? 00:00:00 [nfsd]
root 5854 1 0 20:34 ? 00:00:00 [nfsd]
root 5855 1 0 20:34 ? 00:00:00 [nfsd]
root 5856 1 0 20:34 ? 00:00:00 [nfsd]
root 5857 1 0 20:34 ? 00:00:00 [nfsd]
root 5858 1 0 20:34 ? 00:00:00 [nfsd]
root 5859 1 0 20:34 ? 00:00:00 [nfsd]
root 5860 1 0 20:34 ? 00:00:00 [nfsd]
root 5861 1 0 20:34 ? 00:00:00 [nfsd]
root 5862 1 0 20:34 ? 00:00:00 [nfsd]

[root@localhost sysconfig]# rpcinfo -p
program vers proto port
100000 2 tcp 111 portmapper
100000 2 udp 111 portmapper
100024 1 udp 32768 status
100024 1 tcp 32770 status
100011 1 udp 932 rquotad
100011 2 udp 932 rquotad
100011 1 tcp 935 rquotad
100011 2 tcp 935 rquotad
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 4 udp 2049 nfs
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs
100021 1 udp 32770 nlockmgr
100021 3 udp 32770 nlockmgr
100021 4 udp 32770 nlockmgr
100021 1 tcp 32776 nlockmgr
100021 3 tcp 32776 nlockmgr
100021 4 tcp 32776 nlockmgr
100005 1 udp 956 mountd
100005 1 tcp 959 mountd
100005 2 udp 956 mountd
100005 2 tcp 959 mountd
100005 3 udp 956 mountd
100005 3 tcp 959 mountd

[root@localhost sysconfig]# cat /var/lib/nfs/rmtab
192.168.200.135:192.168.200.0/255.255.255.0:0x00000001
192.168.200.0/255.255.255.0:/usr/share:0x00000002

[root@localhost sysconfig]# cat /var/lib/nfs/etab
/usr/share 192.168.200.0/255.255.255.0(rw,sync,wdelay,hide,nocrossmnt,insecure,root_squash,no_all_squash,subtree_check,secure_locks,mapping=identity,anonuid=600,anongid=600)

[root@localhost sysconfig]# cat /etc/exports
#
/usr/share 192.168.200.0/255.255.255.0(rw,insecure,sync,anonuid=600,anongid=600)

[root@localhost sysconfig]# exportfs -v
/usr/share 192.168.200.0/255.255.255.0(rw,wdelay,insecure,root_squash,anonuid=600,anongid=600)

root@ubuntu:/mnt# showmount 192.168.200.128
Hosts on 192.168.200.128:
192.168.200.0/255.255.255.0
192.168.200.135

root@ubuntu:/mnt# showmount -a 192.168.200.128
All mount points on 192.168.200.128:
192.168.200.0/255.255.255.0:/usr/share
192.168.200.135:192.168.200.0/255.255.255.0

root@ubuntu:/mnt# showmount -e 192.168.200.128
Export list for 192.168.200.128:
/usr/share 192.168.200.0/255.255.255.0

root@ubuntu:/mnt# showmount -d 192.168.200.128
Directories on 192.168.200.128:
/usr/share
192.168.200.0/255.255.255.0

NFS Client

/etc/fstab
* is the default value

rsize = n *4096 sets the nfs read buffer size to n bytes
wsize = n *4096 sets the nfs write buffer size to n bytes
timeo = n *7 sets the RPC transmission timeout to n thenths of a second
retry = n *10000 sets the time to retry a mount before giving up
port = n *2049 sets the NFS server port to which to connect to
bg *no enables mount attempts to run in the background if first mount attempt time out
fg *yes causes mount attempts to run in foreground if the first mount attempt time out
soft *no allows an NFS file operation to fail and terminate
hard *yes allows an NFS file operation to continue retry after reporting "server not responding"

intr *no allows signals (control+c) to interrupt a failed NFS operation

lock enables NFS locking and starts the statd and lockd daemons

exportfs -o async,rw 192.168.200.128:/var/tmp

mount -t nfs server:/share -o rsize=8192, wsize=8192, hard, intr, nolock

mount -t iso9660 -o ro /dev/cdrom /cdrom

fstab file structure:

/etc/fstab six fields: fstab-sync

(1)block-device-to-be-mounted:
(2)mount-point:
(3)fs-type:
(4)mount-options:
(5)dump command to determine if it needs to be dumped or not 0=no
(6)fsck check-order

[root@localhost /]# cat /etc/fstab
# This file is edited by fstab-sync - see 'man fstab-sync' for details
/dev/VolGroup00/LogVol00 / ext3 defaults 1 1
LABEL=/boot /boot ext3 defaults 1 2
none /dev/pts devpts gid=5,mode=620 0 0
none /dev/shm tmpfs defaults 0 0
none /proc proc defaults 0 0
none /sys sysfs defaults 0 0
/dev/VolGroup00/LogVol01 swap swap defaults 0 0

NFS Troubleshooting:
1. when mount -t nfs remote:share local_mount_point, the system stalls and can't mount...

check /var/log/messages file and see error "kernel: RPC: failed to contact portmap (errno -5)"
portmap (rpc ) is not started.

service portmap status ; to verify portmap is running or stopped
service portmap start or /etc/init.d/portmap start to start portmap

make sure also enable portmap at reboot, e.g. 'chkconfig --level 345 portmap on'

why portmap/rpc must be started to use NFS? check out my blog article "understanding UNIX RPC in 1 minute!"

Thanks for viewing!

Installing Net::SSH::Perl - SSH module for Perl

2010-03-25T17:20:00.000-07:00

Before reading further, do you know how to find out what Perl modules are installed on your system? Hmm...

How about "perldoc perllocal" or

sed -n -e "/head2/p" /usr/local/lib/perl/5.10.0/perllocal.pod | cut -dL -f2 | sort

Anyway, there are few other ways to do it. Make sure you know one of them.

Today, I'm given a task to write a Perl script to scan 200 servers' Interwoven, EPA, JVM and web server configurations. Normally, I'd like to use shell script to do anything related to SSH. However, with so many text files to scan why don't we use Perl to do it?

Perl SSH? Google over the Internet (thanks for the US for not censoring Google's searching result), it appears Net::SSH::Perl is a good but tough candidate. Most reviews complain about the painful installation process which makes it a challenge to me. I like pain, right? (not really)

Now, let's see what is Net::SSH::Perl.

"Net::SSH::Perl is an all-Perl module implementing an SSH client. It is compatible with both the SSH1 and SSH2 protocols.

Net::SSH::Perl enables you to simply and securely execute commands on remote machines, and receive the STDOUT, STDERR, and exit status of that remote command."

Good enough. Let's do it. (For more information about Net::SSH::Perl, see CPAN).

Installation of Net::SSH::Perl is really painful if your system doesn't have the required Math packages.

For cygwin, as of today (March 25,2010), because Perl 5.10.1 is compiled with gcc 4.3.4 (20090804) while gcc for cygwin is only at 3.4.4, the -fstack-protector gcc flags kills you. I tried to force install the module, it always stop at complaining the -fstack-protector flag when running perl script calling SSH module.

You have to recompile gcc in cygwin in order to use this module. To compile gcc, you need m4, Math::Pari, GMP, MFPR... OK, just kill me. I'm giving up.

Luckily, we have Ubuntu. With the pleasant apt-get utility, installation is much better than cygwin.

Installing Net::SSH::Perl module on Linux server (Ubuntu 9.x)

I'm using a dirty quick solution, first, let's cpan the module with force; then, we clean up the mess. Don't try it on production systems.

1. Force install Net::SSH::Perl

cpan> force install Net::SSH::Perl

2. Clean up work. Let's see the error messages at the end of installation and re-install these packages manually.

Go to .cpan/build/ and compile the following packages manually.

ILYAZ/modules/Math-Pari-2.01080604.tar.gz : writemakefile NO '/usr/bin/perl Makefile.PL INSTALLDIRS=site' returned status 2304
VIPUL/Crypt-Random-1.25.tar.gz : make_test FAILED but failure ignored because 'force' in effect
VIPUL/Crypt-Primes-0.50.tar.gz : make_test FAILED but failure ignored because 'force' in effect
VIPUL/Crypt-RSA-1.99.tar.gz : make_test FAILED but failure ignored because 'force' in effect
TURNSTEP/Math-GMP-2.06.tar.gz : make NO
TURNSTEP/Net-SSH-Perl-1.34.tar.gz : make_test FAILED but failure ignored because 'force' in effect

For GMP, because it was never installed, download it from http://gmplib.org/.
However, before we install GMP, we need to install m4...
To Install m4, we need to install (nothing)... there's always a clue that leads to another clue before you can find the treasure.

sudo apt-get install m4

download gmp from http://gmplib.org/ and compile/install it

Now, we need to check what Perl modules are installed on our server: (yes, another good interview question).

1. If "instmodsh" is available, use that.

2. Or use "perldoc perllocal". Or

3. Check the perllocal.pod directly:

root@ubuntu:~/perl# sed -n -e "/head2/p" /usr/local/lib/perl/5.10.0/perllocal.pod | cut -dL -f2 | sort
< Class::
< convert::ascii::armour|convert::ascii::armour>
< convert::asn1|convert::asn1>
< convert::pem|convert::pem>
< crypt::blowfish|crypt::blowfish>
< crypt::cbc|crypt::cbc>
< crypt::des|crypt::des>
< crypt::des_ede3|crypt::des_ede3>
< crypt::dh|crypt::dh>
< crypt::dsa|crypt::dsa>
< crypt::idea|crypt::idea>
< crypt::primes|crypt::primes>
< crypt::primes|crypt::primes>
< crypt::primes|crypt::primes>
< crypt::random|crypt::random>
< crypt::random|crypt::random>
< crypt::random|crypt::random>
< crypt::rsa|crypt::rsa>
< data::buffer|data::buffer>
< dbi|dbi>
< digest::bubblebabble|digest::bubblebabble>
< digest::hmac|digest::hmac>
< digest::md2|digest::md2>
< digest::sha1|digest::sha1>
< file::homedir|file::homedir>
< file::which|file::which>
< ipc::run3|ipc::run3>
< math::pari|math::pari>
< net::ssh::perl|net::ssh::perl>
< sort::versions|sort::versions>
< string::crc32|string::crc32>
< test::script|test::script>
< test::simple|test::simple>
< tie::encryptedhash|tie::encryptedhash>

OK. Now, it's time to check our handy Net::SSH:Perl module.

#!/usr/bin/perl

$host = localhost;
$cmd = "uname -n";
print "Trying to ssh into server $host. Please wait...\n";
use Net::SSH::Perl;
my $ssh = Net::SSH::Perl->new($host);
$ssh->login("shan","newyork1");
$ssh->shell;

root@ubuntu:~/perl# perl ssh1.pl

Trying to ssh into server localhost. Please wait...
Math::BigInt: couldn't load specified math lib(s), fallback to Math::BigInt::FastCalc at /usr/local/share/perl/5.10.0/Crypt/DH.pm line 6
Last login: Thu Mar 25 14:57:18 2010 from localhost
shan@ubuntu:~$ uname -a
uname -a
Linux ubuntu 2.6.31-14-generic #48-Ubuntu SMP Fri Oct 16 14:04:26 UTC 2009 i686 GNU/Linux
shan@ubuntu:~$ id
id
uid=1000(shan) gid=1000(shan) groups=4(adm),20(dialout),24(cdrom),46(plugdev),104(lpadmin),115(admin),120(sambashare),1000(shan)
shan@ubuntu:~$
shan@ubuntu:~$ exit
exit
logout

It opens an ssh connection!

The installation is painful but not completely bad.

Shan's UNIX Admin Blog Worldwide Visitors Tracking Report

2010-01-07T14:44:00.000-08:00

Note:
All visitors private data (such as location, IP address, OS, configuration, referring URL, browsing history, etc.) is protected and won't be released to third party.

Welcome to my blog, happy surfing!

December 2009 Visitor Tracking Report

Visitors by Continent (6 Continents including an "Unknown Continent"... Mars?)

(click image to enlarge)

Visitors by Country (21 Countries)

Visitors by Domain (Last 72 hours data only)

Visitors by Organization

Visitors by Operating System

Visitors by Desktop Resolution (The largest one was 3480 x 1200, what a good video card!)

Visitors Site Duration

Visitors Daily Pageview

Perl Regular Expression Summary

2010-01-06T09:07:00.000-08:00


Meta Characters
\s   white space [\f\t\n\r ] or form-feed, tab, newline, carriage return, space
\d   any digit [0-9]
\w   word character [A-Za-z0-9_].  


Matches with m//

m/\w*?/  m(\w*?) m{w*?} or m#http://# ....


Perl's Option Modifiers  /s /i /x or "/six"

/s : #it searches pattern in multiple lines across newlines '\n'; the "." looks within single line
/i : #case (i)nsensitive
/x : #add white space, useful for adding comments in regular expressions

You can combine option modifiers


Word Anchors /\bword\b/ or #\bword\b#

#\bperl#   matches  "perlscripting"

#perl\b#   matches  "useperl"

#\bsearc\B# matches "searches", "searching" but not "search" or "researching"



The Binding Operator  =~

my #anwser = <STDIN> =~ #\byes\b/i;



The Match Variables  $1, $2, ..., $` $& $'

if (#(\S+); (\S+); (\S+)#) {
print "$1    $2     $3 \n";
}

Note: sometimes, you need quote '$1' in print"";


It's always good to save your match in a variable instead of $1 as $1 may be changed by other matches

if ($statement =~ /(\w+)/) {
my $magicword = $1;
...
}


Parentheses for grouping only - non-capturing parentheses (?:)


 1 #!/usr/bin/perl
 2 use strict;
 3
 4 while (<>) {
 5 if ( m#(?:UNIX).*?(Solaris|AIX|Linux)#i )  {
 6     print "$1\n";
 7 }
 8 }


$ cat job.txt
Required skills:
1) 5 to 10 years Perl or Python , shell scripting skills;
2) at least 10 years UNIX e.g. AIX Solaris Linux Administration skills;
3) 5 to 10 years weblogic, tomcat, websphere administration skills;
Preferred skills:
1) UNIX Internal Coding Skills with Python, Perl or C;
2) TCP/IP Internal and Protocols;
3) Understand Java Servlets/JSP/Web Applications;
4) Understand Oracle/Sybase/MySQL;



# $1 is AIX not UNIX because (?:UNIX) is non-capturing.
$ ./98.pl < job.txt
AIX

shan@InternetPC ~/perl
$


Named Captures

In addition to capturing parts of string in $1, $2, $3...
Perl allows you to name captures in hash named %+: the key
is the lable we used and the value is the matched part in regex.



 1 #!/usr/bin/perl
 2 use 5.010;
 3
 4
 5 while (<>) {
 6 if ( m/(?:UNIX)\s*(?<os1>\w+) (?<os2>\w+) (?<os3>\w+) / ) {
 7 say "Required Operating Systems are $+{os1} $+{os2} $+{os3}\n";
 8 }
 9 }

$ ./912.pl < job.txt
Required Operating Systems are AIX Solaris Linux



NEED to know vim copy groups between (...), eg copy block of chars inside parentheses

*****************************
Now that we have a way to label matches, we also need a way to refer to them for back
references. Previously, we used either \1 or \g{1} for this. With a labeled group, we can
use the label in \g{label}:
use 5.010;
my $names = 'Fred Flinstone and Wilma Flinstone';
if( $names =~ m/(?<last_name>\w+) and \w+ \g{last_name}/ ) {
say "I saw $+{last_name}";
}
We can do the same thing with another syntax. Instead of using \g{label}, we use
\k<label>:?
use 5.010;
my $names = 'Fred Flinstone and Wilma Flinstone';
if( $names =~ m/(?<last_name>\w+) and \w+ \k<last_name>/ ) {

******************************

General Quantifiers





Substitutions with S///  : search and replace

$_ = "green scaly dinosaur";
s/(\w+) (\w+)/$2, $1/; # Now it's "scaly, green dinosaur"
s/^/huge, /; # Now it's "huge, scaly, green dinosaur"
s/,.*een//; # Empty replacement: Now it's "huge dinosaur"
s/green/red/; # Failed match: still "huge dinosaur"
s/\w+$/($`!)$&/; # Now it's "huge (huge !)dinosaur"
s/\s+(!\W+)/$1 /; # Now it's "huge (huge!) dinosaur"
s/huge/gigantic/; # Now it's "gigantic (huge!) dinosaur"


There is a useful Boolean value from s///;
it is true if a substitution was successful;
otherwise, it is false:

$_ = "fred flintstone";
if (s/fred/wilma/) {
print "Successfully replaced fred with wilma!\n";
}


Option Modifiers: /s /i /x  or "six"

s{ABC}{}s;

The Binding Operator : =~


Case Shifting

\U      ; # \U escape forces what follows to all upper case
\L      ; # \L escape forces lowercase

$_=" Steve and John are good friends\n";
s/(steve|john)/\U$1/gi        -> STEVE and JOHN ...
s/(steve|john)/|L$1/gi        -> steve and john ...



You can even stack them up.
Using \u with \L means ¡°all lowercase, but capitalize the
first letter¡±:*

s/(fred|barney)/\u\L$1/ig; # $_ is now "I saw Fred with Barney."

The split and join operators
@fields = split /separator/, $string;
@fields = split /:/, "abc:def::g:h"; # gives ("abc", "def", "", "g", "h")

my $result = join $glue, @pieces;
my $x = join ":", 4, 6, 8, 10, 12; # $x is "4:6:8:10:12"

m// in list context

When a pattern match (m//) is used in a list context, the return value is a list of the
memory variables created in the match, or an empty list if the match failed:
$_ = "Hello there, neighbor!";
my($first, $second, $third) = /(\S+) (\S+), (\S+)/;
print "$second is my $third\n";

my $text = "Fred dropped a 5 ton granite block on Mr. Slate";
my @words = ($text =~ /([a-z]+)/ig);
print "Result: @words\n";
# Result: Fred dropped a ton granite block on Mr Slate


Nongreedy Quantifiers "?"   :  *? +?


curl 'http://www.google.com' | perl -e 'while (<>) { m#<style>(.*?)</style>#; print "$&\n";}'



Matching Multiple-Line Text  "/m"  m=multiple lines

This is where Perl beats classic regular expressions. Perl can match multiple lines of text
just as matching single lines.

13 #9.15
14 $_="The only reason I'm using Perl is because \n it is perfect for processing text files \n with the built-in regular expressions in Perl.\n";
15
16 print "Found 'perl' at start of line\n" if /\bperl\b/im;
17 print "$`:$&:$'\n";

Found 'perl' at start of line
The only reason I'm using :Perl: is because
it is perfect for processing text files
with the built-in regular expressions in Perl.


19 #9.16 read entire google.html file into one variable, then add "__HTML__" in front of each line
20 $filename="/home/shan/perl/google.html";
21 open FILE, $filename
22 or die "Can't open '$filename': $!";
23 my $lines = join '', <FILE>;
24 $lines =~ s/^/__HTML__: /gm;
25 print $lines;


Updating Many Files:

 1 #!/usr/bin/perl -w
 2 # perl script to process many files
 3 #
 4 # usage : prcfile.pl input-file
 5 #
 6 # it will run regex operation inside while on input-file
 7 # and saves the original file in the name input-file.org
 8
 9
10 use strict;
11
12 #name your own file extension for backup copy of the original file
13 $^I = ".org";
14
15 chomp (my $date = `date`);
16 while (<>) {
17
18 #insert your regex operations here:
19 s/^/$date:/;
20 s/$/___END___/;
21
22 #update
23 print;
24 }


$./prcfile 9.pl

$ more 9.pl
Mon Dec 14 17:09:45 PST 2009:#!/usr/bin/perl___END___
Mon Dec 14 17:09:45 PST 2009:___END___
Mon Dec 14 17:09:45 PST 2009:___END___
Mon Dec 14 17:09:45 PST 2009:___END___
Mon Dec 14 17:09:45 PST 2009:$_="steve and John are good friends\n";___END___
Mon Dec 14 17:09:45 PST 2009:s/(steve|john)/\U$1/gi;        #-> STEVE and JOHN ...___END___
Mon Dec 14 17:09:45 PST 2009:print "$_\n";___END___
Mon Dec 14 17:09:45 PST 2009:___END___
Mon Dec 14 17:09:45 PST 2009:$_="steve and John are good friends\n";___END___
Mon Dec 14 17:09:45 PST 2009:s/(steve|john)/\L$1/gi;        #-> steve and john ...___END___
Mon Dec 14 17:09:45 PST 2009:print "$_\n";___END___
Mon Dec 14 17:09:45 PST 2009:___END___
Mon Dec 14 17:09:45 PST 2009:#9.15___END___
Mon Dec 14 17:09:45 PST 2009:$_="The only reason I'm using Perl is because \n it is perfect for processing text files \n with the built-in regular expressions in Perl.\n";___END___
Mon Dec 14 17:09:45 PST 2009:___END___
Mon Dec 14 17:09:45 PST 2009:print "Found 'perl' at start of line\n" if /\bperl\b/im;___END___
Mon Dec 14 17:09:45 PST 2009:print "$`:$&:$'\n";___END___
Mon Dec 14 17:09:45 PST 2009:___END___
Mon Dec 14 17:09:45 PST 2009:#9.16___END___
Mon Dec 14 17:09:45 PST 2009:$filename="/home/shan/perl/google.html";___END___
Mon Dec 14 17:09:45 PST 2009:open FILE, $filename___END___
Mon Dec 14 17:09:45 PST 2009:or die "Can't open '$filename': $!";___END___
Mon Dec 14 17:09:45 PST 2009:my $lines = join '', <FILE>;___END___
Mon Dec 14 17:09:45 PST 2009:$lines =~ s/^/__HTML__: /gm;___END___
Mon Dec 14 17:09:45 PST 2009:print $lines;___END___



or, we can use command line to do the same:

perl -p -i.org -w -e 'chomp (my $date=`date`);s/^/$date:/g;' 9.pl


Here are the explanations of these command line options,

1. "perl -p"  :

while ($_ = <STDIN>) {
print $_;
}

$_ , = and STDIN are all optional.

therefore, the above code can be written as:

while (<>) {
print;
}

2. "-i.org"  #sets $^I to ".org"

3. "-w"      #turns on warnings

4. -e " executable codes "

5. @ARGV

Pages in My Blog Ranked as Top 10 Search Results in Google

2010-01-05T19:58:00.000-08:00

Keywords "troubleshooting solaris 10 jumpstart" or "solaris jumpstart interrupt"
Technical area: Solaris Administration
Google Rank: First (click image to enlarge)

Keywords "aix paging space weblogic"
Technical area: AIX Administration, Weblogic Administration, Performance Tuning
Google Rank: Top 3/First

Keywords "crashdump + savecore + solaris 10"
Technical area: Solaris Administration
Google Rank: Top 4 (search engine google.ca)

Keywords "aix sybase tuning"
Technical area: AIX administration, sybase performance tuning
Google Rank : Top 10 (Engine: google.com.co)

Keywords "aix mirrorvg sync"
Technical area: AIX Administration
Google Rank: Top 3 (January 2010 google korea)

Appearance in a Korean user's google korea search engine

Keywords "maxperm sybase ase"
Technical area: AIX/Sybase Performance Tuning
Google Rank: Top 3 (December 2009)

Keywords " rpcinfo clnt_create"
Technical area: UNIX Client/Server RPC
Google Rank: Top 3 (December 2009)

Keywords "showmount -e RPC Program Not Registered"
Technical area: UNIX NFS
Google Rank: Top 3

Keyword "vimshell"
Technical area: UNIX utility
Google Rank : Top 10 (December 2009)

Keywords "userauth_failure" or "ssh state_expect_newkeys"
Technical area: SSH troubleshooting
Google Rank: Top 10 (December 2009)

Keywords "fatal accept_ctx died"
Technical area: ssh/scp/sftp troubleshooting
Google Rank: Top 3

Weblogic Setup Quick Guide

2010-01-05T14:25:00.000-08:00

General Procedures:

1. Setup Unix environments (user/group ids, filesystems, tools, access - sudo, etc.)
2. Install WebLogic, JDK, Web Application Development Tools (ant, XML editors EditiX)
3. If production requires middle tier High Availability, setup Veritas VCS application agent monitoring java process and filesystems. I posted a Weblogic HA setup procedure in 2002 at THIS SITE.
4. Setup Web Application Containers in Weblogic environment

Weblogic Admin port 7001

Variables:
export JRE_HOME=/opt/SDK/jdk
export JAVA_HOME=/opt/SDK/jdk
export CLASSPATH=/data/lib/servlet-api.jar

Sample servlet WAR structure:


myServletWAR\
  META-INF\
  WEB-INF\
     classes\
        HelloServlet.class
     web.xml

web.xml deployment descriptor

<servlet>
web.xml

<servlet>
   <servlet-name>myHello</servlet-name>
   <servlet-class>HelloServlet</servlet-class>
</servlet>

<servlet-mapping>
   <servlet-name>myHello</servlet-name>
   <url-pattern>/hello</url-pattern>
</servlet-mapping>
   <servlet-name></servlet-name></servlet><servlet-mapping><url-pattern></url-pattern>
</servlet-mapping>

Sample Weblogic Installation:

The following Products and JDKs will be installed:
--------------------------------------------------


WebLogic Platform 10.3.2.0
|_____WebLogic Server
|    |_____Core Application Server
|    |_____Administration Console
|    |_____Configuration Wizard and Upgrade Framework
|    |_____Web 2.0 HTTP Pub-Sub Server
|    |_____WebLogic SCA
|    |_____WebLogic JDBC Drivers
|    |_____Third Party JDBC Drivers
|    |_____WebLogic Server Clients
|    |_____WebLogic Web Server Plugins
|    |_____UDDI and Xquery Support
|_____Oracle Enterprise Pack for Eclipse
|    |_____Common Files
|_____JDKs
|_____Sun SDK 1.6.0_14
|_____Oracle JRockit 1.6.0_14 SDK

*Estimated size of installation: 1,251.9 MB

Production Mode
Sun SDK 1.6
JRocket SDK 1.6

admin server ipc4.shanjing.com:7001

Startup of Weblogic

./wlserver_10.3/samples/domains/wl_server/

-bash-3.00$ ./startWebLogic.sh
.
.
JAVA Memory arguments: -Xms512m -Xmx512m
.
WLS Start Mode=Development
.
CLASSPATH=/data/weblogic/mw/wlserver_10.3/samples/server/examples/build/serverclasses:/data/weblogic/mw/patch_wls1032/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/data/weblogic/mw/patch_oepe1032/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/data/weblogic/mw/jrockit_160_14_R27.6.5-32/lib/tools.jar:/data/weblogic/mw/utils/config/10.3/config-launch.jar:/data/weblogic/mw/wlserver_10.3/server/lib/weblogic_sp.jar:/data/weblogic/mw/wlserver_10.3/server/lib/weblogic.jar:/data/weblogic/mw/modules/features/weblogic.server.modules_10.3.2.0.jar:/data/weblogic/mw/wlserver_10.3/server/lib/webservices.jar:/data/weblogic/mw/modules/org.apache.ant_1.7.0/lib/ant-all.jar:/data/weblogic/mw/modules/net.sf.antcontrib_1.0.0.0_1-0b2/lib/ant-contrib.jar::/data/weblogic/mw/wlserver_10.3/common/eval/pointbase/lib/pbembedded57.jar:/data/weblogic/mw/wlserver_10.3/common/eval/pointbase/lib/pbclient57.jar:/data/weblogic/mw/wlserver_10.3/server/lib/xqrl.jar
.
PATH=/data/weblogic/mw/wlserver_10.3/server/bin:/data/weblogic/mw/modules/org.apache.ant_1.7.0/bin:/data/weblogic/mw/jrockit_160_14_R27.6.5-32/jre/bin:/data/weblogic/mw/jrockit_160_14_R27.6.5-32/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin
.
***************************************************
* To start WebLogic Server, use a username and *
* password assigned to an admin-level user. For *
* server administration, use the WebLogic Server *
* console at http://hostname:port/console *
***************************************************
starting weblogic with Java version:
java version "1.6.0_14"
Java(TM) SE Runtime Environment (build 1.6.0_14-b08)
BEA JRockit(R) (build R27.6.5-32_o-121899-1.6.0_14-20091001-2113-linux-ia32, compiled mode)
Starting WLS with line:
/data/weblogic/mw/jrockit_160_14_R27.6.5-32/bin/java -jrockit -Xms512m -Xmx512m -Dweblogic.Name=examplesServer -Djava.security.policy=/data/weblogic/mw/wlserver_10.3/server/lib/weblogic.policy -Xverify:none -da -Dplatform.home=/data/weblogic/mw/wlserver_10.3 -Dwls.home=/data/weblogic/mw/wlserver_10.3/server -Dweblogic.home=/data/weblogic/mw/wlserver_10.3/server -Dweblogic.management.discover=true -Dwlw.iterativeDev= -Dwlw.testConsole= -Dwlw.logErrorsToConsole= -Dweblogic.ext.dirs=/data/weblogic/mw/patch_wls1032/profiles/default/sysext_manifest_classpath:/data/weblogic/mw/patch_oepe1032/profiles/default/sysext_manifest_classpath weblogic.Server
<Dec 1, 2009 7:41:58 PM PST> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with BEA JRockit(R) Version R27.6.5-32_o-121899-1.6.0_14-20091001-2113-linux-ia32 from BEA Systems, Inc.>
<Dec 1, 2009 7:41:59 PM PST> <Info> <Management> <BEA-141107> <Version: WebLogic Server 10.3.2.0 Tue Oct 20 12:16:15 PDT 2009 1267925 >
<Dec 1, 2009 7:42:04 PM PST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
<Dec 1, 2009 7:42:04 PM PST> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool>
<Dec 1, 2009 7:42:05 PM PST> <Notice> <Log Management> <BEA-170019> <The server log file /data/weblogic/mw/wlserver_10.3/samples/domains/wl_server/servers/examplesServer/logs/examplesServer.log is opened. All server side log events will be written to this file.>
<Dec 1, 2009 7:42:13 PM PST> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>
<Dec 1, 2009 7:42:27 PM PST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STANDBY>
<Dec 1, 2009 7:42:27 PM PST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
<Dec 1, 2009 7:42:52 PM PST> <Notice> <Store> <BEA-280067> <JDBC store "exampleJDBCStore" did not find a database table at "examplesWLStore", so it created one using the commands in file "/weblogic/store/io/jdbc/ddl/pointbase.ddl".>
<Dec 1, 2009 7:43:25 PM PST> <Warning> <EJB> <BEA-010202> <Call-by-reference is not enabled for the EJB 'ReviewListenerBean'. The server will have better performance if it is enabled. To enable call-by-reference, set the enable-call-by-reference element to True in the weblogic-ejb-jar.xml deployment descriptor or corresponding annotation for this EJB.>
<Dec 1, 2009 7:43:25 PM PST> <Warning> <EJB> <BEA-010202> <Call-by-reference is not enabled for the EJB 'ReviewManagerBean'. The server will have better performance if it is enabled. To enable call-by-reference, set the enable-call-by-reference element to True in the weblogic-ejb-jar.xml deployment descriptor or corresponding annotation for this EJB.>
129 reviewService INFO [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] openjpa.Runtime - Starting BEA Kodo 4.2.0
131 reviewService INFO [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] openjpa.jdbc.JDBC - Using dictionary class "org.apache.openjpa.jdbc.sql.PointbaseDictionary".
WARN: Creating new records from /META-INF/data.properties

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
After the server has booted, your browser should
automatically launch and point to the
Oracle WebLogic Server Samples Introduction Page
running on this server. If your browser fails to launch,
point your browser to the following URL:
"http://192.168.1.104:7001/examplesWebApp/index.jsp"
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[LoginFilter]: LoginFilter constructed ...
<Dec 1, 2009 7:46:38 PM PST> <Notice> <Log Management> <BEA-170027> <The Server has established connection with the Domain level Diagnostic Service successfully.>
<Dec 1, 2009 7:46:41 PM PST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to ADMIN>
<Dec 1, 2009 7:46:41 PM PST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RESUMING>
<Dec 1, 2009 7:46:44 PM PST> <Notice> <Security> <BEA-090171> <Loading the identity certificate and private key stored under the alias DemoIdentity from the jks keystore file /data/weblogic/mw/wlserver_10.3/server/lib/DemoIdentity.jks.>
<Dec 1, 2009 7:46:45 PM PST> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the jks keystore file /data/weblogic/mw/wlserver_10.3/server/lib/DemoTrust.jks.>
<Dec 1, 2009 7:46:45 PM PST> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the jks keystore file /data/weblogic/mw/jrockit_160_14_R27.6.5-32/jre/lib/security/cacerts.>
<Dec 1, 2009 7:46:46 PM PST> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=KEYNECTIS ROOT CA,OU=ROOT,O=KEYNECTIS,C=FR". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Dec 1, 2009 7:46:46 PM PST> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=T-TeleSec GlobalRoot Class 3,OU=T-Systems Trust Center,O=T-Systems Enterprise Services GmbH,C=DE". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Dec 1, 2009 7:46:46 PM PST> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=T-TeleSec GlobalRoot Class 2,OU=T-Systems Trust Center,O=T-Systems Enterprise Services GmbH,C=DE". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Dec 1, 2009 7:46:48 PM PST> <Notice> <Server> <BEA-002613> <Channel "DefaultSecure[3]" is now listening on 127.0.0.1:7002 for protocols iiops, t3s, ldaps, https.>
<Dec 1, 2009 7:46:48 PM PST> <Notice> <Server> <BEA-002613> <Channel "DefaultSecure[2]" is now listening on 0:0:0:0:0:0:0:1:7002 for protocols iiops, t3s, ldaps, https.>
<Dec 1, 2009 7:46:48 PM PST> <Notice> <Server> <BEA-002613> <Channel "Default[1]" is now listening on fe80:0:0:0:20c:29ff:fe07:b3ab:7001 for protocols iiop, t3, ldap, snmp, http.>
<Dec 1, 2009 7:46:48 PM PST> <Notice> <Server> <BEA-002613> <Channel "Default" is now listening on 192.168.1.104:7001 for protocols iiop, t3, ldap, snmp, http.>
<Dec 1, 2009 7:46:48 PM PST> <Notice> <Server> <BEA-002613> <Channel "Default[3]" is now listening on 127.0.0.1:7001 for protocols iiop, t3, ldap, snmp, http.>
<Dec 1, 2009 7:46:48 PM PST> <Notice> <Server> <BEA-002613> <Channel "Default[2]" is now listening on 0:0:0:0:0:0:0:1:7001 for protocols iiop, t3, ldap, snmp, http.>
<Dec 1, 2009 7:46:48 PM PST> <Notice> <Server> <BEA-002613> <Channel "DefaultSecure" is now listening on 192.168.1.104:7002 for protocols iiops, t3s, ldaps, https.>
<Dec 1, 2009 7:46:48 PM PST> <Notice> <Server> <BEA-002613> <Channel "DefaultSecure[1]" is now listening on fe80:0:0:0:20c:29ff:fe07:b3ab:7002 for protocols iiops, t3s, ldaps, https.>
<Dec 1, 2009 7:46:48 PM PST> <Notice> <WebLogicServer> <BEA-000331> <Started WebLogic Admin Server "examplesServer" for domain "wl_server" running in Development Mode>
Exception in thread "Thread-14" java.lang.InternalError: Can't connect to X11 window server using '192.168.1.71:0.0' as the value of the DISPLAY variable.
at sun.awt.X11GraphicsEnvironment.initDisplay(Native Method)
at sun.awt.X11GraphicsEnvironment.access$100(X11GraphicsEnvironment.java:52)
at sun.awt.X11GraphicsEnvironment$1.run(X11GraphicsEnvironment.java:155)
at sun.awt.X11GraphicsEnvironment.<clinit>(X11GraphicsEnvironment.java:131)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at java.awt.GraphicsEnvironment.getLocalGraphicsEnvironment(GraphicsEnvironment.java:68)
at sun.awt.X11.XToolkit.<clinit>(XToolkit.java:89)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at java.awt.Toolkit$2.run(Toolkit.java:834)
at java.awt.Toolkit.getDefaultToolkit(Toolkit.java:826)
at java.awt.Desktop.isDesktopSupported(Desktop.java:151)
at java.awt.Desktop.getDesktop(Desktop.java:125)
at examples.webapp.lifecycle.OpenBrowserServletContextListener.openBrowser(Unknown Source)
at examples.webapp.lifecycle.OpenBrowserServletContextListener.run(Unknown Source)
at java.lang.Thread.run(Thread.java:619)
<Dec 1, 2009 7:46:55 PM PST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RUNNING>
<Dec 1, 2009 7:46:55 PM PST> <Notice> <WebLogicServer> <BEA-000360> <Server started in RUNNING mode>

Terms:

Web Application Container normally in a WAR file - contains an application's resources such as:
a. servlets (java class)
b. JavaServer Pages (JSPs)
c. JSP tag libraries
d. any static resources such as html pages, image files etc.

Servlets - java class, HTTP Servlet - html inside Java
JSP - JavaServer Pages - java inside html
Weblogic provides J2EE (Java Enterprise Environment) environment and services/APIs to web application containers, this services include EJBs,
database connections (JDBC), Java Messaging Sevice (JMS), XML and more.

Servlets
A servlet is a Java class that runs in a Java-enabled server. An HTTP servlet is a special type of servlet that handles an HTTP request and provides an HTTP response, usually in the form of an HTML page. The most common use of WebLogic HTTP Servlets is to create interactive applications using standard Web browsers for the client-side presentation while WebLogic Server handles the business logic as a server-side process. WebLogic HTTP servlets can access databases, Enterprise JavaBeans, messaging APIs, HTTP sessions, and other facilities of WebLogic Server.

WAR files
Web Archive Files - A WAR file contains
a. one servlet/one JSP along with helper classes,
b. a weblogic.xml deployment descriptor, an XML document containing Weblogic Server specific elements for web applcations
c. optional web.xml deployment descriptor, XML file describes the contents of the war file
d. any static html pages/image files

The WAR file can be deployed alone or packaged in an Enterprise application archive (EAR file) with other application components. If deployed alone,
the archive must end with a .war extension. If deployed in an EAR file, the archive must end with an .ear extension.

Oracle recommends that you package and deploy your stand-alone Web applications as part of an Enterprise application. This is an Oracle best practice,
which allows for easier application migration, additions, and changes. Also, packaging your applications as part of an Enterprise application allows
you to take advantage of the split development directory structure, which provides a number of benefits over the traditional single directory structure.

EAR file (Enterprise Application Archive) .ear file, the .WAR file can be deployed alone or packaged in an EAR file.
JSPs and HTTP servlets can access all services and APIs available in WebLogic Server. These services include EJBs, database connections by way of
Java Database Connectivity (JDBC), Java Messaging Service (JMS), XML, and more.

Installing mod_jk in 1 Minute!

2010-01-05T14:15:00.000-08:00

Apache mod_jk Installation Quick Guide

mod_jk is AJP connector connecting Apache with tomcat. It enables users access tomcat/JSP thru Apache.

Key Components
1. httpd.conf LoadModule
2. workers.properties
3. modules/mod_jk.so

Two ways of installing modules in apache, Static or DSO (Dynamic Shared Object):

Static - compiled with apache ./configure --with-mod_jk
DSO - must have mod_so compiled with apache, use apxs (apache extensons) to add.

"httpd -l" to see static modules compiled with apache, mod_so.c must be on the list to use DSO.

"httpd -M" to see list of all modules both static and dynamic

-bash-3.00$ /usr/local/apache2/bin/httpd -M
Loaded Modules:
core_module (static)
authn_file_module (static)
authn_default_module (static)
authz_host_module (static)
authz_groupfile_module (static)
authz_user_module (static)
authz_default_module (static)
auth_basic_module (static)
auth_digest_module (static)
include_module (static)
filter_module (static)
deflate_module (static)
log_config_module (static)
env_module (static)
headers_module (static)
setenvif_module (static)
version_module (static)
ssl_module (static)
mpm_prefork_module (static)
http_module (static)
mime_module (static)
status_module (static)
autoindex_module (static)
asis_module (static)
cgi_module (static)
negotiation_module (static)
dir_module (static)
actions_module (static)
userdir_module (static)
alias_module (static)
rewrite_module (static)
so_module (static)
jk_module (shared)

To install via DSO:

Building mod_ jk on Linux/*nix

You need to make sure you have the required gcc compiler and libc versions on your Linux system to
compile the Apache Web server and mod_jk . The specific version required is dependent on the Apache
Web server that you will be using. In addition, the script used by mod_jk installation requires installation
of libtool 1.5.2 or higher, and at least autoconf 2.59 .

1. Download the source of the Apache Web server, corresponding to the version of the server you
are using, from http://httpd.apache.org/download.cgi .

2. Unpack the distribution into any convenient directory, such as /home/dev/httpd-2.2.3 .

3. Download the latest Connector source from http://tomcat.apache.org/downloadconnectors.cgi

4. Unpack it to a convenient directory, such as /home/dev/tomcat-connects-1.2.19-src .

5. Compile and install the Apache server. You should follow the detailed instructions in the install
file of your Apache server source directory. With the 2.2.3 server, you need to issue the command
./configure --prefix=/installdir where you can specify the location ( installdir ) where
you want the server to be installed. After configuration finishes, type the make command to compile
the source. Finally, type the make install command to install the server.

6. This step compiles the mod_jk module as a dynamically loadable module. Change into the
native subdirectory. Then enter the command ./configure --with-apxs=/installdir/bin/apxs .
The installdir must be the same as the installation directory of your Web server in step 5;
be aware that this is not the same as the source code directory for the Web server. This creates
a makefile. You can then type make to compile the mod_jk module. You can then find the
mod_jk.so module in the apache-2.0 subdirectory.

Basic Terms:

Servlets

A servlet is a Java class that runs in a Java-enabled server. An HTTP servlet is a special type of servlet that handles an HTTP request and provides an HTTP response, usually in the form of an HTML page. The most common use of WebLogic HTTP Servlets is to create interactive applications using standard Web browsers for the client-side presentation while WebLogic Server handles the business logic as a server-side process. WebLogic HTTP servlets can access databases, Enterprise JavaBeans, messaging APIs, HTTP sessions, and other facilities of WebLogic Server.

Servlets and Java EE

WebLogic Server fully supports HTTP servlets as defined in the Servlet 2.5 specification from Sun Microsystems.

The Servlet 2.5 specification defines the implementation of the servlet API and the method by which servlets are deployed in enterprise applications. Deploying servlets on a Java EE-compliant server, such as WebLogic Server, is accomplished by packaging the servlets and other resources that make up an enterprise application into a single unit, the Web application. A Web application utilizes a specific directory structure to contain its resources and a deployment descriptor that defines how these resources interact and how the application is accessed by a client. See The Web Applications Container.

What You Can Do with Servlets

* Create dynamic Web pages that use HTML forms to get end-user input and provide HTML pages that respond to that input. Examples of this utilization include online shopping carts, financial services, and personalized content.

* Create collaborative systems such as online conferencing.

* Easily deploy servlets written for any Java EE-compliant servlet engine to WebLogic Server.

* Have access to a variety of APIs and features by using servlets running in WebLogic Server. For example:
1) Session tracking—Allows a Web site to track a user’s progress across multiple Web pages. This functionality supports Web sites such as e-commerce sites that use shopping carts. WebLogic Server supports session persistence to a database, providing fail-over between server down time and session sharing between clustered servers. For more information see Session Tracking from a Servlet.
2) JDBC drivers (including Oracle)—JDBC drivers provide basic database access. With WebLogic Server’s multi-tier JDBC implementations, you can take advantage of connection pools, server-side data caching, and transactions. For more information see Accessing Databases.
3) Enterprise JavaBeans—Servlets can use Enterprise JavaBeans (EJB) to encapsulate sessions, data from databases, and other functionality. See Referencing External EJBs, More about the ejb-ref* Elements, and Referencing Application-Scoped EJBs.
4) Java Messaging Service (JMS)—JMS allows your servlets to exchange messages with other servlets and Java programs. See Programming WebLogic JMS.
5) Java JDK APIs—Servlets can use the standard Java JDK APIs.
6) Forwarding requests—Servlets can forward a request to another servlet or other resource. Forwarding a Request.

WAR files
Web Application Development Key Points

JSPs and HTTP servlets can access all services and APIs available in WebLogic Server. These services include EJBs, database connections by way of
Java Database Connectivity (JDBC), Java Messaging Service (JMS), XML, and more.

A Web archive (WAR file) contains:

* One servlet or Java Server Page (JSP), along with any helper classes.
* A weblogic.xml deployment descriptor, which is an XML document containing WebLogic Server-specific elements for Web applications.
* An optional web.xml deployment descriptor, which is a Java EE standard XML document that describes the contents of a WAR file.
* A WAR file can also include HTML or XML pages and supporting files such as image and multimedia files.

The WAR file can be deployed alone or packaged in an Enterprise application archive (EAR file) with other application components. If deployed alone, the
archive must end with a .war extension. If deployed in an EAR file, the archive must end with an .ear extension.

Oracle recommends that you package and deploy your stand-alone Web applications as part of an Enterprise application. This is an Oracle best practice,
which allows for easier application migration, additions, and changes. Also, packaging your applications as part of an Enterprise application allows
you to take advantage of the split development directory structure, which provides a number of benefits over the traditional single directory structure.

Generate SSL Certificate for Apache Web Server

2010-01-05T14:01:00.000-08:00

(A self-signed certificate in my testing web server)

1. Create a directory, for example, under apache2/conf/
# mkdir certworks
# cd certworks

2. Create a configuration file for generating a certificate
# cat myconfig.file
RANDFILE =./random.txt
[req]
default_bits = 1024
default_keyfile = keyfile.pem
attributes = req_attributes
distinguished_name = ShanJing
prompt = no
output_password = mypassword
[ShanJing]
C = US
ST = CA
L = Arcadia
O = ShanJing
OU = Shan's IT
CN = 192.168.1.104
emailAddress = mail@shanjing.com
[req_attributes]
challengePassword = mypassword

3. Create a private key and a Certificate Signing Request
# openssl req -new -out server.csr -config myconfig.file

Generating a 1024 bit RSA private key
.......++++++
...........++++++
writing new private key to 'keyfile.pem'
-----

# ls -al
total 48
drwxr-xr-x 2 root root 4096 Dec 5 11:35 .
drwxr-xr-x 5 root root 4096 Dec 5 11:29 ..
-rw-r--r-- 1 root root 963 Dec 5 11:35 keyfile.pem
-rw-r--r-- 1 root root 351 Dec 5 11:33 myconfig.file
-rw------- 1 root root 1024 Dec 5 11:35 random.txt
-rw-r--r-- 1 root root 737 Dec 5 11:35 server.csr

4. Remove the Passphrase from the private key so that when apache starts, it won't ask for key's Passphrase
# openssl rsa -in keyfile.pem -out server.key
Enter pass phrase for keyfile.pem:
writing RSA key

5. For test, we just generate a self-signed certificate.
# openssl x509 -in server.csr -out server.crt -req -signkey server.key -days 365

Signature ok
subject=/C=US/ST=CA/L=Arcadia/O=ShanJing/OU=Shans IT/CN=192.168.1.104/emailAddress=mail@shanjing.com
Getting Private key

6. Setup mod-ssl in Apache:

Modify the following in conf/extra/httpd-ssl.conf file:

SSLCertificateFile "/usr/local/apache2/conf/certworks/server.crt"

SSLCertificateKeyFile "/usr/local/apache2/conf/certworks/server.key"

Modify httpd.conf and enable/uncomment

Include conf/extra/httpd-ssl.conf

(Note, if mod_ssl is compiled with apache as built-in module, you don't need to put the following line in httpd.conf file:
LoadModule ssl-module modules/mod_ssl.so

To find out if mod_ssl is compiled with apache,

# /usr/local/apache2/bin/apachectl -l | grep mod_ssl
mod_ssl.c

To see currently loaded modules:
[root@ipc4 extra]# /usr/local/apache2/bin/apachectl -t -D DUMP_MODULES
Loaded Modules:
core_module (static)
authn_file_module (static)
authn_default_module (static)
authz_host_module (static)
authz_groupfile_module (static)
authz_user_module (static)
authz_default_module (static)
auth_basic_module (static)
auth_digest_module (static)
include_module (static)
filter_module (static)
deflate_module (static)
log_config_module (static)
env_module (static)
headers_module (static)
setenvif_module (static)
version_module (static)
ssl_module (static)
mpm_prefork_module (static)
http_module (static)
mime_module (static)
status_module (static)
autoindex_module (static)
asis_module (static)
cgi_module (static)
negotiation_module (static)
dir_module (static)
actions_module (static)
userdir_module (static)
alias_module (static)
rewrite_module (static)
so_module (static)
jk_module (shared)
Syntax OK

)

Reference: How SSL security works?

Understands Openssl and Public Key Crytopgraphy in 1 Minute!

2010-01-05T13:58:00.000-08:00

##########
Public Key Crytopgraphy
##########

1. Diffie-Hellman (DH)
2. Digital Signature Algorithm (DSA)
3. RSA (Riverst,Shamir,Adleman)
4. S/MIME

1. Diffie-Hellman
#openssl dhparam -out dhparam.pem -2 1024
#openssl dhparam -in dhparam.pem -noout -C

2. DSA
#openssl dsaparam -out dsaparam.pem 1024
#openssl gendsa -out dsaprivatekey.pem -des3 desparam.pem
#openssl dsa -in dsaprivatekey.pem -pbout -out dsapublickey.pem
#openssl dsa -in dsaprivatekey.pem -out dsaprivatekey.pem -des3 \
-passin pass:oldpasswd -passout pass:newpasswd

3. RSA
#openssl genrsa -out rsaprivatekey.pem -passout pass:open123 -des3 1024
#openssl rsa -in rsaprivatekey.pem -passin pass:open123 -pubout \
-out rsapublickey.pem
#openssl rsautl -encrypt -pubin -inkey rsapublickey.pem -in plaintxt \
-out cipher.txt
#openssl reautl -decrypt -inkey rsaprivatkey.pem -in cipher.txt -out plaintxt
#openssl rsautl -sign -inkey rsaprivatekey.pem -in plain.txt \
-out signature.bin
#openssl rsautl -verify -publin -inkey rsapublickey.pem -in \
signature.bin -out plain.txt

4. S/MIME
#openssl smine -encrypt -in mail.txt -des3 -out mail.enccert.pem
#openssl smine -decrypt -in mail.enc -recip cert.pem -inkey key.pem \
-out mail.sgn
#openssl smine -sign -in mail.txt -signer cert.pem -inkey key.pem \
-out mail.sgn
#openssl smine -verify -in mail.sgn -out mail.txt

#openssl dhparam -out dhparam.pem -2 1024
[shan@ipc4 openssl]$ more dhparam.pem
-----BEGIN DH PARAMETERS-----
MIGHAoGBAM34BWAn2CV8+utL5S9EOew5RYScXu5FjvyisMs/Eyn698FKgu20tonA
O9q9462n3lT6bzsMIOkfn2VnKWO8oqk9gOAkTICgVATiS59NFvN21t8okNHXR4TJ
2rWmhBRcBhZQgGZw0jPlHDE6FEfjqrAoHuu5SvJXZu4L6KV1n7H7AgEC
-----END DH PARAMETERS-----

Reads a set of Diffie-Hellman paramenters from the file dhparam.pem and writes
a C code respresentation of the parameters to stout.

[shan@ipc4 openssl]$ openssl dhparam -in dhparam.pem -noout -C
#ifndef HEADER_DH_H
#include
#endif
DH *get_dh1024()
{
static unsigned char dh1024_p[]={
0xCD,0xF8,0x05,0x60,0x27,0xD8,0x25,0x7C,0xFA,0xEB,0x4B,0xE5,
0x2F,0x44,0x39,0xEC,0x39,0x45,0x84,0x9C,0x5E,0xEE,0x45,0x8E,
0xFC,0xA2,0xB0,0xCB,0x3F,0x13,0x29,0xFA,0xF7,0xC1,0x4A,0x82,
0xED,0xB4,0xB6,0x89,0xC0,0x3B,0xDA,0xBD,0xE3,0xAD,0xA7,0xDE,
0x54,0xFA,0x6F,0x3B,0x0C,0x20,0xE9,0x1F,0x9F,0x65,0x67,0x29,
0x63,0xBC,0xA2,0xA9,0x3D,0x80,0xE0,0x24,0x4C,0x80,0xA0,0x54,
0x04,0xE2,0x4B,0x9F,0x4D,0x16,0xF3,0x76,0xD6,0xDF,0x28,0x90,
0xD1,0xD7,0x47,0x84,0xC9,0xDA,0xB5,0xA6,0x84,0x14,0x5C,0x06,
0x16,0x50,0x80,0x66,0x70,0xD2,0x33,0xE5,0x1C,0x31,0x3A,0x14,
0x47,0xE3,0xAA,0xB0,0x28,0x1E,0xEB,0xB9,0x4A,0xF2,0x57,0x66,
0xEE,0x0B,0xE8,0xA5,0x75,0x9F,0xB1,0xFB,
};
static unsigned char dh1024_g[]={
0x02,
};
DH *dh;

if ((dh=DH_new()) == NULL) return(NULL);
dh->p=BN_bin2bn(dh1024_p,sizeof(dh1024_p),NULL);
dh->g=BN_bin2bn(dh1024_g,sizeof(dh1024_g),NULL);
if ((dh->p == NULL) || (dh->g == NULL))
{ DH_free(dh); return(NULL); }
return(dh);
}

DSA
[shan@ipc4 openssl]$ openssl dsaparam -out dsaparam.pem 1024
Generating DSA parameters, 1024 bit long prime
This could take some time
....+.......+++++++++++++++++++++++++++++++++++++++++++++++++++*
.+...+..+....+.+.................................+......+.......+.....+.....+...
...+.+..................+..............................+....++++++++++++++++++++
+++++++++++++++++++++++++++++++*
[shan@ipc4 openssl]$ openssl gendsa -out dsaprivatekey.pem -des3 dsaparam.pem
Generating DSA key, 1024 bits
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
[shan@ipc4 openssl]$
[shan@ipc4 openssl]$ openssl dsa -in dsaprivatekey.pem -pubout -out
dsapublickey.pem
read DSA key
Enter PEM pass phrase:
writing DSA key
[shan@ipc4 openssl]$ ls
dhparam.pem dsaparam.pem dsaprivatekey.pem dsapublickey.pem openssl.txt
[shan@ipc4 openssl]$ cat dsa*pem
-----BEGIN DSA PARAMETERS-----
MIIBHgKBgQC5Xhv0mFXaAYPwHVU/NKuYwkwd13cYttAoKlqoong7AB9wyCY1f4v5
g7OFoiKSZwZbnZdF6E5yId5VF8VR6usU0jNkef96X9tnwHM4SBjd7sZJsrgB4VFb
cK+BinOhRiplnA2j3iPwK0+UpS9lqwflVPBtZp77qdhr+sCqH40OHQIVAIHQo2SJ
3cyL99bjvxgHuV4yYNJdAoGAYi3X8zyjO5kb23TZEO365NLNd76gxgPjftwCleHk
VileZ5oRr7Ysfd13icvrAHp/STyszTUezWhDSoi5c/sZqK5CkYTyWGxBA2Q4pJzr
o96NAuDx+HY/mzDPiYxW8+vgcFKjK8ZtRPVdYm7/pzOPAsK7uTYl7dsvczAzq4sy
xOQ=
-----END DSA PARAMETERS-----
-----BEGIN DSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,90B967D1B13289CA

jVh7gsM6R6HRSu8yRKP10UOQ5fJGBZTFMc1BbVZqAcT8GRYBuMhJ3MsL35g2chuT
+W3n0KxhGmU9RH+hSMyrPrVWy82FAXdWUjipDyCm/3nfrrvTunE2RCy242axBH9n
JqxFRLDnTgfcgMw3fxK9QGfbKn2+IxG4kQmxJaWbzH9ooCJPuNIllztyRHCGNNyr
9ognjrdFiVh506Yz7xOjPtvnzMry0rZ6B97LjqH0MixMS4HFGUgBeuv06fulBgte
GDIdSYqqYmfhweOFEVpW4WJqV7heJVDDSkjYcVFv0ITfUa3LIZl8HUIf06sog91m
ENWqKfoky5fFRVN3urmcGmOgXs3UdAiIdHop6tWXFskTQ8+FlI5BSDLM8wwqNsAN
hVENfZeSqHL5Qbiq0J1TyjTHO4NcOrrub57wm3tfIXofEIdWbR0Qp+hoGUOZZZ0g
2ODcE6Y/oSTOhTNYVPBSL9MKZO8NmO2kjgllEJdMiJRFr1eGvlMBq3OP+9E+Oadk
tnqjW7ITfjSByGe7V33JR6iuogBCGBFsNB9DhGGshw7KJjy4Swlf119Ba9W1v6Wl
YhwpIyw1YHhY6IVjqtoFmQ==
-----END DSA PRIVATE KEY-----
-----BEGIN PUBLIC KEY-----
MIIBtzCCASsGByqGSM44BAEwggEeAoGBALleG/SYVdoBg/AdVT80q5jCTB3Xdxi2
0CgqWqiieDsAH3DIJjV/i/mDs4WiIpJnBludl0XoTnIh3lUXxVHq6xTSM2R5/3pf
22fAczhIGN3uxkmyuAHhUVtwr4GKc6FGKmWcDaPeI/ArT5SlL2WrB+VU8G1mnvup
2Gv6wKofjQ4dAhUAgdCjZIndzIv31uO/GAe5XjJg0l0CgYBiLdfzPKM7mRvbdNkQ
7frk0s13vqDGA+N+3AKV4eRWKV5nmhGvtix93XeJy+sAen9JPKzNNR7NaENKiLlz
+xmorkKRhPJYbEEDZDiknOuj3o0C4PH4dj+bMM+JjFbz6+BwUqMrxm1E9V1ibv+n
M48Cwru5NiXt2y9zMDOrizLE5AOBhQACgYEAhSOzZHU0XD3InVY1GzMsrmCcR0SC
PYFRtdchZqXpT/D30aul/9k4I57g6Bo4wo0pDcaNcZmaqQCzW8x5jvIeQ3jtXBsb
Dl8oLkl+XubhitHsZSJLblf8/g/+bB19iTOj1EMf3g449o/gn+p52HNhc8NszaxQ
yOkeP1KaA7kKDkU=
-----END PUBLIC KEY-----

[:wopenssl dsa -in dsaprivatekey.com -out dsaprivatekey.pem -des3
-passin pass:letmein -passout pass:letmein01
726 openssl dsa -in dsaprivatekey.pem -out dsaprivatekey.pem -des3
-passin pass:letmein -passout pass:letmein01
727 ls
728 openssl genrsa -out rsaprivatekey.pem -passout pass:letmein -des3
1024
729 openssl rsa -in rsaprivatekey.pem -passin pass:letmein -pubout -out
rsapublickey.pem
730 openssl rsautl -encrypt -pubin -inkey rsapublickey.pem -in
openssl.txt -out openssl.txt.cip

Understand ssh agent forwarding in 1 minute!

2010-01-05T13:45:00.000-08:00

ssh agent forwarding

Key Concepts:
1. understand what is a key challenge
2. agent constructs response to remote server's key challenge without revealing the private key;
3. the "middleman" server's sshd daemon acts as agent and relays ssh authentication traffic between the client and destination hosts;

The difference between basic ssh connection and agent based ssh connection:

The very basic ssh connection is ssh client (instead of agent) uses private key to construct response for remote server's key challenge.

ssh agent forwarding means agent can verify a user's identity without revealing the private key to the remote host.
With agent, the ssh client passes the remote key challenge to the agent. The agent uses the private key to construct
a response key-challenge and sends it back to ssh process which sends it off to the remote sshd.

Example:

(assume ssh port 22 or whatever is opened on all servers)

A user shan on ipc4 has established ssh connection with server1 via agent.

with agent-forwarding, user shan@ipc4 has established ssh access to server1 with agent, now shan@ipc4 needs to access server2 which has his account's public key.

shan@ipc4 issues ssh connection request via agent-forwarding to server2 thru server1; the ssh client on server1 receives the key challenge from server2, it forwards that challenge to the
sshd daemon on the same machine acting as a key agent, server1's sshd relays the key challenge to the ipc4's ssh client
who sends the challenge to the agent on ipc4. ipc4's agent contructs the key response using the private key and sends back to
server1's sshd which acts as agent then sends back to server2.

What's key challenge

server:
encrypt (random number + public key) => key challenge --------> client

client:
decrypt (key challenge + private key) = challenge.txt, key response = MD5 hash ( challenge txt + session ID ) ----- server

server encrypts a large random number with the user's public key to create a key challenge then sends back to client.
the client must use user's private key to decrypt the key challenge.

Key response
When the agent receives the challenge, it decrypts it with the private key. If this key is the "other half" of the public key
on the server, the decryption will be successful, revealing the original random number generated by the server. Only the holder
of the private key could ever extract this random number, so this constitutes proof that the user is the holder of the private key.

The agent takes this random number, appends the SSH session ID (which varies from connection to connection), and creates an MD5
hash value of the resultant string: this result is sent back to the server as the key response.

The server computes the same MD5 hash (random number + session ID) and compares it with the key response from the agent: if they
match, the user must have been in possession of the private key, and access is granted. If not, the next key in the list (of any)
is tried in succession until a valid key is found, or no more authorized keys are available. At that point, access is denied.

Curiously, the actual random number is never exposed in the client/agent exchange - it's sent encrypted to the agent, and
included in an MD5 hash from the agent. It's likely that this is a security precaution designed to make it harder to characterize
the properties of the random number generator on the server by looking at the the client/agent exchange.

Reference: SSH The Definitive Guide (Chapter 6 - Key Management and Agents)

Installing Ubuntu 9.10 on Vmware Hosts and Configure Ubuntu Server

2009-12-26T11:20:00.000-08:00

It said "Install Windows 7 or better", so I installed Ubuntu.....
Redhat is supposed to be the leader in open source community; however, with long releasing cycle and sort of "closed" packing practices, Redhat is gradually losing the leadership. Issues like the painful process of upgrading Python in Redhat Linux distribution has bothered the open source community. It's time to look for a better open source Linux distribution - Ubuntu.

The installation is very easy - download the iso image of ubuntu from its website, create a new vmware host with 8GB virtual disk and 1GB RAM (my laptop has total 4GB RAM), choose "ubuntu" as operating system (these technology leaders really know how to cooperate with each other!), setup NAT network for the new host and pick the iso image as boot image (With vmware's virtual CDROM, I didn't even need to burn the iso file to CD); finally fire up the vmware host then take a break practicing DJHero Daft Punk Megamix. 20 minutes later, my Toshiba laptop powered by Intel Core 2 Duo Processor, 4GB RAM is welcoming a new vmware host member - ubuntu.

(Ubuntu 9.10 Desktop Screenshots)

Bash is installed as default shell, most popular open source tools are installed. There are two distributions - Desktop and Server. Desktop has the GUI environment and most UNIX end-user tools installed, Python 2.6 and gcc along with other development tools are installed; however, server products like apache, tomcat, are not. yum is also not installed. For Sys Admins, you may have to choose to install the desktop or server edition. I actually have a very good solution - install both and convert the desktop edition to server by reinstalling server's extra packages (the actually upgrade process is at the end of this blog).

Using ubuntu is just as easy as installing it. One of the nice features for ubuntu is the ability to update packages automatically. The system silently checks for the latest packages, releases. When a cluster of new packages is available, it will pop up a menu and suggest users to update... Making it short, I really don't want to look back once I start to use ubuntu.

With 3 major UNIX OSs in one laptop (Solaris, RHEL, Ubuntu), the fun just begins... Very soon, I probably need a new laptop with Quad Core CPU and 8GB - 16GB RAM.

Cheers!

Is the terminal's transparent mode correct?

Ubuntu Server Configuration:

Server installation doesn't come with GUI. For most Intel Duo Core CPU laptops/PCs, installing 64-bit server may create problems. Someone has re-compiled ubuntu kernel on these CPUs, if you need to borrow the compiled kernels, google "ubuntu 64 bit kernel for Intel Duo Core". If you don't care about performance and ubuntu's 3-D desktop features (actually I do!) 32 bit kernel is an easy option for most Laptops/PCs.

The standard server installation doesn't have all the packages by default. But the installation is fairly simple and straight.

Command execution is very very fast!

To install yum, sshd, apache and other packages:

sudo apt-get install yum
sudo apt-get install openssh-server
sudo apt-get install gcc
sudo apt-get install apache2

To get a list of selections:
root@ubuntu:~# dpkg --get-selections | awk '{rpint $1}' > pkglist

To install all perl packages:
root@ubuntu:~# for i in $(dpkg --get-selections | grep perl | awk '{print $1}'); do echo $i; apt-get -y install $i; done

To find out what packages are installed:
root@ubuntu:~#dpkg --list

To find out files owned by a package:
root@ubuntu:~# dpkg -L openjdk-6-jre
/.
/usr
/usr/lib
/usr/lib/jvm
/usr/lib/jvm/java-6-openjdk
/usr/lib/jvm/java-6-openjdk/jre
/usr/lib/jvm/java-6-openjdk/jre/bin
/usr/lib/jvm/java-6-openjdk/jre/bin/javaws
/usr/lib/jvm/java-6-openjdk/jre/bin/pluginappletviewer
/usr/lib/jvm/java-6-openjdk/jre/bin/policytool
/usr/lib/jvm/java-6-openjdk/jre/bin/javaws.real

(Ubuntu 9.10 Server, DJHero Daft Punk Megamix, PacMan)

(auto) Install server packages on ubuntu desktop host: (this will give you GUI and all the packages on the server)

create a list of installed packages on both desktop and server, e.g.

On the desktop:
#dpkg --get-selections* | awk '{print $1}' > pkglist.desktop

On the server:
#dpgk --get-selections* | awk '{print $1}' > pkglist.server
scp server's pkglist to the host running desktop edition.

On the desktop host, install the server's additional packages:

for i in $(diff pkglist pkglist2 | grep '<' | cut -d' ' -f2); do echo $i; apt-get -y install $i; done

Next, install tomcat and verify apache2 mod_jk and mod_proxy modules. Create and import a test database into mysql. Perl's base packages are installed, I cpan a few Perl modules for database and web scripting. For additional packages, apt-get and yum are our best friends.

Because my laptop's video card, I am not able to play with Ubuntu's 3D desktop. I'll leave that to my desktop computer.

That's it! In less than an hour, starting from scratch, we build a fully loaded standalone Linux server with beautiful desktop and faster, smoother performance!

Solaris 10 Administration

2009-12-26T11:07:00.000-08:00

(Sun E10000, 64 CPUs and 64 GB RAM, 16 SBs, up to 16 domains)

Solaris Volume Manager

Metastate Database
The state database stores information on disk about the state of VM config.
Multiple copies of the database, called replicas, provide redundancy and
protect against data loss if a copy of the database is corrupted.

1 Disk - min 3 replicas
2-4 Disks - 2 replicas each disk
5 or more disks - 1 replica on each disk

Majority Consensus Alogrithm
- at least half of the replicas are available
- causes the system to panic if fewer than half of the replicas available
- won't start VM unless majority replicas are available

#metadb -a [-f] [-c n] [-l nn] disk_slice

-a adds a state database replica
-f forces the operation
-c n specifies the number of the replicas to add to the slice
-l nnn specifies the size of the new replicas, in blocks
disk_slice specifies the disk_slice will hold the replica

#metadb -a -f c0t0d0s4 c0t0d0s5 c1t0d0s1
#metadb -a -f -c 2 c3t2d0s7 c3t3d0s7

#metadb

#metainit -f name numstripes width component...

#metainit -f d0 2 1 c0t0d0s7 c3t2d0s0

#metastat

#ls -iL /dev/md/dsk

#ls -il /dev/md/rdsk

#growfs -M /export/home /dev/md/rdsk/d0

SVM Operation Examples:
------------------------------------------------------------------------------

vfstab

#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#
fd - /dev/fd fd - no -
/proc - /proc proc - no -
/dev/md/dsk/d1 - - swap - no -
/dev/md/dsk/d0 /dev/md/rdsk/d0 / ufs 1 no logging
/dev/md/dsk/d6 /dev/md/rdsk/d6 /zones ufs 2 yes -
/devices - /devices devfs - no -
ctfs - /system/contract ctfs - no -
objfs - /system/object objfs - no -
swap - /tmp tmpfs - yes -

------------------------------------------------------------------------------

ROOT@server1:/root # metadb
flags first blk block count
a m p luo 16 8192 /dev/dsk/c0t0d0s7
a p luo 8208 8192 /dev/dsk/c0t0d0s7
a p luo 16400 8192 /dev/dsk/c0t0d0s7

a = active
size = 8192blk = 4MB

---------------------------------------------------------------------------

layout of the disk

Part Tag Flag Cylinders Size Blocks
0 root wm 2241 - 4481 8.00GB (2241/0/0) 16780608
1 swap wu 0 - 2240 8.00GB (2241/0/0) 16780608
2 backup wm 0 - 38342 136.91GB (38343/0/0) 287112384
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 0 0 (0/0/0) 0
6 unassigned wm 4482 - 6722 8.00GB (2241/0/0) 16780608
7 unassigned wm 6723 - 6731 32.91MB (9/0/0) 67392

----------------------------------------------------------------------------

metadb - creates state db replicas

metadb -a -f c0t0d0s4 c0t0d0s5 c1t0d0s0 c1t0d0s1
metadb -a -f -c n -l nnn disk_slice
-f force operation
-c n specifies the number of replicas to add to the slice
-l nnn specifies the size of the new replica in blocks
disk_slice specifies the slice will hold the replica

metainit - creates the metadevices.

metainit -f concat/stripe numstripes witdth component

metainit -f d0 2 1 c0t0d0s7 1 c3t2d0s0

metastat - checks the metadevice configuration

ROOT@server1:/root # metastat
d6: Mirror
Submirror 0: d16
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 16780608 blocks (8.0 GB)

d16: Submirror of d6
State: Okay
Size: 16780608 blocks (8.0 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c0t0d0s6 0 No Okay Yes

d1: Mirror
Submirror 0: d11
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 16780608 blocks (8.0 GB)

d11: Submirror of d1
State: Okay
Size: 16780608 blocks (8.0 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c0t0d0s1 0 No Okay Yes

d0: Mirror
Submirror 0: d10
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 16780608 blocks (8.0 GB)

d10: Submirror of d0
State: Okay
Size: 16780608 blocks (8.0 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c0t0d0s0 0 No Okay Yes

Device Relocation Information:
Device Reloc Device ID
c0t0d0 Yes id1,sd@SFUJITSU_MAW3147NC_______DAF4P7400LPR

ROOT@server1:/root # ls -lL /dev/md/dsk
total 0
brw-r----- 1 root sys 85, 0 Jul 19 16:39 d0
brw-r----- 1 root sys 85, 1 Jul 20 06:12 d1
brw-r----- 1 root sys 85, 10 Jul 20 06:12 d10
brw-r----- 1 root sys 85, 11 Jul 20 06:12 d11
brw-r----- 1 root sys 85, 16 Jul 20 06:12 d16
brw-r----- 1 root sys 85, 6 Jul 19 16:40 d6

Create a swap and a dump fs:

1. create partition 3 and 4: (using the number and lable command in format)
partition> p
Current partition table (unnamed):
Total disk cylinders available: 38343 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks
0 root wm 2241 - 4481 8.00GB (2241/0/0) 16780608
1 swap wu 0 - 2240 8.00GB (2241/0/0) 16780608
2 backup wm 0 - 38342 136.91GB (38343/0/0) 287112384
3 unassigned wm 6732 - 22415 56.00GB (15684/0/0) 117441792
4 unassigned wm 22416 - 26897 16.00GB (4482/0/0) 33561216
5 unassigned wm 0 0 (0/0/0) 0
6 unassigned wm 4482 - 6722 8.00GB (2241/0/0) 16780608
7 unassigned wm 6723 - 6731 32.91MB (9/0/0) 67392

2. Create the meta device d3 and d4

/usr/sbin/metainit -f d3 1 1 c0t0d0s3

/usr/sbin/metainit -f d4 1 1 c0t0d0s4

3. check:

metastat

d4: Concat/Stripe
Size: 33561216 blocks (16 GB)
Stripe 0:
Device Start Block Dbase Reloc
c0t0d0s4 0 No Yes

d3: Concat/Stripe
Size: 117441792 blocks (56 GB)
Stripe 0:
Device Start Block Dbase Reloc
c0t0d0s3 0 No Yes

Device Relocation Information:
Device Reloc Device ID
c0t0d0 Yes id1,sd@SFUJITSU_MAW3147NC_______DAF4P7400LPR

-----------------------------------------------------------------------------------------
ROOT@server1:/root # newfs -N -v /dev/md/rdsk/d4
mkfs -F ufs -o N /dev/md/rdsk/d4 33561216 -1 -1 8192 1024 155 1 167 8192 t 0 -1 8 16 n
Warning: 3456 sector(s) in last cylinder unallocated
/dev/md/rdsk/d4: 33561216 sectors in 5463 cylinders of 48 tracks, 128 sectors
16387.3MB in 342 cyl groups (16 c/g, 48.00MB/g, 5824 i/g)
Warning: 0 sector(s) in last cylinder unallocated
/dev/md/rdsk/d4: 33561216 sectors in 4482 cylinders of 8 tracks, 936 sectors
16387.3MB in 321 cyl groups (14 c/g, 51.19MB/g, 6208 i/g)
Error determining alternate superblock locations
-------------------------------------------------------------------------------------------

ROOT@server1:/root # newfs -v /dev/md/rdsk/d4
newfs: construct a new file system /dev/md/rdsk/d4: (y/n)? y
mkfs -F ufs /dev/md/rdsk/d4 33561216 -1 -1 8192 1024 155 1 167 8192 t 0 -1 8 16 n
Warning: 3456 sector(s) in last cylinder unallocated
/dev/md/rdsk/d4: 33561216 sectors in 5463 cylinders of 48 tracks, 128 sectors
16387.3MB in 342 cyl groups (16 c/g, 48.00MB/g, 5824 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
32, 98464, 196896, 295328, 393760, 492192, 590624, 689056, 787488, 885920,
Initializing cylinder groups:
......
super-block backups for last 10 cylinder groups at:
32638496, 32736928, 32835360, 32933792, 33032224, 33130656, 33229088,
33327520, 33425952, 33524384

copy rootdisk's vtoc to second disk

ROOT@server1:/root # prtvtoc -h /dev/dsk/c0t0d0s2 fmthard -s - /dev/rdsk/c2t0d0s2

create the replica of metadb c2t0d0 disk on slice 7
ROOT@server1:/root # metadb -a -c 3 c2t0d0s7
ROOT@server1:/root # metadb
flags first blk block count
a m p luo 16 8192 /dev/dsk/c0t0d0s7
a p luo 8208 8192 /dev/dsk/c0t0d0s7
a p luo 16400 8192 /dev/dsk/c0t0d0s7
a u 16 8192 /dev/dsk/c2t0d0s7
a u 8208 8192 /dev/dsk/c2t0d0s7
a u 16400 8192 /dev/dsk/c2t0d0s7

2. Mirror existing meta devices:

a. create submirror meta devices
metainit -f d26 1 1 c2t0d0s6
metainit -f d21 1 1 c2t0d0s1
metainit -f d20 1 1 c2t0d0s0

b. metastat shows current meta device d1 has only one submirror md d11 -

ROOT@server1:/root # metastat d1
d1: Mirror
Submirror 0: d11
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 16780608 blocks (8.0 GB)

d11: Submirror of d1
State: Okay
Size: 16780608 blocks (8.0 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c0t0d0s1 0 No Okay Yes

Device Relocation Information:
Device Reloc Device ID
c0t0d0 Yes id1,sd@SFUJITSU_MAW3147NC_______DAF4P7400LPR

c. attach one submirror md - d21

ROOT@server1:/root # metattach d1 d21
d1: submirror d21 is attached

d. metastat shows d21 is attached to d1 as a submirror.
now, d1 has two submirror mds d11 and 21

ROOT@server1:/root # metastat d1
d1: Mirror
Submirror 0: d11
State: Okay
Submirror 1: d21
State: Resyncing
Resync in progress: 0 % done
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 16780608 blocks (8.0 GB)

d11: Submirror of d1
State: Okay
Size: 16780608 blocks (8.0 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c0t0d0s1 0 No Okay Yes

d21: Submirror of d1
State: Resyncing
Size: 16780608 blocks (8.0 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c2t0d0s1 0 No Okay Yes

Device Relocation Information:
Device Reloc Device ID
c0t0d0 Yes id1,sd@SFUJITSU_MAW3147NC_______DAF4P7400LPR
c2t0d0 Yes id1,sd@SFUJITSU_MAW3147NC_______DAF4P7400M19

Create a md mirror from slice - d13 and d23 are submirrors of d3

a. create md from slices
metainit -f d13 1 1 c0t0d0s3
metainit -f d23 1 1 c0t1d0s3

b. link d13 as the first submirror of d3
metainit d3 -m d13

c. link d23 as submirror of d3
metattach d3 d23

AIX Administration

2009-12-20T14:25:00.000-08:00

(IBM 9117-570 8-Way LPAR managed by Dual HMCs. EMC DMX Frame 2TB, Sepaton Tape Library. OS: AIX, Applications: Sybase ASE 12.5, Veritas VCS. Function: Backend data warehouse)

AIX Paging Space Management

An very good guide for AIX memory usage is at HERE.

; See real memory size
lsattr -El sys0 | grep realmem

bootinfo -r

svmon -G

; Create Paging
mkps -a -n -s8 rootvg hdisk1
mkps -a -n -s4 rootvg hdisk3

#swapon /dev/paging01

#lsps -a

Display paging:

#lsps -a
Page Space Physical Volume Volume Group Size %Used Active Auto Type
hd6 hdisk0 rootvg 512MB 1 yes yes lv
paging00 hdisk0 rootvg 2560MB 1 yes yes lv

Shrink paging space on the fly:

chps -d decrement_LVs -a y|n paging00

Shrink paging00 by 4 LPs and activate the paging on each subsequent reboot

#chps -d 4 -a y paging00 ; reduce swap space

#swapoff /dev/paging00 ; offline swap space

#rmps paging00 ; to remove paging

;all paging space are defined in /etc/swapspaces file

;to activate all paging space at once
swapon -a

;to deactivate paging03 and remove
swapoff /dev/paging03
rmps paging03

;to reduce the size of paging by two LPs
chps -d2 paging03

;paging summary
mkps
rmps
chps
lsps
swapoff
swapon
chps -s
chps -d
shrinkps
svmon
topas
vmstat
sar
istat
slibclean
ps

Mirrorring Rootdisk

2009-12-20T14:17:00.001-08:00

Set Boot list for different options:
===================================================================
bootlist -m normal -o ; to see the bootlist
bootlist -m service -o ; to see the maintenance bootlist
bootlist -m normal -o hdisk0 hdisk4

To mirror rootvg, current bootdisk is hdisk0.
===================================================================
1. Add disk
extendvg -f rootvg hdisk1
chvg -Qn rootvg

2. Mirror data (create LVM structure and sync data)

# mklvcopy hd1 2 hdisk1
# mklvcopy hd2 2 hdisk1
# mklvcopy hd3 2 hdisk1
# mklvcopy hd4 2 hdisk1
# mklvcopy hd5 2 hdisk1
# mklvcopy hd6 2 hdisk1
# mklvcopy hd7 2 hdisk1
# mklvcopy hd8 2 hdisk1
# mklvcopy hd9var 2 hdisk1
# mklvcopy hd10opt 2 hdisk1
# mklvcopy hd11msb 2 hdisk1
# mklvcopy lvtivoli 2 hdisk1

Or use
mirrorvg rootvg ; this will disable qurum

Or use
mirrorvg -s rootvg ; just create LVM structure, no data sync
syncvg -v rootvg ; sync the rootvg data

or use
mirrorvg -S rootvg ; sync data in the background

3.create BLV and boot record

bosboot -ad hdisk1 ; recreate BLV and the boot record on hdisk1
bosboot -ad -u /dev/hdisk1 or

4.Update bootlist
bootlist -m normal hdisk1 hdisk0
bootlist -m service cd0 rmt0 hdisk0 hdisk1

(Actual operations Screen Shot)

0:root@sactgx0005:/home/root # bootlist -m normal -o
hdisk0
0:root@sactgx0005:/home/root # ipl_varyon -i

PVNAME BOOT DEVICE PVID VOLUME GROUP ID
hdisk0 YES 00031523be6762d10000000000000000 00031523
00004c00
hdisk1 NO 00031523d36a9c5e0000000000000000 00031523
00004c00
hdisk2 NO 00031523f3bacf050000000000000000 00031523
f3badb67
hdisk3 NO 00031523f3baf1de0000000000000000 00031523
f3badb67
hdisk4 NO 00031523f3bb5b570000000000000000 00031523
f3bb694c
hdisk5 NO 00031523f3bb82350000000000000000 00031523
f3bb694c
0:root@sactgx0005:/home/root # bosboot -ad hdisk1

bosboot: Boot image is 19527 512 byte blocks.
0:root@sactgx0005:/home/root # bootlist -m normal -o
hdisk0
0:root@sactgx0005:/home/root # ipl_varyon -i

PVNAME BOOT DEVICE PVID VOLUME GROUP ID
hdisk0 YES 00031523be6762d10000000000000000 00031523
00004c00
hdisk1 YES 00031523d36a9c5e0000000000000000 00031523
00004c00
hdisk2 NO 00031523f3bacf050000000000000000 00031523
f3badb67
hdisk3 NO 00031523f3baf1de0000000000000000 00031523
f3badb67
hdisk4 NO 00031523f3bb5b570000000000000000 00031523
f3bb694c
hdisk5 NO 00031523f3bb82350000000000000000 00031523
f3bb694c

Clone rootdisk:
===================================================================

Other Useful Commands
===================================================================
lspv -m hdisk1 ; To see avaliable physical partitions on a disk

bootinfo -b ; to see the current boot device

ipl_varyon -i ; to see which disks are bootable

alt_disk_inst -C hdisk4
alt_disk_inst -C -B -O hdisk4 ; doesn't update the bootlist

bootinfo -K ; see what kernel
bootinfo -y ; see if 64 bit kernel is available

#bosboot -a -L
#lockstat -a

AIX Sybase ASE Performance Tuning

2009-12-18T20:30:00.001-08:00

Thanks GOOGLE for making this page top 3 search result of "maxperm sybase ase".

Resolving Poor ASE Performance Due to AIX Default Virtual Memory Paging

ASE delivers its best performance when all of its memory resides in physical memory on the machine.
It ASE’s memory is paged out by the operating system’s virtual memory subsystem then ASE performance
can suffer dramatically.

Poor performance due to memory paging on AIX is often reported as problems with ASE’s checkpoint and
/ or housekeeper tasks. These tasks may be slow, appear hung, or encounter timeslice errors.
Transaction dumps may also be affected as ASE cannot properly execute a checkpoint.
Paging particularly affects these tasks because they often process long lists of memory.
If ASE’s memory has been paged out access to it may require a read from disk, resulting in
poor performance. Any other ASE task that must traverse long lists of memory pages may be similarly affected,
resulting in timeslice errors. Logical lock contention may also increase as tasks
which hold locks take longer to run.

While this is a potential problem on all platforms, certain characteristics of AIX make it more likely
to occur on that platform. The default configuration of the AIX kernel allows the file system cache
to consume up to 80% of the available physical memory pages. Depending on system demand this may result
in ASE memory pages being copied to the swap device. Sybase recommends tuning the AIX virtual memory
subsystem to avoid this scenario.

The AIX values minperm and maxperm loosely control the ratio of page frames used for files versus
those used for computational processes (such as ASE). Tuning these values requires determining
the physical memory load of processes running on the host machine. maxperm should then be set
so that file pages do not interfere with process pages.

For example, consider an AIX host that has 8 Gb of physical memory and hosts two ASE servers.
One ASE server is configured to use 3 Gb of max memory and the other is configured to use 2 Gb of max memory.
Other applications on the host require a total of 1 Gb of memory. Therefore the total memory
requirement is 6 Gb (3 Gb + 2 Gb + 1 Gb). As 75% of this host’s physical memory is needed
for applications (6 Gb out of 8 Gb), maxperm should be set no higher than 25%.

In addition to configuring maxperm, Sybase highly recommends setting strict_maxperm to 1. When strict_maxperm
is set to 0 (the default value), AIX may override the maxperm setting at its discretion. Setting strict_maxperm
to 1 informs AIX that this is a hard limit.

Kernel parameters:

Note: Value for tunable maxperm% must be greater than or equal to the value of tunable maxclient%

=======================
maxclient%

Purpose:
Specifies maximum percentage of RAM that can be used for caching client pages.
Similar to maxperm% but cannot be bigger than maxperm%.
Values:
o Default: 80
o Range: 1 to 100%.
o Type: Dynamic
Diagnosis:
If J2 file pages or NFS pages are causing working storage pages to get paged
out, maxclient can be reduced.
Tuning
Decrease the value of maxclient if paging out to paging space is occurring due
to too many J2 client pages or NFS client pages in memory. Increasing the
value can allow more J2 or NFS client pages to be in memory before page
replacement starts.

Refer To:
Miscellaneous I/O Tuning Parameters

========================

maxperm%

Purpose:
Specifies the point above which the page-stealing algorithm steals only file
pages.
Values:
o Default: total number of memory frames * 0.8
o Range: 1 to 100
o Type: Dynamic
Diagnosis:
Monitor disk I/O with iostat n.
Tuning
This value is expressed as a percentage of the total real-memory page frames
in the system. Reducing this value may reduce or eliminate page replacement of
working storage pages caused by high number of file page accesses. Increasing
this value may help NFS servers that are mostly read-only. For example, if
some files are known to be read repetitively, and I/O rates do not decrease
with time from startup, maxperm may be too low.
Refer To:
Tuning VMM Page Replacement with the vmtune Command

---------------------------------------------------------------------------------------
maxfree

Purpose:
Specifies the number of frames on the free list at which page-stealing is to
stop.
Values:
o Default: 128
o Range: 16 to 204800
o Type: Dynamic
Diagnosis:
Observe free-list-size changes with vmstat n.
Tuning
If vmstat n shows free-list size frequently driven below minfree by
application demands, increase maxfree to reduce calls to replenish the free
list. Generally, keep maxfree - minfree equal to or less than 100. Setting the
value too high causes page replacement to run for a longer period of time.
Value must be at least 8 greater than minfree

-------------------------------------------------------------------------------------

maxpin%

Purpose:
Specifies the maximum percentage of real memory that can be pinned.
Values:
o Default: 80 percent
o Range: 1 to 99
o Type: Dynamic
Diagnosis:
Cannot pin memory, although free memory is available.
Tuning
If this value is changed, the new value should ensure that at least 4 MB of
real memory will be left unpinned for use by the kernel. The maxpin values
must be greater than one and less than 100. Change this parameter only in
extreme situations, such as maximum-load benchmarking.

---------------------------------------------------------------------------------------

minfree

Purpose:
Specifies the minimum number of frames on the free list at which the VMM
starts to steal pages to replenish the free list.
Values:
o Default: maxfree - 8
o Range: 8 to 204800
o Type: Dynamic
Diagnosis:
vmstat n
Tuning
Page replacement occurs when the number of free frames reaches minfree. If
processes are being delayed by page stealing, increase minfree to improve
response time. The difference between minfree and maxfree should always be
equal to or greater than maxpgahead.

-----------------------------------------------------------------------------------------

strict_maxperm

Purpose:
If set to 1, the maxperm value will be a hard limit on how much of RAM can be
used as a persistent file cache.
Values:
o Default: 0 (off)
o Range: 0 or 1.
o Type: Dynamic
Diagnosis:
Excessive page outs to page space caused by too many file pages in RAM.
Tuning
Set to 1 in order to make the maxperm value a hard limit (use in conjunction
with the tuning of the maxperm parameter).
Refer To:
Placing a Hard Limit on Persistent File Cache with strict_maxperm

Hands-on examples:

#vmo –a ; display all

#vmo –D ; reset all to default value

594 vmo -p -o maxclient%=50 ; good for both current and reboot values
594 vmo -r -o maxclient%=50 ; good for reboot values
595 vmo -o maxclient%=50 ; maxperm% must be larger/equal to maxclient %
596 vmo -o maxperm%=50 ; good for now
597 vmo -r -o maxperm%=50 ; good for reboot
602 vmo -r -o strict_maxperm=1 ; good for reboot
603 vmo -o strict_maxperm=1 ; good for now
604 vmo –a

1. To list the current and reboot value, range, unit, type and dependencies of
all tunables parameters managed by the vmo command, type:
vmo -L

2. To turn on and reserve 16MB large pages on a POWER4 system, type:
vmo -r -o lgpg_regions=10 -o lgpg_size=16777216

This command will propose bosboot to the user, and warn that a reboot is
necessary before the change will be effective.
3. To display help on nokilluid, type:
vmo -h nokilluid

4. To turn on v_pinshm after the next reboot, type:
vmo -r -o v_pinshm=1

5. To permanently reset all vmo tunable parameters to default, type:
vmo -p -D

6. To list the reboot value for all virtual Memory Manager tuning parameters,
type:
vmo -r -a

7. To list (spreadsheet format) the current and reboot value, range, unit,
type and dependencies of all tunables parameters managed by the vmo
command, type:
vmo -x

AIX Kernel-VM-Tuning

See memory usage

svmon -G -i 1 2

svmon -P pid

Kernel Tuning in AIX 5L Version 5.2 Performance Tools Guide and Reference.

Performance Overview of the Virtual Memory Manager
(VMM) and Tuning VMM Page Replacement

vmo -a ; display
vmo -d ; set to default value
vmo -o ; display or set a tunable to new value

vmo -p ; When used in combination with -o, -d or -D, makes changes apply to both
current and reboot values, that is, turns on the updating of the
/etc/tunables/nextboot file in addition to the updating of the current value.
These combinations cannot be used on Reboot and Bosboot type parameters becasue
their current value can't be changed.

3:root@sactgx0020:/var/log/nmonlog # vmo -a
memory_frames = 4194304
pinnable_frames = 3947805
maxfree = 288
minfree = 256
minperm% = 20
minperm = 808874
maxperm% = 80
maxperm = 3235503
strict_maxperm = 0
maxpin% = 80
maxpin = 3355444
maxclient% = 80
lrubucket = 131072
defps = 1
nokilluid = 0
numpsblks = 6684672
npskill = 52224
npswarn = 208896
v_pinshm = 0
pta_balance_threshold = n/a
pagecoloring = n/a
framesets = 2
mempools = 1
lgpg_size = 0
lgpg_regions = 0
num_spec_dataseg = 0
spec_dataseg_int = 512
memory_affinity = 1
htabscale = n/a
force_relalias_lite = 0
relalias_percentage = 0
data_stagger_interval = 161
large_page_heap_size = 0
kernel_heap_psize = 4096
soft_min_lgpgs_vmpool = 0
vmm_fork_policy = 0
low_ps_handling = 1
mbuf_heap_psize = 4096
strict_maxclient = 1
cpu_scale_memp = 8
lru_poll_interval = 0
lru_file_repage = 1

1:root@sactgx0010:/home/root # vmo -L
NAME CUR DEF BOOT MIN MAX UNIT TYPE
DEPENDENCIES
--------------------------------------------------------------------------------
memory_frames 3840K 3840K 4KB pages S
--------------------------------------------------------------------------------
pinnable_frames 3644K 3644K 4KB pages S
--------------------------------------------------------------------------------
maxfree 128 128 128 16 200K 4KB pages D
minfree
memory_frames
--------------------------------------------------------------------------------
minfree 120 120 120 8 200K 4KB pages D
maxfree
memory_frames
--------------------------------------------------------------------------------
minperm% 20 20 20 1 100 % memory D
maxperm%
--------------------------------------------------------------------------------
minperm 757803 757803 S
--------------------------------------------------------------------------------
maxperm% 80 80 80 1 100 % memory D
minperm%
maxclient%
--------------------------------------------------------------------------------
maxperm 2960K 2960K S
--------------------------------------------------------------------------------
strict_maxperm 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
maxpin% 80 80 80 1 99 % memory D
pinnable_frames
memory_frames
--------------------------------------------------------------------------------
maxpin 3M 3M S
--------------------------------------------------------------------------------
maxclient% 80 80 80 1 100 % memory D
maxperm%
--------------------------------------------------------------------------------
lrubucket 128K 128K 128K 64K 4KB pages D
--------------------------------------------------------------------------------
defps 1 1 1 0 1 boolean D
--------------------------------------------------------------------------------
nokilluid 0 0 0 0 4G-1 uid D
--------------------------------------------------------------------------------
numpsblks 7808K 7808K 4KB pages S
--------------------------------------------------------------------------------
npskill 61K 61K 61K 1 7M-1 4KB pages D
--------------------------------------------------------------------------------
npswarn 244K 244K 244K 0 7M-1 4KB pages D
--------------------------------------------------------------------------------
v_pinshm 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
pta_balance_threshold n/a 50 50 0 99 % pta segment R
--------------------------------------------------------------------------------
pagecoloring n/a 0 0 0 1 boolean B
--------------------------------------------------------------------------------
framesets 2 2 2 1 10 B
--------------------------------------------------------------------------------
mempools 1 1 1 1 4 B
--------------------------------------------------------------------------------
lgpg_size 0 0 0 0 256M bytes B
lgpg_regions
--------------------------------------------------------------------------------
lgpg_regions 0 0 0 0 B
lgpg_size
--------------------------------------------------------------------------------
num_spec_dataseg 0 0 0 0 B
--------------------------------------------------------------------------------
spec_dataseg_int 512 512 512 0 B
--------------------------------------------------------------------------------
memory_affinity 1 1 1 0 1 boolean B
--------------------------------------------------------------------------------
htabscale n/a -1 -1 -4 0 B
--------------------------------------------------------------------------------
force_relalias_lite 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
relalias_percentage 0 0 0 0 32K-1 D
--------------------------------------------------------------------------------
data_stagger_interval 161 161 161 0 0 4KB pages D
lgpg_size
--------------------------------------------------------------------------------
large_page_heap_size 0 0 0 0 8E-1 bytes B
lgpg_size
--------------------------------------------------------------------------------
kernel_heap_psize 4K 4K 4K 4K 16M bytes B
lgpg_size
--------------------------------------------------------------------------------
soft_min_lgpgs_vmpool 0 0 0 0 90 % D
lgpg_size
--------------------------------------------------------------------------------
vmm_fork_policy 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
low_ps_handling 1 1 1 1 2 D
--------------------------------------------------------------------------------
mbuf_heap_psize 4K 4K 4K 4K 16M bytes B
--------------------------------------------------------------------------------
strict_maxclient 1 1 1 0 1 boolean D
--------------------------------------------------------------------------------
cpu_scale_memp 8 8 8 1 64 B
--------------------------------------------------------------------------------
lru_poll_interval 0 0 0 0 60000 milliseconds D
--------------------------------------------------------------------------------
lru_file_repage 1 1 1 0 1 boolean D
--------------------------------------------------------------------------------

n/a means parameter not supported by the current platform or kernel

Parameter types:
S = Static: cannot be changed
D = Dynamic: can be freely changed
B = Bosboot: can only be changed using bosboot and reboot
R = Reboot: can only be changed during reboot
C = Connect: changes are only effective for future socket connections
M = Mount: changes are only effective for future mountings
I = Incremental: can only be incremented

Value conventions:
K = Kilo: 2^10 G = Giga: 2^30 P = Peta: 2^50
M = Mega: 2^20 T = Tera: 2^40 E = Exa: 2^60

AIX Checkpoints

2009-12-18T20:18:00.000-08:00

A general 10-point checklist for new build AIX servers.
Actual check-point and values may vary. This is just a reference.

Quick Checkpoint:
1. Adjust paging size;
2. Configure NIC/etherchannel;
3. Update /etc/security/limits;
4. Check/configure the dump devices;
5. Create a shutdown wrapper;
6. Setup/modify sudo privilege;
7. Install additional packages (SAN, EMC, HDS, etc.)
8. Setup NIS client (if required)
9. Mirror rootvg and verify rootvg mirroring
10. Fine tune kernel memory settings (vmo)

1. Increase paging space size as appropriate:
# mkps –a –n –sXX rootvg

Update the system settings:
# chdev -l aio0 -a minservers='250' -a maxservers='500'
# chdev -l aio0 -a maxreqs=8192 –a autoconfig='available'
# chdev -l sys0 -a cpuguard=enable -a autorestart=true
# chdev -l sys0 -a iostat=true
# chdev -l sys0 -a maxuproc=500

2. Configure NIC adapter and create etherchannel:
Remove and recreate the interface used to build the OS:
# ifconfig enX down detach
# rmdev –dl enX
# cfgmgr -v

Check each NIC adapter's media speed:
# lsdev -Cc adapter | awk '{print $1}' | grep ent | while read i
do
echo "################## $i"
lsattr -El $i -a media_speed
done

Update each NIC adapter's media speed as necessary:
# chdev -l entX -a media_speed=100_Full_Duplex
(For Gigabit Ethernet, use auto_negotiation)

Create the etherchannel interface which should look like:
EtherChannel / Link Aggregation: ent8
Status: Available
Attributes:
adapter_names ent4 EtherChannel Adapters
alt_addr 0x000000000000 Alternate EtherChannel Address
auto_recovery yes Enable automatic recovery after failover
backup_adapter ent6 Adapter used when whole channel fails
hash_mode default Determines how outgoing adapter is chosen
mode standard EtherChannel mode of operation
netaddr 10.150.144.129 Address to ping
noloss_failover yes Enable lossless failover after ping failure
num_retries 2 Times to retry ping before failing
retry_time 5 Wait time (in seconds) between pings
use_alt_addr no Enable Alternate EtherChannel Address
use_jumbo_frame no Enable Gigabit Ethernet Jumbo Frames

Use the etherchannel to set up the server's public network connectivity

3. Update /etc/security/limits:
default:
fsize = -1
core = 2097151
cpu = -1
data = -1
rss = -1
stack = -1
nofiles = 2000

4. Configure the dump device:
# sysdumpdev –K
# sysdumpdev –C
# sysdumpdev -Pp /dev/sysdumpnull
# rmlv hd7
# mklv -y hd7 -a im -t sysdump rootvg 16 <== 4G based on 256 MB PP size # sysdumpdev -Pp /dev/hd7 5. Create the /etc/rc.shutdown script: ##################### Beginning of rc.shutdown ##################### if [ -r /tmp/allowshutdown ];then rm -f /tmp/allowshutdown ps -ef | grep "/opt/VRTSvcs/bin/had" | grep -v grep > /dev/null
if [ $? -eq 0 ];then
echo
echo "WARNING: VCS should be stopped before shutdown is run"
echo
exit 1
fi

echo "Running /etc/rc.shutdown, please wait...."
exec 1>/tmp/shutdown.log 2>&1
date

#
# Add the processes to be shut down in background here
#
# sleep 120

else
echo
banner `hostname`
echo
echo "WARNING: If you really want to shutdown this system"
echo " you must create the file /tmp/allowshutdown"
echo
exit 1
fi
######################## End of rc.shutdown ########################

# chmod 700 /etc/rc.shutdown

6. Setup sudo: edit /etc/sudoers for proper setting

7. Install EMC (optional)
# Install EMC Software and License Key
echo "PP_LPAR_KEY_FIX=1" >> /etc/environment

Install EMC ODM definitions:
# cd /emc_package_home
# installp -agXd. EMC

Install EMC PowerPath:
# installp -agXd. EMCpower

Add EMC PowerPath license key:
# emcpreg -add wxyz-wxyz-wxyz-wxyz-wxyz-wxyz

Install Solution Enabler:
# installp -agXd. EMCsymm

Config sendmail for security

Reboot:
# touch /tmp/allowshutdown
# shsutdown -rF

Add the appropriate netgroups to /etc/passwd

Create the /etc/no_shell file:

##################### Beginning of /etc/no_shell #####################
#!/bin/sh
/bin/cat 1>&2 <> /etc/passwd

8. Setup NIS client and Test NIS:

Add the following entry to /etc/environment:
# echo "YPBIND_MAXWAIT=10" >> /etc/environment

Create ypservers from list:

Change the domain name:
(For PROD/DR servers)
# /usr/sbin/chypdom -B prod.us.db.com
(For DEV/UAT servers)
# /usr/sbin/chypdom -B dev.us.db.com

This command updates /etc/rc.nfs:
if [ -x /usr/bin/domainname ]; then
/usr/bin/domainname
fi

Start ypbind:
# /usr/sbin/mkclient -B -S

This command updates /etc/rc.nfs:
if [ -x /usr/lib/netsvc/yp/ypbind ]; then
start ypbind /usr/lib/netsvc/yp/ypbind -ypsetme
/usr/sbin/ypset
fi

Verify that ypbind is running:
# ps -ef | grep ypbind
# ypwhich
# ypwhich -m
# ypcat passwd

Append the following line to /etc/group for NIS group access:
# echo "+:" >> /etc/group

Update host resolution order (if needed):
/etc/netsvc.conf --> hosts=local,bind - OR -
NSORDER=local,bind;export NSORDER
(Default is bind,nis,local)

9. Mirror rootvg
# mirrorvg –S rootvg
# bosboot -ad /dev/ipldevice
# bootlist -m normal hdisk0 hdisk1

10.Tune VMM (vmo, ioo)
vmo -a
vmo -o
For Sybase ASE servers on AIX 5.xL, to ensure the best performance on Sybase, set the following parameters:
maxperf% = 25% (default is 80%)
strict_maxperf = 1

For details on vmo and database performance issue, please refer to this blog article.

AIX Breaking Root Disk Mirrors for OS Upgrade

2009-12-18T18:51:00.000-08:00

Before performing any OS patching, it is common practice to break the root disk mirrors. One mirror is used to perform the patching and the second mirror is saved for fail back purposes. This note describes a procedure to break root disk for AIX OS upgrade. This procedure was tested successfully on AIX 5.3 TL7 but should work on all AIX 5.3+ technology levels.

Note:
1. Do a mksysb backup before patching to provide an alternative method of rebuilding the OS from the NIM server using mksysb copy.
2. The following example assumes that the rootvg contains mirrored copies of LVs on hdisk0 and hdisk1.

###
### Step 1: Save the system configuration and break the root mirrors
###
Capture and save outputs from the following commands:
# lspv
# lsvg
# lsvg -o
# lsvg -l `lsvg -o`
# lsvg -p `lsvg -o`
# bootlist -m normal -o

Export all non-rootvg volume groups
Extract the major and minor number of the disk the system was booted from:

# ls -al /dev/ipldevice
crw------- 2 root system 22, 9 Feb 10 11:15 /dev/ipldevice

Confirm the disk the system was booted from:
# ls -al /dev | grep "22, 9"
brw------- 1 root system 22, 9 Feb 11 15:50 hdisk0
crw------- 2 root system 22, 9 Feb 10 11:15 ipldevice
crw------- 2 root system 22, 9 Feb 10 11:15 rhdisk0

Check the current bootlist:
# bootlist -m normal -o
hdisk0 blv=hd5
hdisk1 blv=hd5

Remove LV mirrored copies from hdisk1:
# unmirrorvg rootvg hdisk1

Remove the boot block from hdisk1:
# chpv -c hdisk1

Remove hdisk1 from rootvg:
# reducevg rootvg hdisk1

If you see the following error:
0516-016 ldeletepv: Cannot delete physical volume with allocated
partitions. Use either migratepv to move the partitions or
reducevg with the -d option to delete the partitions.
0516-884 reducevg: Unable to remove physical volume hdisk1
some LVs are present only in hdisk1 and have not been mirrored.

Migrate them to hdisk0 and then remove hdisk1 from rootvg:
# migratepv hdisk1 hdisk0
# reducevg rootvg hdisk1

Confirm that hdisk1 has been removed from rootvg:
# lsvg -p rootvg
rootvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
hdisk0 active 546 250 107..87..22..00..34

###
### Step 2: Create the alternate boot disk
###
Create the alternate boot disk on hdisk1:
# alt_disk_copy -d hdisk1
Check the volume groups:
# lsvg
rootvg
altinst_rootvg <= altinst_rootvg contains rootvg on hdisk1

# lsvg -o
rootvg <= altinst_rootvg is varied off

The boot list should now point to the alternate disk only:
# bootlist -m normal -o
hdisk1 blv=hd5

###
### Step 3: Boot from the alternate boot disk (hdisk1)
###
Reboot:
# shutdown -rF
Verify that the system was booted from the alternate boot disk (hdisk1):
# ls -al /dev/ipldevice
crw------- 2 root system 22, 1 Feb 20 15:50 /dev/ipldevice

# ls -al /dev | grep "22, 1"
brw------- 1 root system 22, 1 Feb 20 16:07 hdisk1
crw------- 2 root system 22, 1 Feb 20 15:50 ipldevice
crw------- 2 root system 22, 1 Feb 20 15:50 rhdisk1

Check the volume groups:
# lsvg
rootvg
old_rootvg <= old_rootvg contains rootvg on hdisk0

# lsvg -o
rootvg <= old_rootvg is varied off
Change the boot list to point to the original boot disk only:
# bootlist -m normal hdisk0
# bootlist -m normal -o
hdisk0 blv=hd5

###
### Step 4: Boot from the original boot disk (hdisk0)
###
Reboot:
# shutdown -rF
Confirm that the system was booted from the original boot disk (hdisk0):
# ls -al /dev/ipldevice
crw------- 2 root system 22, 9 Feb 10 11:15 /dev/ipldevice
# ls -al /dev | grep "22, 9"
brw------- 1 root system 22, 9 Feb 11 15:50 hdisk0
crw------- 2 root system 22, 9 Feb 10 11:15 ipldevice
crw------- 2 root system 22, 9 Feb 10 11:15 rhdisk0

Proceed with the OS upgrade

###
### Step 5: If the upgrade is successful
###
Remove the alternate boot disk:
# exportvg alinst_rootvg

Remove the boot block from hdisk1:
# chpv -c hdisk1

Add hdisk1 to rootvg:
# extendvg -f rootvg hdisk1

Reestablish root mirroring:
# mirrorvg -S rootvg

Update the boot list:
# bootlist -m normal hdisk0 hdisk1

Rebuild the boot volume:
# bosboot -ad /dev/ipldevice
Import non-rootvg volume groups

###
### Step 6:
### If the upgrade is unsuccessful, boot from the alternate boot disk
### to failback to the OS before the upgrade
###
Update the boot list to point to the alternate boot disk:
# bootlist -m normal hdisk1
# bootlist -m normal -o
hdisk1 blv=hd5

Reboot:
# shutdown -rF
Check the volume groups:
# lsvg
rootvg
old_rootvg <= old_rootvg contains rootvg on hdisk0

# lsvg -o
rootvg <= old_rootvg is varied off

Remove old_rootvg permanently:
# alt_rootvg_op -X old_rootvg - OR -
# alt_disk_install -X old_rootvg

Add hdisk0 to rootvg and reestablish root mirroring
Import non-rootvg volume groups

AIX Boot Related Administration Notes (1)

2009-12-18T18:46:00.000-08:00

AIX Boot Related Administration Notes

RAMFS:

RAMFS files are accessed using a prefix of /../, eg. rc.boot uses /../usr/sbin/fsck to find fsck.

Mount root out onto /mnt, or you will be unable to access the RAMFS files.

;To see the last boot device

#bootinfo -b

;To see all boot process related messages

#alog -t boot -o

Multi-user initialization completed

ot: executing "copycore"

Changed: tty

Changed: ipldevice

Changed: IPL_rootvg

mergedev replaced 3 files in the hardfile /dev directory

rc.boot: run time mount of /var

rc.boot: run time mount of /tmp

rc.boot: checking free space in /tmp

rc.boot: executing "strload"

rc.boot: allow 64-bit apps

rc.boot: executing "cfglivedump -c"

rc.boot: executing "cfgmgr"

cfgmgr is running in phase 2

----------------

Time: 0 LEDS: 0x538

Invoking top level program -- "/etc/methods/cfgprobe -c /etc/drivers/coreprobe.e

xt"

Time: 0 LEDS: 0x539

Return code = 0

*** no stdout ****

*** no stderr ****

----------------

stdin

;Each fs has two superblocks at 1 and 31.

;To copy the superblock from block 31 to block 1

# dd count=1 bs=4k skip=31 seek=1 if=/dev/hd4 of=/dev/hd4

;To unlock rootvg

#chvg -u rootvg

E1F1 is the led that indicates that a system is looking for a boot devices list in the NVRAM. Please follow the procedurebelow in order to start your system in maintenance mode and then recreate the boot list:

Booting in Service Mode

This document describes how to boot the system into Service mode (also known as Maintenance mode) to install the machine,restore an operating system backup, or perform maintenance on the rootvg volume group.

The information in this document applies to AIX Versions 3.x, 4.x and 5.x.

Booting PCI-based systems into Service mode

PCI machine-specific information

Accessing rootvg and mounting file systems

Related documentation

Booting PCI-based systems into Service mode

When booting a PowerPC into Service mode, cd0 or rmt0 must be before the hdisk in the bootlist. If not, change the bootlist

at boot time. On some models, you can set the machine to use a default bootlist that includes both cd0 and rmt0. If a bootable CD or tape is in the CD-ROM or tape drive, the machine will boot from this device.

For most of the newer PCI-based models, selecting the default bootlist, with a bootable tape or CD loaded in the machine, causes the system to automatically boot from that device. Generally, the next menu on the screen asks the administrator

to define the system console.

For all machines discussed here, if you are using a graphical terminal, you will use a function key such as F5. If you are using an ASCII terminal, use an equivalent number key such as 5. Use the numbers across the top of the keyboard, not thenumbers on the numeric keypad. On ASCII terminals, the icons may not be displayed on the screen; the number can be pressedbetween the second and third beeps, the second beep being a series of three clicks.

PCI machine-specific information

The following systems all use the F5 or 5 key to read from the default boot list, which is written into the system firmware:

MODEL 7017 7024 7025 7026 7043 7137

-------------- ------- ------- ------- ------- ------- -------

TYPE S70 E20 F30 H10 43P-140 F3L

S7A E30 F40 H50 43P-150

S80 F50 H70 43P-240

B80 43P-260

On these machines, use 5 (on the keyboard, not the keypad) if you are using an ASCII terminal. On a locally attached graphics console, use the F5 function key. The F5 or 5 key must be pressed just after the keyboard icon or message is displayed on the console. If you have either a 7026-M80, 7026-H80 or a 7025-F80, then the 5 key will be the default whether you have an ascii or graphics console.

The following systems use the F1 key to enter System Management Services mode (SMS):

MODEL 6040 7042 7247 7249

------- ------- ------- ------- -------

TYPE 620 850 82x 860

You should be in an Easy-Setup menu. Select the Start Up menu. Clear the current bootlist settings and then select the CD-ROM for choice 1 and hdd (the hard disk) for choice 2. Select OK. Insert the CD-ROM and select the EXIT icon. The machine should now boot from the CD-ROM.

The following systems use the F2 key to enter SMS:

MODEL 6015 6050 6070 7020 7248

------- ------- ------- ------- ------- -------

TYPE 440 830 850 40P 43P

Select Select Boot Device from the initial menu on the screen, and then select Restore Default Settings from the list. Press the Esc key to exit all the menus, and then reboot the machine. The system should boot from your bootable media.

For information on accessing the rootvg volume group, see the next section in this document.

Accessing rootvg and mounting file systems For AIX Versions 4 and 5, choose Start Maintenance Mode for System Recovery , option 3. The next screen will be called Maintenance; select option 1, Access a Root Volume Group. At the next screen, type 0 to continue, and select the appropriate volume group by typing the number next to it. A screen like the following will display.

Example:

Access a Root Volume Group

Type the number for a volume group to display the logical volume information and press Enter.

1) Volume Group 0073656f2608e46a contains these disks:

hdisk0 2063 04-C0-00-4,0

Once a volume group has been selected, information will be displayed about that volume group.

Example:

Volume Group Information

------------------------------------------------------------------------------

Volume Group ID 0073656f2608e46a includes the following logical volumes:

hd6 hd5 hd8 hd4 hd2 hd9var

hd3 hd1

------------------------------------------------------------------------------

Type the number of your choice and press Enter.

1) Access this Volume Group and start a shell

2) Access this Volume Group and start a shell before mounting filesystems

99) Previous Menu

If the logical volumes listed do not include logical volumes like hd4, hd2, hd3, and so on, you may have selected the wrong volume group. Press 99 to back up one screen and select again.

Now you may select one of two options: Access this volume group and start a shell , option 1, or Access this volume group and start a shell before mounting file systems , option 2. Option 2 allows you to perform file system maintenance on /, /usr, /tmp, and /var before mounting them.

NOTE: If you intend to use SMIT or vi, set your terminal type in preparation for editing the file. xxx stands for a terminal

type such as lft, ibm3151, or vt100.

TERM=

export TERM

Errors from these steps may indicate failed or corrupt disks in rootvg. These problems should be corrected. For additional assistance, contact your vendor, your local branch office, or your AIX support center.

Related documentation

For more in-depth coverage of this subject, the following IBM publication is recommended:

AIX Version 4.3 System Management Guide: Operating System and Devices

AIX Version 5.1 System Management Guide: Operating System and Devices

IBM documentation can also be accessed online through the following URL:

http://www.rs6000.ibm.com/resource/aix_resource/Pubs/index.html

Similar documents can be accessed through the following URL:

http://techsupport.services.ibm.com/server/support?view=pSeries

Once you have a valid prompt, you can recreate the boot device list (it's supposed you already know the hdisk number from which AIX starts, let's supposed it's hdisk0

bootlist -m normal hdisk0

If ths action does not work, repeat the procedure and issue

bosboot -ad /dev/ipldevice

Just before the bootlist command

Accessing Database with Perl DBI

2009-12-16T00:21:00.000-08:00

The best book to read for this subject is "Programming the Perl DBI".

#1. Install Perl DBI::DBD module
(You may need to remove any installed older versions of perl-DBI-mysql package)

Several ways to install Perl modules.
1. Download source and compile;
2. Use perl -MCPAN -e 'install DBD::mysql';
3. Use 'cpan';

For more details about installing Perl modules, please visit here.

I pick 3, use cpan.

cpan DBI
cpan DBD::mysql

(sample run of cpan File::HomeDir)
[root@ipc4 ~]# cpan File::HomeDir
CPAN: Storable loaded ok
Going to read /home/shan/perl/.cpan/Metadata
Database was generated on Tue, 15 Dec 2009 22:06:58 GMT
Running install for module File::HomeDir
Running make for A/AD/ADAMK/File-HomeDir-0.88.tar.gz
CPAN: LWP::UserAgent loaded ok
Fetching with LWP:
ftp://cpan-du.viaverio.com/pub/CPAN/authors/id/A/AD/ADAMK/File-HomeDir-0.88.tar.gz
CPAN: Digest::MD5 loaded ok
Fetching with LWP:
ftp://cpan-du.viaverio.com/pub/CPAN/authors/id/A/AD/ADAMK/CHECKSUMS
Checksum for /home/shan/perl/.cpan/sources/authors/id/A/AD/ADAMK/File-HomeDir-0.88.tar.gz ok
Scanning cache /home/shan/perl/.cpan/build for sizes
File-HomeDir-0.88/
...
CPAN.pm: Going to build A/AD/ADAMK/File-HomeDir-0.88.tar.gz

Checking if your kit is complete...
Looks good
...
Appending installation info to /usr/lib/perl5/5.8.5/i386-linux-thread-multi/perllocal.pod
/usr/bin/make install -- OK

#2. Use instmodsh utility to manage installed modules
(if instmodsh doesn't show all installed modules, use perldoc perllocal or check perllocal file directly to see the installed modules)

root@drbl-01 DBD-mysql-4.010]# instmodsh
Available commands are:
l – List all installed modules
m – Select a module
q – Quit the program
cmd? l
Installed modules are:
CGI
Cwd
DBD::mysql
DBI
ExtUtils::MakeMaker
FCGI
File::HomeDir
HTML::Parser
IO
Perl
Test::Simple
mod_perl2

cmd? m DBI
Available commands are:
f [all|prog|doc] - List installed files of a given type
d [all|prog|doc] - List the directories used by a module
v - Validate the .packlist - check for missing files
t - Create a tar archive of the module
q - Quit the module

DBI cmd?
Available commands are:
f [all|prog|doc] - List installed files of a given type
d [all|prog|doc] - List the directories used by a module
v - Validate the .packlist - check for missing files
t - Create a tar archive of the module
q - Quit the module
DBI cmd? d
all directories in DBI are:
/usr/bin
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/Bundle
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBD
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBD/Gofer/Policy
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBD/Gofer/Transport
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBI
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBI/Const
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBI/Const/GetInfo
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBI/DBD
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBI/Gofer
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBI/Gofer/Serializer
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBI/Gofer/Transport
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBI/ProfileDumper
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBI/SQL
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBI/Util
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/Win32
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/auto/DBI
/usr/share/man/man1
/usr/share/man/man3

You can us 't' command to create a tarball to install modules on those same arch servers that don't have access to the Internet directly (e.g. behind firewall).

#3. Grant user privilege to access database
[root@ipc4 perl]# mysql -u root -p
mysql > grant all privileges on database-name.* to 'user'@'localhost' identified by 'password';

#4. Create a testing database with two tables

[shan@ipc4 ~]$ mysql -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 94
Server version: 5.1.39-community MySQL Community Server (GPL)

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> create database bestbuy;
Query OK, 1 row affected (0.00 sec)

mysql> use bestbuy;
Database changed

mysql> create table location(loc_id integer,
-> name varchar(50),
-> address varchar(60),
-> city varchar(20),
-> state char(2),
-> zipcode char(5));
Query OK, 0 rows affected (0.07 sec)

mysql> create table purchase
-> (item_id integer not null,
-> item_name varchar(30) not null,
-> purchase_date datetime,
-> loc_id integer,
-> unit_price decimal(8,2),
-> quantity decimal(6,2),
-> amount decimal(8,2),
-> comment varchar(100));
Query OK, 0 rows affected (0.06 sec)

mysql> insert into location(loc_id,name,city,state) values(1,'BestBuy','Pasadena','CA');
Query OK, 1 row affected (0.01 sec)

mysql> insert into location(loc_id,name,city,state) values(1,'BestBuy','Duarte','CA');
Query OK, 1 row affected (0.00 sec)

mysql> insert into location(loc_id,name,city,state) values(1,'BestBuy','Sierra Madre','CA');
Query OK, 1 row affected (0.00 sec)

mysql> describe purchase;
+---------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------------+--------------+------+-----+---------+-------+
| item_id | int(11) | NO | | NULL | |
| item_name | varchar(30) | NO | | NULL | |
| purchase_date | datetime | YES | | NULL | |
| loc_id | int(11) | YES | | NULL | |
| unit_price | decimal(8,2) | YES | | NULL | |
| quantity | decimal(6,2) | YES | | NULL | |
| amount | decimal(8,2) | YES | | NULL | |
| comment | varchar(100) | YES | | NULL | |
+---------------+--------------+------+-----+---------+-------+
8 rows in set (0.00 sec)

mysql> insert into purchase(item_id,item_name,purchase_date,loc_id,unit_price,quantity,amount,comment)
-> values(1,'Nitendo DJHero','2009-12-15 19:15:00',1,100,1,108,'Prepare for a DJ job at a LA club.');
Query OK, 1 row affected (0.00 sec)

mysql> insert into purchase(item_id,item_name,purchase_date,loc_id,unit_price,quantity,amount,comment) values(1,'Nitendo Wii','2009-12-15 19:15:00',1,199,1,218,'Prepare for a DJ job at a LA club.');
Query OK, 1 row affected (0.00 sec)

mysql> insert into purchase(item_id,item_name,purchase_date,loc_id,unit_price,quantity,amount,comment) values(1,'MacBookPro','2009-12-15 19:15:00',1,1699,1,1810,'Prepare for a DJ job at a LA club.');
Query OK, 1 row affected (0.00 sec)

mysql> insert into purchase(item_id,item_name,purchase_date,loc_id,unit_price,quantity,amount,comment) values(2,'Samsung LN55B650','2009-12-15 19:15:00',1,199,1,2150,'Prepare for a DJ job at a LA club.');
Query OK, 1 row affected (0.00 sec)

mysql> insert into purchase(item_id,item_name,purchase_date,loc_id,unit_price,quantity,amount,comment) values(3,'Samsung LN55iB8000','2009-12-15 19:15:00',1,1999,1,2150,'Prepare for a DJ job at a LA club.');
Query OK, 1 row affected (0.00 sec)

mysql> select * from location;
+--------+---------+---------+--------------+-------+---------+
| loc_id | name | address | city | state | zipcode |
+--------+---------+---------+--------------+-------+---------+
| 1 | BestBuy | NULL | Pasadena | CA | NULL |
| 1 | BestBuy | NULL | Duarte | CA | NULL |
| 1 | BestBuy | NULL | Sierra Madre | CA | NULL |
+--------+---------+---------+--------------+-------+---------+
3 rows in set (0.00 sec)

mysql> select * from purchase;
+---------+--------------------+---------------------+--------+------------+----------+---------+------------------------------------+
| item_id | item_name | purchase_date | loc_id | unit_price | quantity | amount | comment |
+---------+--------------------+---------------------+--------+------------+----------+---------+------------------------------------+
| 1 | Nitendo DJHero | 2009-12-15 19:15:00 | 1 | 100.00 | 1.00 | 108.00 | Prepare for a DJ job at a LA club. |
| 1 | Nitendo Wii | 2009-12-15 19:15:00 | 1 | 199.00 | 1.00 | 218.00 | Prepare for a DJ job at a LA club. |
| 1 | MacBookPro | 2009-12-15 19:15:00 | 1 | 1699.00 | 1.00 | 1810.00 | Prepare for a DJ job at a LA club. |
| 2 | Samsung LN55B650 | 2009-12-15 19:15:00 | 1 | 199.00 | 1.00 | 2150.00 | Prepare for a DJ job at a LA club. |
| 3 | Samsung LN55iB8000 | 2009-12-15 19:15:00 | 1 | 1999.00 | 1.00 | 2150.00 | Prepare for a DJ job at a LA club. |
+---------+--------------------+---------------------+--------+------------+----------+---------+------------------------------------+
5 rows in set (0.00 sec)

mysql> select item_id,item_name,purchase_date,loc_id,amount from purchase;
+---------+--------------------+---------------------+--------+---------+
| item_id | item_name | purchase_date | loc_id | amount |
+---------+--------------------+---------------------+--------+---------+
| 1 | Nitendo DJHero | 2009-12-15 19:15:00 | 1 | 108.00 |
| 1 | Nitendo Wii | 2009-12-15 19:15:00 | 1 | 218.00 |
| 1 | MacBookPro | 2009-12-15 19:15:00 | 1 | 1810.00 |
| 2 | Samsung LN55B650 | 2009-12-15 19:15:00 | 1 | 2150.00 |
| 3 | Samsung LN55iB8000 | 2009-12-15 19:15:00 | 1 | 2150.00 |
+---------+--------------------+---------------------+--------+---------+
5 rows in set (0.00 sec)

#5. Use Perl's DBI::mysql to modify and query the database
1 #!/usr/bin/perl -w
2
3 use DBI;
4
5 ## mysql user database name
6 $db ="bestbuy";
7
8 ## mysql database user name
9 $user = "shan";
10
11 ## mysql database password
12 $pass = "newyork1";
13
14 ## user hostname : This should be "localhost" but it can be diffrent too
15 $host="localhost";
16
17 # Get database handle
18 $dbh = DBI->connect("DBI:mysql:$db:$host", $user, $pass, {RaiseError=>1});
19
20 # 1. Use $dbh->do method to perform sql modification type operations
21
22 $results = $dbh->do(q{insert into purchase(item_id,item_name,purchase_date,loc_id,unit_price,quantity,amount,comment)
23 values(1,'SONY Laptop','2009-11-27 5:30:00',1,359,1,375,'Black Friday Bloody Reward')});
24 die "Unable to perform insert:$DBI::errstr\n" unless (defined $results);
25
26
27
28 # 2. Use $dbh->prepare/$dhb->execute methods to perform query type operations
29
30 $sth = $dbh->prepare(q{select * from purchase})
31
32 or die "Unable to prep our query:".$dbh->errstr."\n";
33
34 # sth is a statement handle retured by method prepare , DBI::st=HASH(0x8f7fbc4)->fetchrow_array()
35 my $rv = $sth->execute
36 or die "Unable to execute our query:".$dbh-errstr."\n";
37
38
39 # Output 2
40 print "The output of prepare-execute method sQL operation:\n";
41
42 while ( @row2 = $sth->fetchrow_array () ) {
43 my $temp2 = join "\t", @row2;
44 print " $temp2 \n";
45 }

Execution :
The above script insert record "SONY Laptop" and shows all records in purchase database.

[shan@ipc4 perl]$ perl bestbuy.pl
The output of prepare-execute method sQL operation:
1 Nitendo DJHero 2009-12-15 19:15:00 1 100.00 1.00 108.00 Prepare for a DJ job at a LA club.
1 Nitendo Wii 2009-12-15 19:15:00 1 199.00 1.00 218.00 Prepare for a DJ job at a LA club.
1 MacBookPro 2009-12-15 19:15:00 1 1699.00 1.00 1810.00 Prepare for a DJ job at a LA club.
2 Samsung LN55B650 2009-12-15 19:15:00 1 199.00 1.00 2150.00 Prepare for a DJ job at a LA club.
3 Samsung LN55iB8000 2009-12-15 19:15:00 1 1999.00 1.00 2150.00 Prepare for a DJ job at a LA club.
1 SONY Laptop 2009-11-27 05:30:00 1 359.00 1.00 375.00 Black Friday Bloody Reward

Useful Hint for DBAs:
Tired of re-typing a long SQL command after making a typo?
Add the following line in your home directory's .inputrc file, you will be able to use vi style editing mode inside mysql shell.
e.g. ESC k to recall the last command you typed, or ESC j to travel down the command line history .. cheers!

set editing-mode vi

#6.Troubleshooting:

#a. My initial (failed) installation of DBD::mysql
#cpan DBD::mysql

Cannot find the file 'mysql_config'! Your execution PATH doesn't seem
not contain the path to mysql_config. Resorting to guessed values!
Can't exec "mysql_config": No such file or directory at Makefile.PL line 464.
Can't find mysql_config. Use --mysql_config option to specify where mysql_config is located
Can't exec "mysql_config": No such file or directory at Makefile.PL line 464.
Can't find mysql_config. Use --mysql_config option to specify where mysql_config is located
Can't exec "mysql_config": No such file or directory at Makefile.PL line 464.
Can't find mysql_config. Use --mysql_config option to specify where mysql_config is located

#b. The root cause is it needs a file mysql_config which comes with MySQL-devel-community-5.x.x-0.rhel4.

Download it,

[shan@ipc4 rpm]$wget http://opensource.become.com/mysql/Downloads/MySQL-5.4/MySQL-devel-community-5.4.3-0.rhel4.i386.rpm
Or use curl,
[shan@ipc4 rpm]$curl http://opensource.become.com/mysql/Downloads/MySQL-5.4/MySQL-devel-community-5.4.3-0.rhel4.i386.rpm -o MySQL-devel-community-5.4.3-0.rhel4.i386.rpm

Install it:
[root@ipc4 rpm]# rpm -ivh MySQL-devel-community-5.4.3-0.rhel4.i386.rpm

Note, make sure you download the correct dev package to match your MySQL server.

#c. Reinstall DBD::mysql
[root@ipc4 perl]# cpan DBD::mysql
CPAN: Storable loaded ok
Going to read /home/shan/perl/.cpan/Metadata
Database was generated on Tue, 15 Dec 2009 22:06:58 GMT
Running install for module DBD::mysql
Running make for C/CA/CAPTTOFU/DBD-mysql-4.013.tar.gz
CPAN: Digest::MD5 loaded ok
Checksum for /home/shan/perl/.cpan/sources/authors/id/C/CA/CAPTTOFU/DBD-mysql-4.013.tar.gz ok
Scanning cache /home/shan/perl/.cpan/build for sizes
DBD-mysql-4.013/
cflags (mysql_config) = -I/usr/include/mysql -g -pipe -m32 -DUNIV_LINUX
embedded (mysql_config) =
libs (mysql_config) = -rdynamic -L/usr/lib/mysql -lmysqlclient -lz -lcrypt -lnsl -lm -lmygcc
mysql_config (guessed ) = mysql_config
nocatchstderr (default ) = 0
nofoundrows (default ) = 0
ssl (guessed ) = 0
testdb (default ) = test
testhost (default ) =
testpassword (default ) =
testsocket (default ) =
testuser (guessed ) = root

….

You may get errors like below, that’s non-critical, it merely complains the DBD::mysql test can’t access the ‘test’ database with root account and password “NO”. You can always manually force the installation by entering the working directory ~.CPAN/build/module_name/ and type “make install”.

nt.t line 15
t/80procs...................skipped
all skipped: ERROR: Access denied for user 'root'@'localhost' (using password: NO). Can't continue test
t/85init_command............skipped
all skipped: ERROR: Access denied for user 'root'@'localhost' (using password: NO). Can't continue test
Failed Test Stat Wstat Total Fail Failed List of Failed
-------------------------------------------------------------------------------
t/00base.t 2 512 6 12 200.00% 1-6
35 tests skipped.
Failed 1/36 test scripts, 97.22% okay. 6/6 subtests failed, 0.00% okay.
make: *** [test_dynamic] Error 255
/usr/bin/make test -- NOT OK
Running make install
make test had returned bad status, won't install without force

Manually install:

[root@ipc4 DBD-mysql-4.013]# make install
Installing /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/auto/DBD/mysql/mysql.so
Installing /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/auto/DBD/mysql/mysql.bs
Files found in blib/arch: installing files in blib/lib into architecture dependent library tree
Installing /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBD/mysql.pm
Installing /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBD/mysql/GetInfo.pm
Installing /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBD/mysql/INSTALL.pod
Installing /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/Bundle/DBD/mysql.pm
Installing /usr/share/man/man3/DBD::mysql::INSTALL.3pm
Installing /usr/share/man/man3/Bundle::DBD::mysql.3pm
Installing /usr/share/man/man3/DBD::mysql.3pm
Writing /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/auto/DBD/mysql/.packlist
Appending installation info to /usr/lib/perl5/5.8.5/i386-linux-thread-multi/perllocal.pod

#d. Verify the installation:
[root@ipc4 ~]# instmodsh
Available commands are:
l - List all installed modules
m - Select a module
q - Quit the program
cmd? l
Installed modules are:
DBD::mysql
DBI
Perl
mod_perl2
cmd? m DBD::mysql
Available commands are:
f [all|prog|doc] - List installed files of a given type
d [all|prog|doc] - List the directories used by a module
v - Validate the .packlist - check for missing files
t - Create a tar archive of the module
q - Quit the module
DBD::mysql cmd? d
all directories in DBD::mysql are:
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/Bundle/DBD
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBD
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBD/mysql
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/auto/DBD/mysql
/usr/share/man/man3
DBD::mysql cmd? v
DBD::mysql has no missing files
DBD::mysql cmd? f
all files in DBD::mysql are:
/usr/share/man/man3/DBD::mysql::INSTALL.3pm
/usr/share/man/man3/DBD::mysql.3pm
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBD/mysql/INSTALL.pod
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBD/mysql.pm
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/auto/DBD/mysql/mysql.bs
/usr/share/man/man3/Bundle::DBD::mysql.3pm
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBD/mysql/GetInfo.pm
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/auto/DBD/mysql/mysql.so
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/Bundle/DBD/mysql.pm
DBD::mysql cmd? q
cmd? q
[root@ipc4 ~]#

#e. Clean up:
If your server is installed RHEL 4.0 and it already has MySQL, Perl installed, you may need to do some cleaning work to make sure you are using the correct versions of the package for each components. The sooner you do the cleaning job, the better. Once you installed/configured Perl, MySQL, Python, PHP, etc, etc, you find out you need to upgrade a package that’s required by 100 packages, you will feel the pain. That’s why platform engineering is very important for open source technology.

On my system, I have RHEL 4.0, which installed very early version of mysqlclient and perl-DBD-mysql. In fact, the old version perl-DBD-mysql-2.9xx is not compatible with MySQL server 5.x. I need to remove all those obsolete packages without damaging the system.

remove mysqlclient10-3.23 and its dependencies:

[root@ipc4 rpm]# rpm -qa | grep mysql
mysqlclient10-3.23.58-4.RHEL4.1
[root@ipc4 rpm]# rpm -ql $(rpm -qa | grep mysql)
/etc/ld.so.conf.d/mysqlclient10-i386.conf
/usr/lib/mysql/libmysqlclient.so.10
/usr/lib/mysql/libmysqlclient.so.10.0.0
/usr/lib/mysql/libmysqlclient_r.so.10
/usr/lib/mysql/libmysqlclient_r.so.10.0.0

[root@ipc4 rpm]# rpm -e $(rpm -qa | grep mysql)
error: Failed dependencies:
libmysqlclient.so.10 is needed by (installed) perl-DBD-MySQL-2.9004-3.1.i386
libmysqlclient.so.10 is needed by (installed) MySQL-python-1.0.0-1.RHEL4.1.i386
libmysqlclient.so.10 is needed by (installed) MyODBC-2.50.39-21.RHEL4.1.i386
mysqlclient10 is needed by (installed) MySQL-python-1.0.0-1.RHEL4.1.i386

[root@ipc4 rpm]# rpm -e MySQL-python-1.0.0-1.RHEL4.1.i386
[root@ipc4 rpm]# rpm -e MyODBC-2.50.39-21.RHEL4.1.i386
[root@ipc4 rpm]# rpm -e MySQL-python-1.0.0-1.RHEL4.1.i386

also, remove the old perl-DBD-MySQL
[root@ipc4 rpm]# rpm -e perl-DBD-MySQL-2.9004-3.1.i386

Auto fetch webpages/sign-in websites with curl and perl

2009-12-13T14:31:00.000-08:00

#1 - Use curl to sign-in Ebay:
a. create a cookie jar from ebay main page
curl -s -L -c cookies.txt -b cookies.txt -e ';auto' -o ebay.html 'http://www.ebay.com/'

b. download sign-in page
curl -s -L -c cookies.txt -b cookies.txt -e ';auto' -o ebay-signin.html 'http://signin.ebay.com/ws/eBayISAPI.dll?SignIn'

c. login with your ebay account and password
curl -s -L -c cookies.txt -b cookies.txt -e ';auto' -o ebay-pass.html \
-d MfcISAPICommand=SignInWelcome -d siteid=0 -d co_partnerId=2 \
-d UsingSSL=1 -d ru= -d pp= -d pa1= -d pa2= -d pa3= -d i1=-1 \
-d pageType=-1 -d rtmData= -d userid=your-ebay-account -d pass=passwd \
'https://signin.ebay.com/ws/eBayISAPI.dll?SignIn&UsingSSL=1&pUserId=&co_partnerId=2&siteid=0'

d. save your 'my-ebay' page
curl -s -L -c cookies.txt -b cookies.txt -e ';auto' -o myebay.html 'http://my.ebay.com/ws/eBayISAPI.dll?MyEbay'

Now, use a browser to open local file myebay.html, you see your MyEbay page if you entered your password correctly.

#2 - Use curl to download a range of pages from a website.
( create a directory "tmp" in your home directory to hold the download html pages. )
# this will download 21 files ch1.htm to ch21.htm and save them as ./tmp/ch-1.html ... ./tmp/ch-21.html
curl http://docs.rinet.ru/P7/ch[1-21].htm -o ./tmp/ch-#1.html

Simple Perl script to fetch page

#!/usr/bin/perl
use LWP::UserAgent;
use HTTP::Request::Common qw(POST);
$UA = LWP::UserAgent->new();
$req = HTTP::Request->new( GET => "http://www.ebay.com/" );
$resp = $UA->request($req);
# check for error. Print page if it's OK
if ( ( $resp->code() >= 200 ) && ( $resp->code() < 400 ) ) {
print $resp->decoded_content;
} else {
print "Error: " . $resp->status_line . "\n";
}

(to be continued...)

How much privacy we have?

2009-12-13T13:39:00.000-08:00

Picture of one of my blog visitors' neighborhood ... Yes, I just get your info if you're viewing this page now. :-)
(The image is processed to protect privacy)

Picture of an oversea visitor's neighborhood ...

Picture of a Jersey visitor's neighborhood ...

My Desktop (click image for full view)

2009-12-07T20:21:00.000-08:00

Besides Mintty's improved function over Putty, the cool transparent mode allows users to read files in the background without switching the terminal back and forth. It's very helpful when you read a pdf file while coding in vim!

My laptop : cygwin+mintty and vim/vimshell.( At this moment, vimshell can only be patched into vim 7.0. You need to recompile vim if you have 7.x)

My Desktop:

Interesting Tasks for Shell (Scripting)

2009-12-07T00:44:00.000-08:00

Here are my responses to a few help-requests at unix.ittoolbox.com:

1. The original request:
"hi
i have string like below
aabccdee
how to extract just the unique values from above..
i want my output to be
abcde
"
My solution:
[shan@ipc4 ~]$ sed -n 's@$[a-zA-Z]$\1*@\1@pg' < input_file > txt.output

or (a smart guy's tr solution.)
echo "aabccdee" | tr -s [[:alpha:]]

2. The original request:
i have a around ~15k csv files created in the month of nov,
each file has around 100 fields, i have to get the unique of 10th field.

MY solution:
find . -name "*in" -exec awk -F" " '{print $3}' {} \; | tr -sc 0-9A-Za-z '\012' | sort | uniq -c | sort -nr

sample output: (count : 10th field)
43 Makefile
38 YPPUSH
36 YPDIR
36 in
19 YPSRCDIR ...

3. The original request:
I have a file of the following format:
gi|*170079665*|ref|YP_001728985.1| gi|*15829256*|ref|NP_308029.1|
99.76 820 2 0 1 820 1 820 0.0 1645
gi|*170079666*|ref|YP_001728986.1| gi|*15829257*|ref|NP_308030.1|
99.35 310 2 0 1 310 1 310 4e-177 613
:
:
:
:

For each of the lines in this file I want to create a file having the two
numbers in bold in this format :
*170079665
**15829256*

My solution: (there are simple ones with perl/python... but, hey, shell is great!)
$ mkdir tmp; export j=1
$ for i in $(cut -d'|' -f2,6 <>> ./tmp/file.$j
echo '*'$(echo $i | cut -d"|" -f2) >> ./tmp/file.$j
let j=$j+1
done
input-file:
gi|*170079665*|ref|YP_001728985.1| gi|*15829256*|ref|NP_308029.1| 99.76 820 2 0 1 820 1 820 0.0 1645
gi|*170079800*|ref|YP_001728235.1| gi|*15829846*|ref|NP_308029.1| 99.76 820 2 0 1 820 1 820 0.0 1645

output:
$ ls tmp
file.1 file.2 file.3 file.4
$ cat ./tmp/file.1
*170079665*
**15829256*
$ cat ./tmp/file.2
*170079800*
**15829846* ...

cheers!