Showing posts with label Scripting ( Shell Perl Python). Show all posts
Showing posts with label Scripting ( Shell Perl Python). Show all posts

Always look on the bright side of Life!

The best chapter of book 'Learning Python'

"Once you finish the quiz, you’ve officially reached the end of this book. Now that you

know Python inside and out, your next step, should you choose to take it, is to explore
the libraries, techniques, and tools available in the application domains in which you
work. Because Python is so widely used, you’ll find ample resources for using it in
almost any application you can think of—from GUIs, the Web, and databases to numeric
programming, robotics, and system administration.

This is where Python starts to become truly fun, but this is also where this book’s story
ends, and others’ begin. For pointers on where to turn after this book, see the list of
recommended follow-up texts in the Preface. Good luck with your journey. And of course,
Always look on the bright side of Life!

-Mark Lutz, the author of "Learning Python"


bash login profiles for all scenarios

When invoked as bash (not sh) the start-up files that bash reads for a
login shell are, in order:

i) /etc/profile

and then the first of any of the following that exists:

ii) ~/.bash_profile OR
iii) ~/.bash_login OR
iv) ~/.profile


Non-login interactive bash shells (when changing or starting new shells
during a session) read only ~/.bashrc.

Non-login non-interactive bash shells (ie-when launched from inside a
script) check the startup file in $BASH_ENV, or $ENV if that doesn't
exist.

An easy way to make sure all three invocations of bash reference the same
start-up file is:

i) use .bashrc as your customized startup file

ii) create a .bash_profile containing only:

export BASH_ENV=~/.bashrc
if [ -f ~/.bashrc ]; then source ~/.bashrc; fi



This way, after reading /etc/profile a login shell will read the first
file found, ~/.bash_profile, which in turn reads ~/.bashrc

Non-login interactive shells will read ~/.bashrc as usual

Non-login non-interactive shells will read the ~/.bashrc assigned in
$BASH_ENV.


Reference 1: BASH's sourcing profiles order in execution

[sjing@prodpmcf151 ~]$ bash -cx 'env'
+ '[' -f /etc/bashrc ']'
+ source /etc/bashrc
++ '[' 17934 -gt 99 ']'
+++ id -gn
+++ id -un
++ '[' techdept = sjing ']'
++ umask 022
++ '[' '' ']'
++ shopt -q login_shell
++ for i in '/etc/profile.d/*.sh'
++ '[' -r /etc/profile.d/colorls.sh ']'
++ '[' '' ']'
++ . /etc/profile.d/colorls.sh
++ for i in '/etc/profile.d/*.sh'
++ '[' -r /etc/profile.d/glib2.sh ']'
++ '[' '' ']'
++ . /etc/profile.d/glib2.sh
++ for i in '/etc/profile.d/*.sh'
++ '[' -r /etc/profile.d/gnome-ssh-askpass.sh ']'
++ '[' '' ']'
++ . /etc/profile.d/gnome-ssh-askpass.sh
++ for i in '/etc/profile.d/*.sh'
++ '[' -r /etc/profile.d/krb5-workstation.sh ']'
++ '[' '' ']'
++ . /etc/profile.d/krb5-workstation.sh
++ for i in '/etc/profile.d/*.sh'
++ '[' -r /etc/profile.d/lang.sh ']'
++ '[' '' ']'
++ . /etc/profile.d/lang.sh
++ for i in '/etc/profile.d/*.sh'
++ '[' -r /etc/profile.d/less.sh ']'
++ '[' '' ']'
++ . /etc/profile.d/less.sh
++ for i in '/etc/profile.d/*.sh'
++ '[' -r /etc/profile.d/vim.sh ']'
++ '[' '' ']'
++ . /etc/profile.d/vim.sh
++ for i in '/etc/profile.d/*.sh'
++ '[' -r /etc/profile.d/which-2.sh ']'
++ '[' '' ']'
++ . /etc/profile.d/which-2.sh
++ unset i
++ unset pathmunge
+ '[' '!' 1 ']'
+ env

Reference 2: UNIX/LINUX UNICODE FAQ
Reference 3: Useful 'find' Page

Python: Check Python Syntax with -m Switch

Use Python -m switch and py_compile module to check Python syntax.

Python code with error at line 11

1 #!/usr/bin/python
2
3 import os
4 import time
5
6 prcfile="/home/sjing/prod/man/python/batch_cf/tmpfile"
7 while True:
8 fsize=int(os.path.getsize(prcfile))
9 print "file %s size is : %d" % (prcfile,fsize)
10 time.sleep(10)
11 if fsize == int(os.path.getsize(prcfile))
12 print "transaction is done. Exit\n"
13 break



-bash-3.2$ python -m py_compile test.py
File "test.py", line 11
if fsize == int(os.path.getsize(prcfile))
^
SyntaxError: invalid syntax




python -v will disply module initialization message:

-bash-3.2$ python -v -m py_compile test.py
# installing zipimport hook
import zipimport # builtin
# installed zipimport hook
# /appl/mtg/dev/vendor/python/lib/python2.4/site-packages/site.pyc matches /appl/mtg/dev/vendor/python/lib/python2.4/site-packages/site.py
import site # precompiled from /appl/mtg/dev/vendor/python/lib/python2.4/site-packages/site.pyc
import imp # builtin
# /usr/lib64/python2.4/os.pyc matches /usr/lib64/python2.4/os.py
import os # precompiled from /usr/lib64/python2.4/os.pyc
import posix # builtin
# /usr/lib64/python2.4/posixpath.pyc matches /usr/lib64/python2.4/posixpath.py
import posixpath # precompiled from /usr/lib64/python2.4/posixpath.pyc
# /usr/lib64/python2.4/stat.pyc matches /usr/lib64/python2.4/stat.py
import stat # precompiled from /usr/lib64/python2.4/stat.pyc
import errno # builtin
# /usr/lib64/python2.4/UserDict.pyc matches /usr/lib64/python2.4/UserDict.py
import UserDict # precompiled from /usr/lib64/python2.4/UserDict.pyc
# /usr/lib64/python2.4/copy_reg.pyc matches /usr/lib64/python2.4/copy_reg.py
import copy_reg # precompiled from /usr/lib64/python2.4/copy_reg.pyc
# /usr/lib64/python2.4/types.pyc matches /usr/lib64/python2.4/types.py
import types # precompiled from /usr/lib64/python2.4/types.pyc
# /usr/lib64/python2.4/site.pyc matches /usr/lib64/python2.4/site.py
import site # precompiled from /usr/lib64/python2.4/site.pyc
import japanese # directory /usr/lib64/python2.4/site-packages/japanese
# /usr/lib64/python2.4/site-packages/japanese/__init__.pyc matches /usr/lib64/python2.4/site-packages/japanese/__init__.py
import japanese # precompiled from /usr/lib64/python2.4/site-packages/japanese/__init__.pyc
import japanese.aliases # directory /usr/lib64/python2.4/site-packages/japanese/aliases
# /usr/lib64/python2.4/site-packages/japanese/aliases/__init__.pyc matches /usr/lib64/python2.4/site-packages/japanese/aliases/__init__.py
import japanese.aliases # precompiled from /usr/lib64/python2.4/site-packages/japanese/aliases/__init__.pyc
import encodings # directory /usr/lib64/python2.4/encodings
# /usr/lib64/python2.4/encodings/__init__.pyc matches /usr/lib64/python2.4/encodings/__init__.py
import encodings # precompiled from /usr/lib64/python2.4/encodings/__init__.pyc
# /usr/lib64/python2.4/codecs.pyc matches /usr/lib64/python2.4/codecs.py
import codecs # precompiled from /usr/lib64/python2.4/codecs.pyc
import _codecs # builtin
# /usr/lib64/python2.4/encodings/aliases.pyc matches /usr/lib64/python2.4/encodings/aliases.py
import encodings.aliases # precompiled from /usr/lib64/python2.4/encodings/aliases.pyc
# zipimport: found 10 names in /appl/mtg/dev/vendor/python/lib64/python2.4/site-packages/python_sybase-0.39-py2.4-linux-x86_64.egg
# zipimport: found 50 names in /appl/mtg/dev/vendor/python/lib64/python2.4/site-packages/simplejson-2.0.9-py2.4-linux-x86_64.egg
# zipimport: found 77 names in /appl/mtg/dev/vendor/html5lib/lib/python/html5lib-0.90-py2.4.egg
# /usr/lib64/python2.4/new.pyc matches /usr/lib64/python2.4/new.py
import new # precompiled from /usr/lib64/python2.4/new.pyc
# /usr/lib64/python2.4/warnings.pyc matches /usr/lib64/python2.4/warnings.py
import warnings # precompiled from /usr/lib64/python2.4/warnings.pyc
# /usr/lib64/python2.4/linecache.pyc matches /usr/lib64/python2.4/linecache.py
import linecache # precompiled from /usr/lib64/python2.4/linecache.pyc
# /usr/lib64/python2.4/encodings/ascii.pyc matches /usr/lib64/python2.4/encodings/ascii.py
import encodings.ascii # precompiled from /usr/lib64/python2.4/encodings/ascii.pyc
Python 2.4.3 (#1, Jun 11 2009, 14:09:37)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
import marshal # builtin
# /usr/lib64/python2.4/traceback.pyc matches /usr/lib64/python2.4/traceback.py
import traceback # precompiled from /usr/lib64/python2.4/traceback.pyc
File "test.py", line 11
if fsize == int(os.path.getsize(prcfile))
^
SyntaxError: invalid syntax
# clear __builtin__._
# clear sys.path
# clear sys.argv
# clear sys.ps1
# clear sys.ps2
# clear sys.exitfunc
# clear sys.exc_type
# clear sys.exc_value
# clear sys.exc_traceback
# clear sys.last_type
# clear sys.last_value
# clear sys.last_traceback
# clear sys.path_hooks
# clear sys.path_importer_cache
# clear sys.meta_path
# restore sys.stdin
# restore sys.stdout
# restore sys.stderr
# cleanup __main__
# cleanup[1] japanese
# cleanup[1] errno
# cleanup[1] imp
# cleanup[1] _codecs
# cleanup[1] new
# cleanup[1] zipimport
# cleanup[1] warnings
# cleanup[1] site
# cleanup[1] signal
# cleanup[1] traceback
# cleanup[1] linecache
# cleanup[1] posix
# cleanup[1] marshal
# cleanup[1] japanese.aliases
# cleanup[1] encodings
# cleanup[1] encodings.ascii
# cleanup[1] codecs
# cleanup[1] types
# cleanup[1] encodings.aliases
# cleanup[1] exceptions
# cleanup[2] copy_reg
# cleanup[2] posixpath
# cleanup[2] os.path
# cleanup[2] stat
# cleanup[2] UserDict
# cleanup[2] os
# cleanup sys
# cleanup __builtin__
# cleanup ints: 810 unfreed ints in 21 out of 26 blocks
# cleanup floats




Correct the code's error and check again

1 #!/usr/bin/python
2
3 import os
4 import time
5
6 prcfile="/home/sjing/prod/man/python/batch_cf/tmpfile"
7 while True:
8 fsize=int(os.path.getsize(prcfile))
9 print "file %s size is : %d" % (prcfile,fsize)
10 time.sleep(10)
11 if fsize == int(os.path.getsize(prcfile)):
12 print "transaction is done. Exit\n"
13 break



-bash-3.2$ python -m py_compile test.py
-bash-3.2$

No error found.


Debugging Python in ipython and ipdb

In this blog, I'm going to demonstrate some very useful features of ipython, especially when debugging python using ipdb; examining objects and step into pdb/ipdb from Python code, etc.

You should know how to use pdb to debug python, if not, please visit my other blog article.

I'll briefly describe how to exam objects and name space in iPython and then demonstrate how to debug Python in iPython/ipdb.

Note: a quick link to ipython. (make sure you come back here after reading the external page)

(*** Very Important ***)
If your ipython is 0.10 or earlier version, you need to do the following to fix a bug in checkline() method in IPython/Debugger.py.

shan@ub1:~/code$ python
Python 2.6.4 (r264:75706, Dec 7 2009, 18:45:15)
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import IPython
>>> IPython.Debugger.__file__
'/usr/local/lib/python2.6/dist-packages/IPython/Debugger.py'

wget the updated Debugger.py from here and replace the above Debugger.py with the new one. That's it.


Let's start with examining object and name space in ipython:

1. Start ipdb from ipython: (a very useful way to debug python code)

First, let's start ipython

shan@ub1:~/code$ ipython
Python 2.6.4 (r264:75706, Dec 7 2009, 18:45:15)
Type "copyright", "credits" or "license" for more information.

IPython 0.10 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object'. ?object also works, ?? prints more.

In [1]:
The first few lines already show the most important tricks of ipython. The problem is most people rarely read it!


Now, let's import test code and see some of the features python IDLE doesn't offer.

shan@ub1:~/code$ ipython
Python 2.6.4 (r264:75706, Dec 7 2009, 18:45:15)
Type "copyright", "credits" or "license" for more information.

IPython 0.10 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object'. ?object also works, ?? prints more.

In [1]: import test2

In [2]: import sys

In [3]: import os

In [4]: who
os sys test2

In [5]: whos
Variable Type Data/Info
------------------------------
os module
sys module
test2 module

In [6]: test2?
Type: module
Base Class:
String Form:
Namespace: Interactive
File: /home/shan/code/test2.py
Docstring:



In [7]: test2??
Type: module
Base Class:
String Form:
Namespace: Interactive
File: /home/shan/code/test2.py
Source:
#/usr/bin/env python

class Employee:
def __init__(self, name, title=None, phone=5551212):
self.name = name
self.title = title
self.phone = phone

def employee():
shan = Employee("Shan Jing","UNIX ADMIN","2720748")
import pdb;pdb.set_trace()
joe = Employee("Joe Smith","Application Developer","7207837")
print (shan.name, shan.title, shan.phone)
print (joe.name, joe.title, joe.phone)


if __name__ == '__main__':
employee()

In [8]:


In the above, we typed 7 commands (3 imports, who, whos, test2? and test2??):

From the output, I'm sure you already know what each command does. But just in case you need to know the detail about these commands, type '%cmd?' in ipython. (see below). By the way, this was printed in the last line of the ipython's startup message - "object? -> Details about 'object'. ?? prints more details about the object."

In [9]: %who?
Type: Magic function
Base Class:
String Form: >
Namespace: IPython internal
File: /usr/local/lib/python2.6/dist-packages/IPython/Magic.py
Definition: %who(self, parameter_s='')
Docstring:
Print all interactive variables, with some minimal formatting.

If any arguments are given, only variables whose type matches one of
these are printed. For example:

%who function str

will only list functions and strings, excluding all other types of
variables. To find the proper type names, simply use type(var) at a
command line to see how python prints type names. For example:

In [1]: type('hello')
Out[1]:

indicates that the type name for strings is 'str'.

%who always excludes executed names loaded through your configuration
file and things which are internal to IPython.

This is deliberate, as typically you may load many modules and the
purpose of %who is to show you only what you've manually defined.


whos gives extra information about an object.



Next, let's see how to use ipython to debug our code.

If you are already inside ipython, to start pdb/ipdb, all you need to do is run the code with -d option:

In [14]: run -d test2.py
*** Blank or comment
*** Blank or comment
Breakpoint 1 at /home/shan/code/test2.py:3
NOTE: Enter 'c' at the ipdb> prompt to start your script.
> (1)()

ipdb> c
> /home/shan/code/test2.py(3)()
2
1---> 3 class Employee:
4 def __init__(self, name, title=None, phone=5551212):

ipdb> l
1 #/usr/bin/env python

2
1---> 3 class Employee:
4 def __init__(self, name, title=None, phone=5551212):
5 self.name = name
6 self.title = title
7 self.phone = phone
8
9 def employee():
10 shan = Employee("Shan Jing","UNIX ADMIN","2720748")
11 import pdb;pdb.set_trace()
ipdb> h

Documented commands (type help ):
========================================
EOF bt cont enable jump pdef r tbreak w
a c continue exit l pdoc restart u whatis
alias cl d h list pinfo return unalias where
args clear debug help n pp run unt
b commands disable ignore next q s until
break condition down j p quit step up

Miscellaneous help topics:
==========================
exec pdb

Undocumented commands:
======================
retval rv

As we can see, 'run -d code' brings us into ipdb. The rest is same as pdb except ipdb offers good features like command line completion, etc.


2. Set ipdb breakpoints from Python source code

First, you can always set pdb breakpoints as I described in a previous blog. After you pdb.set_trace(), when you run codes from ipython, it will open pdb instead of ipdb. The differences between pdb and ipdb is ipdb offers some features that ipython offers while pdb doesn't.

Now, if you want to set breakpoint and opens ipdb inside ipython when your code reaches the breakpoint, all you need to do is:

insert the following code at your breakpoint:

from IPython.Debugger import Tracer; breakpoint1 = Tracer()


Challenge: Write a vim key map to map F1 key in insert mode to insert a breakpoint in your source code without the need of typing the above import code. Also, map F6 key to open a shell in a split window then start ipython/ipdb to debug the code you are editting. Hint: vim's % variable and ConqueShell or Vimshell.

Answer: please refer to my previous blog Debugging Python with pdb.

Debugging Python in pdb and setup breakpoints from Python code

This blog describes some techniques/tips on how to debug Python using the Python debugger pdb.

My "Vim Python IDE"
The following screen shot shows VIM configured as an almost full function Python IDE - there are 3 windows in VIM. I edit source code in the upper left window and open the lower left window for ipython, which shows imported objects in current namespace; finally, I have pdb debugging the source code at the right side window. All these can be done with a single key stroke inside VIM! (click the image to enlarge and click again to maximize it)




First, I'll show how to use basic Python pdb commands to debug Python code; then I'll show how to setup pdb breakpoints from the source code.

Then, I'll list vim key mapping that inserts breakpoints by just pressing F1 in vim and use F5 key to debug Python code without leaving Vim editor. Use this technique, you can debug Python code and travel back-and-forth between vim and pdb quickly.

At the end, I'll list additional vim key mappings (Thanks ConqueShell!!) that enable a single key stroke to trigger a pdb/ipython/ipdb in a split window inside Vim. This will turn your Vim into a full function Python IDE, you can edit and debug source code, evaluate attributes, objects in one vim screen!

Assume I'm working on the follow code and want to debug it:
shan@ub1:~/code$ nl test.py
1 #/usr/bin/env python

2 class Employee:
3 def __init__(self, name, title=None, phone=5551212):
4 self.name = name
5 self.title = title
6 self.phone = phone

7 def employee():
8 shan = Employee("Shan Jing","UNIX ADMIN","2720748")
9 joe = Employee("Joe Smith","Application Developer","7207837")
10 print (shan.name, shan.title, shan.phone)
11 print (joe.name, joe.title, joe.phone)


12 if __name__ == '__main__':
13 employee()

1. Use pdb to debug python (from Python IDLE):

Basic procedures:
a. start python IDLE
b. import pdb module
c. import python-code
d. type pdb.run('module.method()') and you will stop at (Pdb) prompt:

For exmaple:

shan@ub1:~/code$ python
Python 2.6.4 (r264:75706, Dec 7 2009, 18:45:15)
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pdb
>>> import test
>>> pdb.run('test.employee()')
> (1)()->None
(Pdb)




Now, type pdb command 's' to step into your code.

(Pdb) s
--Call--
> /home/shan/code/test.py(9)employee()
The line
>/home/shan/code/test.py(9)employee()
means the we stop at code /home/shan/code/test.py line 9 which is employee() method/function.

The "--Call--" indicates we step into a function/method, if you don't want to step into a function but want to execute it, use the 'n' command instead (explained later)

type pdb command 'l' to list source code. If you want to see more lines of code, you can type 'l range' eg 'l 12,22' to display line 12 to 22.


(Pdb) l
4 def __init__(self, name, title=None, phone=5551212):
5 self.name = name
6 self.title = title
7 self.phone = phone
8
9 -> def employee():
10 shan = Employee("Shan Jing","UNIX ADMIN","2720748")
11 joe = Employee("Joe Smith","Application Developer","7207837")
12 print (shan.name, shan.title, shan.phone)
13 print (joe.name, joe.title, joe.phone)
14



Please note, from the above output, the following line shows the NEXT line of code Python is about to execute.

'9 -> def employee():'

When using 's' (step command), it's good to know two other related commands - 'n' and 'c'.

Command 'n' (next) will execute the command one line at a time without stepping into functions.

Command 'c' (continue) will continue to execute until the end of current function.

Another useful pdb command is 'w' (where), it shows how the python compiler gets to the current line. It's very useful if your current position is nested.

Finally, to inspect attribute's value, use p (print) command. Or just type attribute's name.

Here's a complete run:

shan@ub1:~/code$ python
Python 2.6.4 (r264:75706, Dec 7 2009, 18:45:15)
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pdb
>>> import test
>>> pdb.run('test.employee()')
> (1)()
(Pdb) s
--Call--
> /home/shan/code/test.py(9)employee()
-> def employee():
(Pdb) l
4 def __init__(self, name, title=None, phone=5551212):
5 self.name = name
6 self.title = title
7 self.phone = phone
8
9 -> def employee():
10 shan = Employee("Shan Jing","UNIX ADMIN","2720748")
11 joe = Employee("Joe Smith","Application Developer","7207837")
12 print (shan.name, shan.title, shan.phone)
13 print (joe.name, joe.title, joe.phone)
14
(Pdb) s
> /home/shan/code/test.py(10)employee()
-> shan = Employee("Shan Jing","UNIX ADMIN","2720748")
(Pdb) s
--Call--
> /home/shan/code/test.py(4)__init__()
-> def __init__(self, name, title=None, phone=5551212):
(Pdb) s
> /home/shan/code/test.py(5)__init__()
-> self.name = name
(Pdb) s
> /home/shan/code/test.py(6)__init__()
-> self.title = title
(Pdb) s
> /home/shan/code/test.py(7)__init__()
-> self.phone = phone
(Pdb) s
--Return--
> /home/shan/code/test.py(7)__init__()->None
-> self.phone = phone
(Pdb) s
> /home/shan/code/test.py(11)employee()
-> joe = Employee("Joe Smith","Application Developer","7207837")
(Pdb) shan.title
'UNIX ADMIN'
(Pdb) print shan.phone
2720748
(Pdb) w
/usr/lib/python2.6/bdb.py(368)run()
-> exec cmd in globals, locals
(1)()
> /home/shan/code/test.py(11)employee()
-> joe = Employee("Joe Smith","Application Developer","7207837")
(Pdb) c
('Shan Jing', 'UNIX ADMIN', '2720748')
('Joe Smith', 'Application Developer', '7207837')

2. Setting breakpoints from Python code

Sometimes, you don't want to go thru the entire code line by line to debug. All you need is to jump into certain point in your code and debug that section only, you can do that by setting up a breakpoint in Python code.

Just add 'import pdb;pdb.set_trace()' at the point where you want to mark as breakpoint then run your code as normal, it will stop at the breakpoint and gives you pdb prompt.

See below line 9:

1 #/usr/bin/env python

2 class Employee:
3 def __init__(self, name, title=None, phone=5551212):
4 self.name = name
5 self.title = title
6 self.phone = phone

7 def employee():
8 shan = Employee("Shan Jing","UNIX ADMIN","2720748")
9 import pdb;pdb.set_trace()
10 joe = Employee("Joe Smith","Application Developer","7207837")
11 print (shan.name, shan.title, shan.phone)
12 print (joe.name, joe.title, joe.phone)


13 if __name__ == '__main__':
14 employee()


shan@ub1:~/code$ python test2.py
> /home/shan/code/test2.py(12)employee()
-> joe = Employee("Joe Smith","Application Developer","7207837")
(Pdb) l
7 self.phone = phone
8
9 def employee():
10 shan = Employee("Shan Jing","UNIX ADMIN","2720748")
11 import pdb;pdb.set_trace()
12 -> joe = Employee("Joe Smith","Application Developer","7207837")
13 print (shan.name, shan.title, shan.phone)
14 print (joe.name, joe.title, joe.phone)
15
16
17 if __name__ == '__main__':
(Pdb) w
/home/shan/code/test2.py(18)()
-> employee()
> /home/shan/code/test2.py(12)employee()
-> joe = Employee("Joe Smith","Application Developer","7207837")
(Pdb) shan.title
'UNIX ADMIN'
(Pdb) c
('Shan Jing', 'UNIX ADMIN', '2720748')
('Joe Smith', 'Application Developer', '7207837')
shan@ub1:~/code$


Assume you are editing your code in Vim editor, wouldn't it be nice if just hit one key to set a breakpoint and hit another key to debug the code without leaving Vim?

Put the following lines in ~/.vimrc

"pdb setting : insert pdb breakpoints
imap <F1> import pdb;pdb.set_trace()
au BufRead *.py nmap <F5> :!python %

Now, at Vim's insert mode, find the place you want to insert a breakpoint, hit F1 key. To debug your code, (save it first) and hit F5 key, you will be stopped at the breakpoint in a shell, once done debugging just type 'exit' to close pdb and go back to vim continue your coding.

If you have conqueshell installed in vim, using the following Vim mappings, you will be able to open ipython and debug your code from a new buffer in a split window on the same screen on vim. Basically, you turn Vim into a dynamic Python/iPython IDE with a full function debugger!

" ConqueShell Mappings
nmap <F1> :ConqueTermSplit bash
nmap <F2> :ConqueTermVSplit bash
nmap <F3> :ConqueTermSplit ipython
nmap <F4> :ConqueTermVSplit ipython
nmap <F6> :execute 'ConqueTermSplit ipython '.expand('%:p')
nmap <F7> :execute 'ConqueTermVSplit ipython '.expand('%:p')
let g:ConqueTerm_EscKey = ''

I have posted my .vimrc in another blog.

If you want to use ipython (which is a much better IDE) and ipdb to debug Python, please read my next blog - Debugging Python with ipython and ipdb.


.vimrc for Python


"My .vimrc file (Shan Jing)
"
" Part 1. Python Vim settings
" Part 2. Misc Settings
" Part 3. Global Format Settings

" <F1> opens bash in a horizonal split window
" <F2> opens bash in a vertical split window
" <F3> opens ipython in a horizonal split window
" <F4> opens ipython in a vertical split window
" <F5> debug/run current python file in a shell
" <c-,> opens a shell in a buffer/work with tabnew
" <c-w>t<c-w>H swap windows shapes H->V
" <c-w>t<c-w>K swap windows shapes V->H
" :map or :map! to show current mappings
" :nmap - Display normal mode maps
" :imap - Display insert mode maps
" :vmap - Display visual and select mode maps
" :smap - Display select mode maps
" :xmap - Display visual mode maps
" :cmap - Display command-line mode maps
" :omap - Display operator pending mode maps


"1. Python vim settings
set tabstop=4 " numbers of spaces of tab character
set shiftwidth=4 " numbers of spaces to (auto)indent
set smarttab
set expandtab " tabs are converted to spaces,use only when required
set autoindent
set smartindent


if has("autocmd")
" clear au, note: au = autocmd
au!
" Restore cursor position
au BufReadPost * if line("'\"") > 0|if line("'\"") <= line("$")|exe("norm '\"")|else|exe "norm $"|endif|endif

"omnicompletion
filetype plugin on
set ofu=syntaxcomplete#Complete
au FileType python set omnifunc=pythoncomplete#
Complete
au FileType javascript set omnifunc=javascriptcomplete#CompleteJS
au FileType html set omnifunc=htmlcomplete#CompleteTags
au FileType css set omnifunc=csscomplete#CompleteCSS
au FileType xml set omnifunc=xmlcomplete#CompleteTags
au FileType php set omnifunc=phpcomplete#CompletePHP
au FileType c set omnifunc=ccomplete#Complete

"http://vim.wikia.com/wiki/Python_-_check_syntax_and_run_script
"nmap <buffer> <F5> :wmwG:r!python %<cr>`.
au BufRead *.py set makeprg=python\ -c\ \"import\ py_compile,sys;\ sys.stderr=sys.stdout;\ py_compile.compile(r'%')\"
au BufRead *.py set efm=%C\ %.%#,%A\ \ File\ \"%f\"\\,\ line\ %l%.%#,%Z%[%^\ ]%\\@=%m
au BufRead *.py nmap <F5> :!python %<cr>
au BufRead *.py set nowrap
au BufRead *.py set go+=b
au BufRead *.py set tabstop=4
"Python settings: auto indent the next line after these key words:
au BufRead *.py set smartindent cinwords=def,if,elif,else,for,while,try,except,finall

" Automatically chmod +x Python, Perl and Shell scripts
au BufWritePost *.py !chmod +x %
au BufWritePost *.sh !chmod +x %
au BufWritePost *.pl !chmod +x %

"remove all training space when editing
"au BufWritePre *.py :%s/\s\+$//e
endif

"3. Other global setttings
"3a. A Python Calculator
"3b. ConqueShell Mappings
"3c. vim 7+ tabs settings
"3d. color scheme settings
"3e. general coding settings for c,cpp,php,java, etc
"3f. global format settings


"A Python Calculator for python aware vim
"usage : command mode: Calc x*y
:command! -nargs=+ Calc :py print <args>
:py from math import *

" ConqueShell Mappings
nmap <F1> :ConqueTermSplit bash<cr>
nmap <F2> :ConqueTermVSplit bash<cr>
nmap <F3> :ConqueTermSplit ipython<cr>
nmap <F4> :ConqueTermVSplit ipython<cr>
nmap <F6> :ConqueTerm bash -o vi<cr>
let g:ConqueTerm_EscKey = '<c-k>'

"map <F9> :previous<cr> " map F9 to open previous buffer
"map <F10> :next<cr> " map F10 to open next buffer

" turn off highlighted search
map <silent> <c-h> :silent noh<cr>

"editting and sourcing .vimrc
map ,e :vsp ~/.vimrc<cr> " edit my .vimrc file
map ,u :source ~/.vimrc<cr> " update vim setting

"insert the directory name of the current buffer
imap <2> <c-r>=expand('%:p:h')<cr>

"For vim 7.0+ tabs limit (default is 10)
set tabpagemax=15
"open a new tab
map ,t :tabnew<cr>



"command mode mappings
"insert the directory name of the current buffer at command mode
"e.g., :!ls <2> will do ls dir at current cursor
cmap <F2> <c-r>=expand('%:p:h')<cr>


"color scheme setting
if has("gui_running")
" See ~/.gvimrc
set guifont=Monospace\ 10 " use this font
set lines=50 " height = 50 lines
set columns=100 " width = 100 columns
set background=light " adapt colors for background
set selectmode=mouse,key,cmd
set keymodel=
else
colorscheme elflord " use this color scheme
set background=dark " adapt colors for background
endif

if has("autocmd")
" Filetypes (au = autocmd)
au FileType helpfile set nonumber " no line numbers when viewing help
au FileType helpfile nnoremap <buffer><cr> <c-]> " Enter selects subject
au FileType helpfile nnoremap <buffer><bs> <c-t> " Backspace to go back

" When using mutt, text width=72
au FileType mail,tex set textwidth=72
au FileType cpp,c,java,sh,pl,php,asp set autoindent
au FileType cpp,c,java,sh,pl,php,asp set smartindent
au FileType cpp,c,java,sh,pl,php,asp set cindent
"au BufRead mutt*[0-9] set tw=72


" File formats
au BufNewFile,BufRead *.pls set syntax=dosini
au BufNewFile,BufRead modprobe.conf set syntax=modconf
"remove all training space when editing
au BufWritePre *.py :%s/\s\+$//e
endif

"Global format settings
set nocompatible " use vim defaults
set ls=2 " allways show status line
set scrolloff=3 " keep 3 lines when scrolling
set showcmd " display incomplete commands
set hlsearch " highlight searches
set incsearch " do incremental searching
set ruler " show the cursor position all the time
set visualbell t_vb= " turn off error beep/flash
set novisualbell " turn off visual bell
set nobackup " do not keep a backup file
set number " show line numbers
set ignorecase " ignore case when searching
"set noignorecase " don't ignore case
set title " show title in console title bar
set ttyfast " smoother changes
"set ttyscroll=0 " turn off scrolling, didn't work well with PuTTY
set modeline " last lines in document sets vim mode
set modelines=3 " number lines checked for modelines
set shortmess=atI " Abbreviate messages
set nostartofline " don't jump to first character when paging
set whichwrap=b,s,h,l,<,>,[,] " move freely between files


syntax on

Installing Net::SSH::Perl - SSH module for Perl

Before reading further, do you know how to find out what Perl modules are installed on your system? Hmm...

How about "perldoc perllocal" or

sed -n -e "/head2/p" /usr/local/lib/perl/5.10.0/perllocal.pod | cut -dL -f2 | sort


Anyway, there are few other ways to do it. Make sure you know one of them.

Today, I'm given a task to write a Perl script to scan 200 servers' Interwoven, EPA, JVM and web server configurations. Normally, I'd like to use shell script to do anything related to SSH. However, with so many text files to scan why don't we use Perl to do it?

Perl SSH? Google over the Internet (thanks for the US for not censoring Google's searching result), it appears Net::SSH::Perl is a good but tough candidate. Most reviews complain about the painful installation process which makes it a challenge to me. I like pain, right? (not really)

Now, let's see what is Net::SSH::Perl.

"Net::SSH::Perl is an all-Perl module implementing an SSH client. It is compatible with both the SSH1 and SSH2 protocols.

Net::SSH::Perl enables you to simply and securely execute commands on remote machines, and receive the STDOUT, STDERR, and exit status of that remote command."

Good enough. Let's do it. (For more information about Net::SSH::Perl, see CPAN).

Installation of Net::SSH::Perl is really painful if your system doesn't have the required Math packages.

For cygwin, as of today (March 25,2010), because Perl 5.10.1 is compiled with gcc 4.3.4 (20090804) while gcc for cygwin is only at 3.4.4, the -fstack-protector gcc flags kills you. I tried to force install the module, it always stop at complaining the -fstack-protector flag when running perl script calling SSH module.

You have to recompile gcc in cygwin in order to use this module. To compile gcc, you need m4, Math::Pari, GMP, MFPR... OK, just kill me. I'm giving up.

Luckily, we have Ubuntu. With the pleasant apt-get utility, installation is much better than cygwin.


Installing Net::SSH::Perl module on Linux server (Ubuntu 9.x)


I'm using a dirty quick solution, first, let's cpan the module with force; then, we clean up the mess. Don't try it on production systems.


1. Force install Net::SSH::Perl

cpan> force install Net::SSH::Perl


2. Clean up work. Let's see the error messages at the end of installation and re-install these packages manually.

Go to .cpan/build/ and compile the following packages manually.

ILYAZ/modules/Math-Pari-2.01080604.tar.gz : writemakefile NO '/usr/bin/perl Makefile.PL INSTALLDIRS=site' returned status 2304
VIPUL/Crypt-Random-1.25.tar.gz : make_test FAILED but failure ignored because 'force' in effect
VIPUL/Crypt-Primes-0.50.tar.gz : make_test FAILED but failure ignored because 'force' in effect
VIPUL/Crypt-RSA-1.99.tar.gz : make_test FAILED but failure ignored because 'force' in effect
TURNSTEP/Math-GMP-2.06.tar.gz : make NO
TURNSTEP/Net-SSH-Perl-1.34.tar.gz : make_test FAILED but failure ignored because 'force' in effect



For GMP, because it was never installed, download it from http://gmplib.org/.
However, before we install GMP, we need to install m4...
To Install m4, we need to install (nothing)... there's always a clue that leads to another clue before you can find the treasure.


sudo apt-get install m4

download gmp from http://gmplib.org/ and compile/install it



Now, we need to check what Perl modules are installed on our server: (yes, another good interview question).

1. If "instmodsh" is available, use that.

2. Or use "perldoc perllocal". Or

3. Check the perllocal.pod directly:

root@ubuntu:~/perl# sed -n -e "/head2/p" /usr/local/lib/perl/5.10.0/perllocal.pod | cut -dL -f2 | sort
< Class::
< convert::ascii::armour|convert::ascii::armour>
< convert::asn1|convert::asn1>
< convert::pem|convert::pem>
< crypt::blowfish|crypt::blowfish>
< crypt::cbc|crypt::cbc>
< crypt::des|crypt::des>
< crypt::des_ede3|crypt::des_ede3>
< crypt::dh|crypt::dh>
< crypt::dsa|crypt::dsa>
< crypt::idea|crypt::idea>
< crypt::primes|crypt::primes>
< crypt::primes|crypt::primes>
< crypt::primes|crypt::primes>
< crypt::random|crypt::random>
< crypt::random|crypt::random>
< crypt::random|crypt::random>
< crypt::rsa|crypt::rsa>
< data::buffer|data::buffer>
< dbi|dbi>
< digest::bubblebabble|digest::bubblebabble>
< digest::hmac|digest::hmac>
< digest::md2|digest::md2>
< digest::sha1|digest::sha1>
< file::homedir|file::homedir>
< file::which|file::which>
< ipc::run3|ipc::run3>
< math::pari|math::pari>
< net::ssh::perl|net::ssh::perl>
< sort::versions|sort::versions>
< string::crc32|string::crc32>
< test::script|test::script>
< test::simple|test::simple>
< tie::encryptedhash|tie::encryptedhash>


OK. Now, it's time to check our handy Net::SSH:Perl module.


#!/usr/bin/perl

$host = localhost;
$cmd = "uname -n";
print "Trying to ssh into server $host. Please wait...\n";
use Net::SSH::Perl;
my $ssh = Net::SSH::Perl->new($host);
$ssh->login("shan","newyork1");
$ssh->shell;



root@ubuntu:~/perl# perl ssh1.pl

Trying to ssh into server localhost. Please wait...
Math::BigInt: couldn't load specified math lib(s), fallback to Math::BigInt::FastCalc at /usr/local/share/perl/5.10.0/Crypt/DH.pm line 6
Last login: Thu Mar 25 14:57:18 2010 from localhost
shan@ubuntu:~$ uname -a
uname -a
Linux ubuntu 2.6.31-14-generic #48-Ubuntu SMP Fri Oct 16 14:04:26 UTC 2009 i686 GNU/Linux
shan@ubuntu:~$ id
id
uid=1000(shan) gid=1000(shan) groups=4(adm),20(dialout),24(cdrom),46(plugdev),104(lpadmin),115(admin),120(sambashare),1000(shan)
shan@ubuntu:~$
shan@ubuntu:~$ exit
exit
logout



It opens an ssh connection!

The installation is painful but not completely bad.

Perl Regular Expression Summary


Meta Characters
\s white space [\f\t\n\r ] or form-feed, tab, newline, carriage return, space
\d any digit [0-9]
\w word character [A-Za-z0-9_].


Matches with m//

m/\w*?/ m(\w*?) m{w*?} or m#http://# ....


Perl's Option Modifiers /s /i /x or "/six"

/s : #it searches pattern in multiple lines across newlines '\n'; the "." looks within single line
/i : #case (i)nsensitive
/x : #add white space, useful for adding comments in regular expressions

You can combine option modifiers


Word Anchors /\bword\b/ or #\bword\b#

#\bperl# matches "perlscripting"

#perl\b# matches "useperl"

#\bsearc\B# matches "searches", "searching" but not "search" or "researching"



The Binding Operator =~

my #anwser = <STDIN> =~ #\byes\b/i;



The Match Variables $1, $2, ..., $` $& $'

if (#(\S+); (\S+); (\S+)#) {
print "$1 $2 $3 \n";
}

Note: sometimes, you need quote '$1' in print"";


It's always good to save your match in a variable instead of $1 as $1 may be changed by other matches

if ($statement =~ /(\w+)/) {
my $magicword = $1;
...
}


Parentheses for grouping only - non-capturing parentheses (?:)


1 #!/usr/bin/perl
2 use strict;
3
4 while (<>) {
5 if ( m#(?:UNIX).*?(Solaris|AIX|Linux)#i ) {
6 print "$1\n";
7 }
8 }


$ cat job.txt
Required skills:
1) 5 to 10 years Perl or Python , shell scripting skills;
2) at least 10 years UNIX e.g. AIX Solaris Linux Administration skills;
3) 5 to 10 years weblogic, tomcat, websphere administration skills;
Preferred skills:
1) UNIX Internal Coding Skills with Python, Perl or C;
2) TCP/IP Internal and Protocols;
3) Understand Java Servlets/JSP/Web Applications;
4) Understand Oracle/Sybase/MySQL;



# $1 is AIX not UNIX because (?:UNIX) is non-capturing.
$ ./98.pl < job.txt
AIX

shan@InternetPC ~/perl
$


Named Captures

In addition to capturing parts of string in $1, $2, $3...
Perl allows you to name captures in hash named %+: the key
is the lable we used and the value is the matched part in regex.



1 #!/usr/bin/perl
2 use 5.010;
3
4
5 while (<>) {
6 if ( m/(?:UNIX)\s*(?<os1>\w+) (?<os2>\w+) (?<os3>\w+) / ) {
7 say "Required Operating Systems are $+{os1} $+{os2} $+{os3}\n";
8 }
9 }

$ ./912.pl < job.txt
Required Operating Systems are AIX Solaris Linux



NEED to know vim copy groups between (...), eg copy block of chars inside parentheses

*****************************
Now that we have a way to label matches, we also need a way to refer to them for back
references. Previously, we used either \1 or \g{1} for this. With a labeled group, we can
use the label in \g{label}:
use 5.010;
my $names = 'Fred Flinstone and Wilma Flinstone';
if( $names =~ m/(?<last_name>\w+) and \w+ \g{last_name}/ ) {
say "I saw $+{last_name}";
}
We can do the same thing with another syntax. Instead of using \g{label}, we use
\k<label>:?
use 5.010;
my $names = 'Fred Flinstone and Wilma Flinstone';
if( $names =~ m/(?<last_name>\w+) and \w+ \k<last_name>/ ) {

******************************

General Quantifiers





Substitutions with S/// : search and replace

$_ = "green scaly dinosaur";
s/(\w+) (\w+)/$2, $1/; # Now it's "scaly, green dinosaur"
s/^/huge, /; # Now it's "huge, scaly, green dinosaur"
s/,.*een//; # Empty replacement: Now it's "huge dinosaur"
s/green/red/; # Failed match: still "huge dinosaur"
s/\w+$/($`!)$&/; # Now it's "huge (huge !)dinosaur"
s/\s+(!\W+)/$1 /; # Now it's "huge (huge!) dinosaur"
s/huge/gigantic/; # Now it's "gigantic (huge!) dinosaur"


There is a useful Boolean value from s///;
it is true if a substitution was successful;
otherwise, it is false:

$_ = "fred flintstone";
if (s/fred/wilma/) {
print "Successfully replaced fred with wilma!\n";
}


Option Modifiers: /s /i /x or "six"

s{ABC}{}s;

The Binding Operator : =~


Case Shifting

\U ; # \U escape forces what follows to all upper case
\L ; # \L escape forces lowercase

$_=" Steve and John are good friends\n";
s/(steve|john)/\U$1/gi -> STEVE and JOHN ...
s/(steve|john)/|L$1/gi -> steve and john ...



You can even stack them up.
Using \u with \L means ¡°all lowercase, but capitalize the
first letter¡±:*

s/(fred|barney)/\u\L$1/ig; # $_ is now "I saw Fred with Barney."

The split and join operators
@fields = split /separator/, $string;
@fields = split /:/, "abc:def::g:h"; # gives ("abc", "def", "", "g", "h")

my $result = join $glue, @pieces;
my $x = join ":", 4, 6, 8, 10, 12; # $x is "4:6:8:10:12"

m// in list context

When a pattern match (m//) is used in a list context, the return value is a list of the
memory variables created in the match, or an empty list if the match failed:
$_ = "Hello there, neighbor!";
my($first, $second, $third) = /(\S+) (\S+), (\S+)/;
print "$second is my $third\n";

my $text = "Fred dropped a 5 ton granite block on Mr. Slate";
my @words = ($text =~ /([a-z]+)/ig);
print "Result: @words\n";
# Result: Fred dropped a ton granite block on Mr Slate


Nongreedy Quantifiers "?" : *? +?


curl 'http://www.google.com' | perl -e 'while (<>) { m#<style>(.*?)</style>#; print "$&\n";}'



Matching Multiple-Line Text "/m" m=multiple lines

This is where Perl beats classic regular expressions. Perl can match multiple lines of text
just as matching single lines.

13 #9.15
14 $_="The only reason I'm using Perl is because \n it is perfect for processing text files \n with the built-in regular expressions in Perl.\n";
15
16 print "Found 'perl' at start of line\n" if /\bperl\b/im;
17 print "$`:$&:$'\n";

Found 'perl' at start of line
The only reason I'm using :Perl: is because
it is perfect for processing text files
with the built-in regular expressions in Perl.


19 #9.16 read entire google.html file into one variable, then add "__HTML__" in front of each line
20 $filename="/home/shan/perl/google.html";
21 open FILE, $filename
22 or die "Can't open '$filename': $!";
23 my $lines = join '', <FILE>;
24 $lines =~ s/^/__HTML__: /gm;
25 print $lines;


Updating Many Files:

1 #!/usr/bin/perl -w
2 # perl script to process many files
3 #
4 # usage : prcfile.pl input-file
5 #
6 # it will run regex operation inside while on input-file
7 # and saves the original file in the name input-file.org
8
9
10 use strict;
11
12 #name your own file extension for backup copy of the original file
13 $^I = ".org";
14
15 chomp (my $date = `date`);
16 while (<>) {
17
18 #insert your regex operations here:
19 s/^/$date:/;
20 s/$/___END___/;
21
22 #update
23 print;
24 }


$./prcfile 9.pl

$ more 9.pl
Mon Dec 14 17:09:45 PST 2009:#!/usr/bin/perl___END___
Mon Dec 14 17:09:45 PST 2009:___END___
Mon Dec 14 17:09:45 PST 2009:___END___
Mon Dec 14 17:09:45 PST 2009:___END___
Mon Dec 14 17:09:45 PST 2009:$_="steve and John are good friends\n";___END___
Mon Dec 14 17:09:45 PST 2009:s/(steve|john)/\U$1/gi; #-> STEVE and JOHN ...___END___
Mon Dec 14 17:09:45 PST 2009:print "$_\n";___END___
Mon Dec 14 17:09:45 PST 2009:___END___
Mon Dec 14 17:09:45 PST 2009:$_="steve and John are good friends\n";___END___
Mon Dec 14 17:09:45 PST 2009:s/(steve|john)/\L$1/gi; #-> steve and john ...___END___
Mon Dec 14 17:09:45 PST 2009:print "$_\n";___END___
Mon Dec 14 17:09:45 PST 2009:___END___
Mon Dec 14 17:09:45 PST 2009:#9.15___END___
Mon Dec 14 17:09:45 PST 2009:$_="The only reason I'm using Perl is because \n it is perfect for processing text files \n with the built-in regular expressions in Perl.\n";___END___
Mon Dec 14 17:09:45 PST 2009:___END___
Mon Dec 14 17:09:45 PST 2009:print "Found 'perl' at start of line\n" if /\bperl\b/im;___END___
Mon Dec 14 17:09:45 PST 2009:print "$`:$&:$'\n";___END___
Mon Dec 14 17:09:45 PST 2009:___END___
Mon Dec 14 17:09:45 PST 2009:#9.16___END___
Mon Dec 14 17:09:45 PST 2009:$filename="/home/shan/perl/google.html";___END___
Mon Dec 14 17:09:45 PST 2009:open FILE, $filename___END___
Mon Dec 14 17:09:45 PST 2009:or die "Can't open '$filename': $!";___END___
Mon Dec 14 17:09:45 PST 2009:my $lines = join '', <FILE>;___END___
Mon Dec 14 17:09:45 PST 2009:$lines =~ s/^/__HTML__: /gm;___END___
Mon Dec 14 17:09:45 PST 2009:print $lines;___END___



or, we can use command line to do the same:

perl -p -i.org -w -e 'chomp (my $date=`date`);s/^/$date:/g;' 9.pl


Here are the explanations of these command line options,

1. "perl -p" :

while ($_ = <STDIN>) {
print $_;
}

$_ , = and STDIN are all optional.

therefore, the above code can be written as:

while (<>) {
print;
}

2. "-i.org" #sets $^I to ".org"

3. "-w" #turns on warnings

4. -e " executable codes "

5. @ARGV

Accessing Database with Perl DBI

The best book to read for this subject is "Programming the Perl DBI".

#1. Install Perl DBI::DBD module
(You may need to remove any installed older versions of perl-DBI-mysql package)

Several ways to install Perl modules.
1. Download source and compile;
2. Use perl -MCPAN -e 'install DBD::mysql';
3. Use 'cpan';

For more details about installing Perl modules, please visit here.


I pick 3, use cpan.


cpan DBI
cpan DBD::mysql

(sample run of cpan File::HomeDir)
[root@ipc4 ~]# cpan File::HomeDir
CPAN: Storable loaded ok
Going to read /home/shan/perl/.cpan/Metadata
Database was generated on Tue, 15 Dec 2009 22:06:58 GMT
Running install for module File::HomeDir
Running make for A/AD/ADAMK/File-HomeDir-0.88.tar.gz
CPAN: LWP::UserAgent loaded ok
Fetching with LWP:
ftp://cpan-du.viaverio.com/pub/CPAN/authors/id/A/AD/ADAMK/File-HomeDir-0.88.tar.gz
CPAN: Digest::MD5 loaded ok
Fetching with LWP:
ftp://cpan-du.viaverio.com/pub/CPAN/authors/id/A/AD/ADAMK/CHECKSUMS
Checksum for /home/shan/perl/.cpan/sources/authors/id/A/AD/ADAMK/File-HomeDir-0.88.tar.gz ok
Scanning cache /home/shan/perl/.cpan/build for sizes
File-HomeDir-0.88/
...
CPAN.pm: Going to build A/AD/ADAMK/File-HomeDir-0.88.tar.gz

Checking if your kit is complete...
Looks good
...
Appending installation info to /usr/lib/perl5/5.8.5/i386-linux-thread-multi/perllocal.pod
/usr/bin/make install -- OK



#2. Use instmodsh utility to manage installed modules
(if instmodsh doesn't show all installed modules, use perldoc perllocal or check perllocal file directly to see the installed modules)

root@drbl-01 DBD-mysql-4.010]# instmodsh
Available commands are:
l – List all installed modules
m – Select a module
q – Quit the program
cmd? l
Installed modules are:
CGI
Cwd
DBD::mysql
DBI
ExtUtils::MakeMaker
FCGI
File::HomeDir
HTML::Parser
IO
Perl
Test::Simple
mod_perl2


cmd? m DBI
Available commands are:
f [all|prog|doc] - List installed files of a given type
d [all|prog|doc] - List the directories used by a module
v - Validate the .packlist - check for missing files
t - Create a tar archive of the module
q - Quit the module

DBI cmd?
Available commands are:
f [all|prog|doc] - List installed files of a given type
d [all|prog|doc] - List the directories used by a module
v - Validate the .packlist - check for missing files
t - Create a tar archive of the module
q - Quit the module
DBI cmd? d
all directories in DBI are:
/usr/bin
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/Bundle
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBD
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBD/Gofer/Policy
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBD/Gofer/Transport
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBI
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBI/Const
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBI/Const/GetInfo
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBI/DBD
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBI/Gofer
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBI/Gofer/Serializer
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBI/Gofer/Transport
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBI/ProfileDumper
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBI/SQL
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBI/Util
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/Win32
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/auto/DBI
/usr/share/man/man1
/usr/share/man/man3

You can us 't' command to create a tarball to install modules on those same arch servers that don't have access to the Internet directly (e.g. behind firewall).


#3. Grant user privilege to access database
[root@ipc4 perl]# mysql -u root -p
mysql > grant all privileges on database-name.* to 'user'@'localhost' identified by 'password';


#4. Create a testing database with two tables

[shan@ipc4 ~]$ mysql -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 94
Server version: 5.1.39-community MySQL Community Server (GPL)

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> create database bestbuy;
Query OK, 1 row affected (0.00 sec)

mysql> use bestbuy;
Database changed

mysql> create table location(loc_id integer,
-> name varchar(50),
-> address varchar(60),
-> city varchar(20),
-> state char(2),
-> zipcode char(5));
Query OK, 0 rows affected (0.07 sec)


mysql> create table purchase
-> (item_id integer not null,
-> item_name varchar(30) not null,
-> purchase_date datetime,
-> loc_id integer,
-> unit_price decimal(8,2),
-> quantity decimal(6,2),
-> amount decimal(8,2),
-> comment varchar(100));
Query OK, 0 rows affected (0.06 sec)

mysql> insert into location(loc_id,name,city,state) values(1,'BestBuy','Pasadena','CA');
Query OK, 1 row affected (0.01 sec)

mysql> insert into location(loc_id,name,city,state) values(1,'BestBuy','Duarte','CA');
Query OK, 1 row affected (0.00 sec)

mysql> insert into location(loc_id,name,city,state) values(1,'BestBuy','Sierra Madre','CA');
Query OK, 1 row affected (0.00 sec)


mysql> describe purchase;
+---------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------------+--------------+------+-----+---------+-------+
| item_id | int(11) | NO | | NULL | |
| item_name | varchar(30) | NO | | NULL | |
| purchase_date | datetime | YES | | NULL | |
| loc_id | int(11) | YES | | NULL | |
| unit_price | decimal(8,2) | YES | | NULL | |
| quantity | decimal(6,2) | YES | | NULL | |
| amount | decimal(8,2) | YES | | NULL | |
| comment | varchar(100) | YES | | NULL | |
+---------------+--------------+------+-----+---------+-------+
8 rows in set (0.00 sec)


mysql> insert into purchase(item_id,item_name,purchase_date,loc_id,unit_price,quantity,amount,comment)
-> values(1,'Nitendo DJHero','2009-12-15 19:15:00',1,100,1,108,'Prepare for a DJ job at a LA club.');
Query OK, 1 row affected (0.00 sec)

mysql> insert into purchase(item_id,item_name,purchase_date,loc_id,unit_price,quantity,amount,comment) values(1,'Nitendo Wii','2009-12-15 19:15:00',1,199,1,218,'Prepare for a DJ job at a LA club.');
Query OK, 1 row affected (0.00 sec)

mysql> insert into purchase(item_id,item_name,purchase_date,loc_id,unit_price,quantity,amount,comment) values(1,'MacBookPro','2009-12-15 19:15:00',1,1699,1,1810,'Prepare for a DJ job at a LA club.');
Query OK, 1 row affected (0.00 sec)

mysql> insert into purchase(item_id,item_name,purchase_date,loc_id,unit_price,quantity,amount,comment) values(2,'Samsung LN55B650','2009-12-15 19:15:00',1,199,1,2150,'Prepare for a DJ job at a LA club.');
Query OK, 1 row affected (0.00 sec)

mysql> insert into purchase(item_id,item_name,purchase_date,loc_id,unit_price,quantity,amount,comment) values(3,'Samsung LN55iB8000','2009-12-15 19:15:00',1,1999,1,2150,'Prepare for a DJ job at a LA club.');
Query OK, 1 row affected (0.00 sec)

mysql> select * from location;
+--------+---------+---------+--------------+-------+---------+
| loc_id | name | address | city | state | zipcode |
+--------+---------+---------+--------------+-------+---------+
| 1 | BestBuy | NULL | Pasadena | CA | NULL |
| 1 | BestBuy | NULL | Duarte | CA | NULL |
| 1 | BestBuy | NULL | Sierra Madre | CA | NULL |
+--------+---------+---------+--------------+-------+---------+
3 rows in set (0.00 sec)



mysql> select * from purchase;
+---------+--------------------+---------------------+--------+------------+----------+---------+------------------------------------+
| item_id | item_name | purchase_date | loc_id | unit_price | quantity | amount | comment |
+---------+--------------------+---------------------+--------+------------+----------+---------+------------------------------------+
| 1 | Nitendo DJHero | 2009-12-15 19:15:00 | 1 | 100.00 | 1.00 | 108.00 | Prepare for a DJ job at a LA club. |
| 1 | Nitendo Wii | 2009-12-15 19:15:00 | 1 | 199.00 | 1.00 | 218.00 | Prepare for a DJ job at a LA club. |
| 1 | MacBookPro | 2009-12-15 19:15:00 | 1 | 1699.00 | 1.00 | 1810.00 | Prepare for a DJ job at a LA club. |
| 2 | Samsung LN55B650 | 2009-12-15 19:15:00 | 1 | 199.00 | 1.00 | 2150.00 | Prepare for a DJ job at a LA club. |
| 3 | Samsung LN55iB8000 | 2009-12-15 19:15:00 | 1 | 1999.00 | 1.00 | 2150.00 | Prepare for a DJ job at a LA club. |
+---------+--------------------+---------------------+--------+------------+----------+---------+------------------------------------+
5 rows in set (0.00 sec)


mysql> select item_id,item_name,purchase_date,loc_id,amount from purchase;
+---------+--------------------+---------------------+--------+---------+
| item_id | item_name | purchase_date | loc_id | amount |
+---------+--------------------+---------------------+--------+---------+
| 1 | Nitendo DJHero | 2009-12-15 19:15:00 | 1 | 108.00 |
| 1 | Nitendo Wii | 2009-12-15 19:15:00 | 1 | 218.00 |
| 1 | MacBookPro | 2009-12-15 19:15:00 | 1 | 1810.00 |
| 2 | Samsung LN55B650 | 2009-12-15 19:15:00 | 1 | 2150.00 |
| 3 | Samsung LN55iB8000 | 2009-12-15 19:15:00 | 1 | 2150.00 |
+---------+--------------------+---------------------+--------+---------+
5 rows in set (0.00 sec)


#5. Use Perl's DBI::mysql to modify and query the database
1 #!/usr/bin/perl -w
2
3 use DBI;
4
5 ## mysql user database name
6 $db ="bestbuy";
7
8 ## mysql database user name
9 $user = "shan";
10
11 ## mysql database password
12 $pass = "newyork1";
13
14 ## user hostname : This should be "localhost" but it can be diffrent too
15 $host="localhost";
16
17 # Get database handle
18 $dbh = DBI->connect("DBI:mysql:$db:$host", $user, $pass, {RaiseError=>1});
19
20 # 1. Use $dbh->do method to perform sql modification type operations
21
22 $results = $dbh->do(q{insert into purchase(item_id,item_name,purchase_date,loc_id,unit_price,quantity,amount,comment)
23 values(1,'SONY Laptop','2009-11-27 5:30:00',1,359,1,375,'Black Friday Bloody Reward')});
24 die "Unable to perform insert:$DBI::errstr\n" unless (defined $results);
25
26
27
28 # 2. Use $dbh->prepare/$dhb->execute methods to perform query type operations
29
30 $sth = $dbh->prepare(q{select * from purchase})
31
32 or die "Unable to prep our query:".$dbh->errstr."\n";
33
34 # sth is a statement handle retured by method prepare , DBI::st=HASH(0x8f7fbc4)->fetchrow_array()
35 my $rv = $sth->execute
36 or die "Unable to execute our query:".$dbh-errstr."\n";
37
38
39 # Output 2
40 print "The output of prepare-execute method sQL operation:\n";
41
42 while ( @row2 = $sth->fetchrow_array () ) {
43 my $temp2 = join "\t", @row2;
44 print " $temp2 \n";
45 }


Execution :
The above script insert record "SONY Laptop" and shows all records in purchase database.


[shan@ipc4 perl]$ perl bestbuy.pl
The output of prepare-execute method sQL operation:
1 Nitendo DJHero 2009-12-15 19:15:00 1 100.00 1.00 108.00 Prepare for a DJ job at a LA club.
1 Nitendo Wii 2009-12-15 19:15:00 1 199.00 1.00 218.00 Prepare for a DJ job at a LA club.
1 MacBookPro 2009-12-15 19:15:00 1 1699.00 1.00 1810.00 Prepare for a DJ job at a LA club.
2 Samsung LN55B650 2009-12-15 19:15:00 1 199.00 1.00 2150.00 Prepare for a DJ job at a LA club.
3 Samsung LN55iB8000 2009-12-15 19:15:00 1 1999.00 1.00 2150.00 Prepare for a DJ job at a LA club.
1 SONY Laptop 2009-11-27 05:30:00 1 359.00 1.00 375.00 Black Friday Bloody Reward



Useful Hint for DBAs:
Tired of re-typing a long SQL command after making a typo?
Add the following line in your home directory's .inputrc file, you will be able to use vi style editing mode inside mysql shell.
e.g. ESC k to recall the last command you typed, or ESC j to travel down the command line history .. cheers!

set editing-mode vi



#6.Troubleshooting:

#a. My initial (failed) installation of DBD::mysql
#cpan DBD::mysql

Cannot find the file 'mysql_config'! Your execution PATH doesn't seem
not contain the path to mysql_config. Resorting to guessed values!
Can't exec "mysql_config": No such file or directory at Makefile.PL line 464.
Can't find mysql_config. Use --mysql_config option to specify where mysql_config is located
Can't exec "mysql_config": No such file or directory at Makefile.PL line 464.
Can't find mysql_config. Use --mysql_config option to specify where mysql_config is located
Can't exec "mysql_config": No such file or directory at Makefile.PL line 464.
Can't find mysql_config. Use --mysql_config option to specify where mysql_config is located


#b. The root cause is it needs a file mysql_config which comes with MySQL-devel-community-5.x.x-0.rhel4.

Download it,

[shan@ipc4 rpm]$wget http://opensource.become.com/mysql/Downloads/MySQL-5.4/MySQL-devel-community-5.4.3-0.rhel4.i386.rpm
Or use curl,
[shan@ipc4 rpm]$curl http://opensource.become.com/mysql/Downloads/MySQL-5.4/MySQL-devel-community-5.4.3-0.rhel4.i386.rpm -o MySQL-devel-community-5.4.3-0.rhel4.i386.rpm

Install it:
[root@ipc4 rpm]# rpm -ivh MySQL-devel-community-5.4.3-0.rhel4.i386.rpm


Note, make sure you download the correct dev package to match your MySQL server.



#c. Reinstall DBD::mysql
[root@ipc4 perl]# cpan DBD::mysql
CPAN: Storable loaded ok
Going to read /home/shan/perl/.cpan/Metadata
Database was generated on Tue, 15 Dec 2009 22:06:58 GMT
Running install for module DBD::mysql
Running make for C/CA/CAPTTOFU/DBD-mysql-4.013.tar.gz
CPAN: Digest::MD5 loaded ok
Checksum for /home/shan/perl/.cpan/sources/authors/id/C/CA/CAPTTOFU/DBD-mysql-4.013.tar.gz ok
Scanning cache /home/shan/perl/.cpan/build for sizes
DBD-mysql-4.013/
cflags (mysql_config) = -I/usr/include/mysql -g -pipe -m32 -DUNIV_LINUX
embedded (mysql_config) =
libs (mysql_config) = -rdynamic -L/usr/lib/mysql -lmysqlclient -lz -lcrypt -lnsl -lm -lmygcc
mysql_config (guessed ) = mysql_config
nocatchstderr (default ) = 0
nofoundrows (default ) = 0
ssl (guessed ) = 0
testdb (default ) = test
testhost (default ) =
testpassword (default ) =
testsocket (default ) =
testuser (guessed ) = root

….


You may get errors like below, that’s non-critical, it merely complains the DBD::mysql test can’t access the ‘test’ database with root account and password “NO”. You can always manually force the installation by entering the working directory ~.CPAN/build/module_name/ and type “make install”.

nt.t line 15
t/80procs...................skipped
all skipped: ERROR: Access denied for user 'root'@'localhost' (using password: NO). Can't continue test
t/85init_command............skipped
all skipped: ERROR: Access denied for user 'root'@'localhost' (using password: NO). Can't continue test
Failed Test Stat Wstat Total Fail Failed List of Failed
-------------------------------------------------------------------------------
t/00base.t 2 512 6 12 200.00% 1-6
35 tests skipped.
Failed 1/36 test scripts, 97.22% okay. 6/6 subtests failed, 0.00% okay.
make: *** [test_dynamic] Error 255
/usr/bin/make test -- NOT OK
Running make install
make test had returned bad status, won't install without force


Manually install:

[root@ipc4 DBD-mysql-4.013]# make install
Installing /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/auto/DBD/mysql/mysql.so
Installing /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/auto/DBD/mysql/mysql.bs
Files found in blib/arch: installing files in blib/lib into architecture dependent library tree
Installing /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBD/mysql.pm
Installing /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBD/mysql/GetInfo.pm
Installing /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBD/mysql/INSTALL.pod
Installing /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/Bundle/DBD/mysql.pm
Installing /usr/share/man/man3/DBD::mysql::INSTALL.3pm
Installing /usr/share/man/man3/Bundle::DBD::mysql.3pm
Installing /usr/share/man/man3/DBD::mysql.3pm
Writing /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/auto/DBD/mysql/.packlist
Appending installation info to /usr/lib/perl5/5.8.5/i386-linux-thread-multi/perllocal.pod


#d. Verify the installation:
[root@ipc4 ~]# instmodsh
Available commands are:
l - List all installed modules
m - Select a module
q - Quit the program
cmd? l
Installed modules are:
DBD::mysql
DBI
Perl
mod_perl2
cmd? m DBD::mysql
Available commands are:
f [all|prog|doc] - List installed files of a given type
d [all|prog|doc] - List the directories used by a module
v - Validate the .packlist - check for missing files
t - Create a tar archive of the module
q - Quit the module
DBD::mysql cmd? d
all directories in DBD::mysql are:
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/Bundle/DBD
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBD
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBD/mysql
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/auto/DBD/mysql
/usr/share/man/man3
DBD::mysql cmd? v
DBD::mysql has no missing files
DBD::mysql cmd? f
all files in DBD::mysql are:
/usr/share/man/man3/DBD::mysql::INSTALL.3pm
/usr/share/man/man3/DBD::mysql.3pm
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBD/mysql/INSTALL.pod
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBD/mysql.pm
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/auto/DBD/mysql/mysql.bs
/usr/share/man/man3/Bundle::DBD::mysql.3pm
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/DBD/mysql/GetInfo.pm
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/auto/DBD/mysql/mysql.so
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/Bundle/DBD/mysql.pm
DBD::mysql cmd? q
cmd? q
[root@ipc4 ~]#



#e. Clean up:
If your server is installed RHEL 4.0 and it already has MySQL, Perl installed, you may need to do some cleaning work to make sure you are using the correct versions of the package for each components. The sooner you do the cleaning job, the better. Once you installed/configured Perl, MySQL, Python, PHP, etc, etc, you find out you need to upgrade a package that’s required by 100 packages, you will feel the pain. That’s why platform engineering is very important for open source technology.


On my system, I have RHEL 4.0, which installed very early version of mysqlclient and perl-DBD-mysql. In fact, the old version perl-DBD-mysql-2.9xx is not compatible with MySQL server 5.x. I need to remove all those obsolete packages without damaging the system.

remove mysqlclient10-3.23 and its dependencies:

[root@ipc4 rpm]# rpm -qa | grep mysql
mysqlclient10-3.23.58-4.RHEL4.1
[root@ipc4 rpm]# rpm -ql $(rpm -qa | grep mysql)
/etc/ld.so.conf.d/mysqlclient10-i386.conf
/usr/lib/mysql/libmysqlclient.so.10
/usr/lib/mysql/libmysqlclient.so.10.0.0
/usr/lib/mysql/libmysqlclient_r.so.10
/usr/lib/mysql/libmysqlclient_r.so.10.0.0

[root@ipc4 rpm]# rpm -e $(rpm -qa | grep mysql)
error: Failed dependencies:
libmysqlclient.so.10 is needed by (installed) perl-DBD-MySQL-2.9004-3.1.i386
libmysqlclient.so.10 is needed by (installed) MySQL-python-1.0.0-1.RHEL4.1.i386
libmysqlclient.so.10 is needed by (installed) MyODBC-2.50.39-21.RHEL4.1.i386
mysqlclient10 is needed by (installed) MySQL-python-1.0.0-1.RHEL4.1.i386

[root@ipc4 rpm]# rpm -e MySQL-python-1.0.0-1.RHEL4.1.i386
[root@ipc4 rpm]# rpm -e MyODBC-2.50.39-21.RHEL4.1.i386
[root@ipc4 rpm]# rpm -e MySQL-python-1.0.0-1.RHEL4.1.i386

also, remove the old perl-DBD-MySQL
[root@ipc4 rpm]# rpm -e perl-DBD-MySQL-2.9004-3.1.i386