Handy Utilities

Handy utilities:
1. Remove file extensions - say we have 1000s .txt files, we need to remove the .txt extension, eg. bug.txt  -> bug

find . -name "*.txt" -exec rename .txt '' {} +

Note:

• {} + works the same way as fine's '-print0 | xargs -0 action', which works on a subset instead of every single line.

• rename takes 3 arguments, rename from to files, eg. rename .txt .text *.txt renames all *.txt to .text

Always look on the bright side of Life!

The best chapter of book 'Learning Python'

"Once you finish the quiz, you’ve officially reached the end of this book. Now that you
know Python inside and out, your next step, should you choose to take it, is to explore
the libraries, techniques, and tools available in the application domains in which you
work. Because Python is so widely used, you’ll find ample resources for using it in
almost any application you can think of—from GUIs, the Web, and databases to numeric
programming, robotics, and system administration.

This is where Python starts to become truly fun, but this is also where this book’s story
ends, and others’ begin. For pointers on where to turn after this book, see the list of
recommended follow-up texts in the Preface. Good luck with your journey. And of course,
“Always look on the bright side of Life!”

-Mark Lutz, the author of "Learning Python"

Configuring Python Matplotlib plot to GUI/X and Configuring TK for Python

To configure Python Matplotlib to display to GUI/X. You need to configure Matplotlib backend to be 'TKAgg' (default is Agg).

Problem:
Most Python compilers delivered with major Linux distributions don't have tcl/tk configured during the build. To test, try 'import Tkinter' in your Python. The underlying shared object is _tkinter.so. Tkinter is required by Matplotlib to enable TKAgg backend.

If your Python is not configured for Tk, you will need to rebuild Python compiler with tcl/tk and possibly rebuild matplotlib.

The key of configuring tcl/tk for Python compiler is to include tcl/tk header files while building Python compiler.

If you don't have root on the system, the best way is to download tcl/tk. Otherwise, you can just rpm tcl/tk devel RPMs. In either cases, you should install the correct version of header files as your system tcl/tk binary/libs.

Below are the detail steps to build everything from source code, it takes about 1 hour.

1. Build tcl 8.4.13
Download at:
http://www.linuxfromscratch.org/blfs/view/6.2.0/general/tcl.html
use --prefix to install at a non-standard location, e.g.
./configure --prefix=/home/sjing/tools

2. Build tk

Download at: http://www.linuxfromscratch.org/blfs/view/6.2.0/general/tk.html
use ./configure --prefix to install at a non-standard location, e.g.
./configure --prefix=/home/sjing/tools


3. Build Python with tkinter and Unicode4 (check your current system python compiler's ucs setting, see Reference 1 below. This will ensure your new Python compiler is compatible with your system's)
./configure --enable-unicode=ucs4 --prefix=/home/sjing/tools

To include tcl/tk in Python, you need to compile Python with tcl/tk header files in the -I path.

If you download tcl/tk from the above sites (step 1, 2), please edit Makefile and add tcl.h/tk.h headers path in CPPFLAGS macro

for example: assume /home/sjing/tools/include has tk.h andtcl.h from step 1 and 2.

CPPFLAGS= -I. -I$(srcdir)/Include -I/home/sjing/tools/include


-bash-3.2$ rpm -qa | grep ^tk
tk-devel-8.4.13-5.el5_1.1
tk-8.4.13-5.el5_1.1
tk-devel-8.4.13-5.el5_1.1
tk-8.4.13-5.el5_1.1
-bash-3.2$ rpm -qa | grep ^tcl
tcl-8.4.13-4.el5
tcl-devel-8.4.13-4.el5
tcl-8.4.13-4.el5
tclx-8.4.0-5.fc6
tcl-devel-8.4.13-4.el5


Once make can find the header files, build it and see the process of
building _tkinter.so file.

-------------
make

building '_tkinter' extension
gcc -pthread -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -fno-strict-aliasing -DWITH_APPINIT=1 -I/usr/X11/include -I. -I/home/sjing/downloads/Python-2.4.3/./Include -I/home/sjing/tools/include -I/usr/local/include -I/home/sjing/downloads/Python-2.4.3/Include -I/home/sjing/downloads/Python-2.4.3 -c /home/sjing/downloads/Python-2.4.3/Modules/_tkinter.c -o build/temp.linux-x86_64-2.4/_tkinter.o
gcc -pthread -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -fno-strict-aliasing -DWITH_APPINIT=1 -I/usr/X11/include -I. -I/home/sjing/downloads/Python-2.4.3/./Include -I/home/sjing/tools/include -I/usr/local/include -I/home/sjing/downloads/Python-2.4.3/Include -I/home/sjing/downloads/Python-2.4.3 -c /home/sjing/downloads/Python-2.4.3/Modules/tkappinit.c -o build/temp.linux-x86_64-2.4/tkappinit.o
gcc -pthread -shared build/temp.linux-x86_64-2.4/_tkinter.o build/temp.linux-x86_64-2.4/tkappinit.o -L/usr/X11/lib -L/home/sjing/tools/lib -L/usr/local/lib -ltk8.4 -ltcl8.4 -lX11 -o build/lib.linux-x86_64-2.4/_tkinter.so
--------------


4. Build matplotlib
Edit setupext.py and add tcl/tk location in 'basdir' dictionary.
(x84_64 bit uses 'linux2' key)
50 basedir = {
...
57 'linux2' : ['/usr/local', '/usr','/home/sjing/tools'],
...
78 }

To confirm Tkinter will be built with matplotlib.
Make sure you use the Python compiler you just built with tcl/tk by adding the path of Python in front of your old python path.

-bash-3.2$ export PATH=/home/sjing/tools/bin:$PATH
( or /home/sjing/tools/bin/python setup.py build )

-bash-3.2$ python setup.py build
basedirlist is: ['/usr/local', '/usr', '/home/sjing/tools']
============================================================================
BUILDING MATPLOTLIB
matplotlib: 1.0.1
python: 2.4.3 (#1, Mar 29 2011, 14:57:37) [GCC 4.1.2
20080704 (Red Hat 4.1.2-48)]
platform: linux2

REQUIRED DEPENDENCIES
numpy: 1.2.0
freetype2: 9.10.3

OPTIONAL BACKEND DEPENDENCIES
libpng: 1.2.10
Tkinter: Tkinter: 39220, Tk: 8.4, Tcl: 8.4

Install matplotlib module:
-bash-3.2$ python setup.py install --prefix=/home/sjing/tools


5. Download matplotlib examples at http://matplotlib.sourceforge.net/examples/index.html


run the examples that use tk
animation example code: animate_decay_tk_blit.py
mplot3d example code: 2dcollections3d_demo.py


6. As long as your newly build and system python compilers are the same version and build with the same paramters (ucs), you should be able to run all your existing python codes/modules with the new compiler. But the best way is your System Adminsitrator update the system wide Python compile with tcl/tk following this guide.


Reference 1:

To check your Python is built with ucs2 or ucs4
1. ucs4
>>> import sys
>>> print sys.maxunicode
1114111

2. ucs2
>>> import sys
>>> print sys.maxunicode
65535

Reference 2:

matplotlibrc (lines defining backend and interactive mode)
--------------------------------------
backend : TKAgg
backend_fallback: True
interactive : False
------------------

Memcache and Binary Protocol

A very good Memcache Story explains memcache basic.

Article about binary protocol.

bash login profiles for all scenarios

When invoked as bash (not sh) the start-up files that bash reads for a
login shell are, in order:

i) /etc/profile

and then the first of any of the following that exists:

ii) ~/.bash_profile OR
iii) ~/.bash_login OR
iv) ~/.profile


Non-login interactive bash shells (when changing or starting new shells
during a session) read only ~/.bashrc.

Non-login non-interactive bash shells (ie-when launched from inside a
script) check the startup file in $BASH_ENV, or $ENV if that doesn't
exist.

An easy way to make sure all three invocations of bash reference the same
start-up file is:

i) use .bashrc as your customized startup file

ii) create a .bash_profile containing only:

 export BASH_ENV=~/.bashrc
 if [ -f ~/.bashrc ]; then source ~/.bashrc; fi



This way, after reading /etc/profile a login shell will read the first
file found, ~/.bash_profile, which in turn reads ~/.bashrc

Non-login interactive shells will read ~/.bashrc as usual

Non-login non-interactive shells will read the ~/.bashrc assigned in
$BASH_ENV.


Reference 1: BASH's sourcing profiles order in execution

[sjing@prodpmcf151 ~]$ bash -cx 'env'
+ '[' -f /etc/bashrc ']'
+ source /etc/bashrc
++ '[' 17934 -gt 99 ']'
+++ id -gn
+++ id -un
++ '[' techdept = sjing ']'
++ umask 022
++ '[' '' ']'
++ shopt -q login_shell
++ for i in '/etc/profile.d/*.sh'
++ '[' -r /etc/profile.d/colorls.sh ']'
++ '[' '' ']'
++ . /etc/profile.d/colorls.sh
++ for i in '/etc/profile.d/*.sh'
++ '[' -r /etc/profile.d/glib2.sh ']'
++ '[' '' ']'
++ . /etc/profile.d/glib2.sh
++ for i in '/etc/profile.d/*.sh'
++ '[' -r /etc/profile.d/gnome-ssh-askpass.sh ']'
++ '[' '' ']'
++ . /etc/profile.d/gnome-ssh-askpass.sh
++ for i in '/etc/profile.d/*.sh'
++ '[' -r /etc/profile.d/krb5-workstation.sh ']'
++ '[' '' ']'
++ . /etc/profile.d/krb5-workstation.sh
++ for i in '/etc/profile.d/*.sh'
++ '[' -r /etc/profile.d/lang.sh ']'
++ '[' '' ']'
++ . /etc/profile.d/lang.sh
++ for i in '/etc/profile.d/*.sh'
++ '[' -r /etc/profile.d/less.sh ']'
++ '[' '' ']'
++ . /etc/profile.d/less.sh
++ for i in '/etc/profile.d/*.sh'
++ '[' -r /etc/profile.d/vim.sh ']'
++ '[' '' ']'
++ . /etc/profile.d/vim.sh
++ for i in '/etc/profile.d/*.sh'
++ '[' -r /etc/profile.d/which-2.sh ']'
++ '[' '' ']'
++ . /etc/profile.d/which-2.sh
++ unset i
++ unset pathmunge
+ '[' '!' 1 ']'
+ env

Reference 2: UNIX/LINUX UNICODE FAQ
Reference 3: Useful 'find' Page