“Perfer et obdura; dolor hic tibi proderit olim.” Be patient and tough; some day this pain will be useful to you.
The sharpening of saws will continue in the year 2013 or year 1 according to the Mayan calendar. Saws sharpened so far:
I have been told that my command line skills are pretty impressive. Such as the ability to let text scroll in different windows and sometimes even making the computer speak. Today I will attempt to share this knowledge with you. Most of it is unwritten and almost automatic, so it is hard to write it all down. However, this is no excuse and we don’t want these valuable lessons to be lost forever. Again I created a list of 7 habits of highly effective people.
1. Avoid typing and know your history
Typing is good and it is bad. It is good, because it is faster than a mouse, a GUI and all that fancy stuff. It is bad since it is work and costs energy. By the way you should definitely be able to touch type. This is a life changing thing if you are serious about working with computers. Get cheap software to teach yourself or search online for free alternative.
When it comes to avoiding typing, we have 4 possibilities – tab completion, master one editor (Vim for instance see above), aliasing and using history. Aliasing allows us to abbreviate a complex command. For instance we can create the following alias for finding Python files:
alias fpy="find . -name '*.py'"
Now we only need to type fpy to find Python files. Typing alias shows all the defined aliases.
History gives us access to previously typed commands. We can replay the previous command with:
- Pressing the up arrow
- k in vi mode. In fact we can scroll up and down the history with the j and k keys
!$ gives us the last word of the previous command. We can view a file and then edit it:
cat somefile vi !$
A common habit of people is to search through the history and then execute a command using the corresponding number:
history|grep searchterm !42
If you remember the first few letters of the command for example “ca” from cat then you can just perform the command with ! in front:
2. Conquer common commands
Covering all the popular commands is impossible without writing a complete book. Having written two books I know how much effort is involved. This is certainly not today’s objective. First, we start with mkdir. Sometimes we want to create a directory dir1 with subdirectories dir2 and dir3. We can do it quickly with -p switch:
mkdir -p dir1/dir2/dir3
Another special secret is to use find with xargs. For example, this snippet finds Python files modified within the past 7 days and searches for “numpy”:
find . -name '*.py' -mtime -7|xargs grep numpy
I also like to watch and tail. Especially tail -f, which is incredibly useful if you want to have text scrolling in your terminal.
We can find many more tricks at commandline fu such as this oneliner to quickly start a HTTP server:
python -m SimpleHTTPServer
3. Navigate with ease
You probably know that cd brings you to your home directory. cd - returns you to the previous directory. More advanced things can be done with pushd and popd. Other solutions exist. I quite like bashmarks. Once installed bashmarks adds 1 letter shell commands to save bookmarks of directories. You can navigate to those directories using the bookmark name.
Saves a bookmark of the current directory as “bookdir”.
Goes to the bookmarked directory.
4. Automate everything with scripts
A lot of things can be automated with shell scripts:
- repetitive tasks, backup
- data retrieval
When debugging scripts I like to use -x and -v switches. I also like to use sed and awk. Sed and awk are important tools in the shell scripting toolkit. However, they are pretty complex to master and they deserve a more thorough discussion than is possible here. Maybe another time. Some sed and awk resources:
5. Use regexes in moderation
Regular expressions can solve a lot of problems. Just as often you start with one problem and end up with two. Knowing when enough is enough is therefore important. First you should think about better techniques or specialized libraries. This means no parsing of HTML with regexes. Second do it in stages. Sometimes you can break up a regex in smaller chunks and apply them one by one. I guess the rule of thumb is that if you can write the regex in one go, then it is simple enough.
6. Getting help, man
man is the most sexist Unix command in the history of the Universe. Are man pages for real men? Do real men search the web or read books? Those are not the questions we should be asking. We can search the man pages for a phrase with the -k option. Searching for Python on my machine gives me:
man -k python Inline::Python(3pm) - Write Perl subs and classes in Python pydoc(1) - the Python documentation tool python(1) - an interpreted, interactive, object-oriented programming language python(1), pythonw(1) - an interpreted, interactive, object-oriented programming language pythonw(1) - run python script allowing GUI pythonw(1) - run python script with GUI
The apropos command does the same, so we could have just as well typed apropos python. The info command is an alternative to the man pages. You might find info pages easier to read.
7. Sharpen thy saw
Maximum productivity is achieved by constant sharpening of the saw. Even after achieving a high level of mastery, we need to continue learning. To help in this quest here are some resources:
- Ad Hoc Data Analysis From The Unix Command Line wikibook
- Guide to Unix
- The Linux Documentation Project
- Bash FAQ
- Practical Unix course from Stanford
- Fabric, Cuisine and Watchdog for server administration
- The Twisted Way
- Examples on how to customize Matplotlib plots
- New Years Python meme 2012 by Alex Clark
- Advanced Indexing and Array Concepts
- Pulsar – a Concurrent framework for Python
- Memory plots with memory_profiler
- A Guide to Python Frameworks for Hadoop
- Classic EM in Python: Multinomial sampling
- Recursive-descent in Python, using generators
Before I leave you here is a list of NumPy Cookbook reviews that I am aware of: