Bash
From AdminWiki
(→Bash Programming) |
(→Error and Output Redirect) |
||
Line 76: | Line 76: | ||
- | == | + | == Redirection == |
+ | === Standard Input, Output and Error === | ||
- | + | ==== Digression for a better understanding of file descriptors ==== | |
- | + | If you are on a system with a /proc filesystem (and may god have mercy on your soul if you are not), you will find that there are interesting symlinks in /proc/<pid>/fd. Let me elaborate. | |
+ | Consider the following: you are logged on, have an interactive shell like bash, and you are the owner of your [[terminal]] (tty). Now, find out your [[PID]]: | ||
+ | |||
+ | philip@dinky:~$ w |head -3 | ||
+ | 15:15:04 up 72 days, 17 min, 35 users, load average: 0.15, 0.11, 0.09 | ||
+ | USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT | ||
+ | philip tty2 - 11Feb09 58days 0.10s 0.00s /bin/login -- | ||
+ | philip@dinky:~$ ps waux |grep bash |grep tty2 | ||
+ | philip 22008 0.0 0.0 21396 1372 tty2 S Feb11 0:00 -bash | ||
+ | philip@dinky:~$ | ||
+ | |||
+ | now, with that information, let's have a peek into our filedescriptors: | ||
+ | |||
+ | philip@dinky:~$ ls -l /proc/22008/fd | ||
+ | total 0 | ||
+ | lrwx------ 1 philip philip 64 2009-04-10 15:16 0 -> /dev/tty2 | ||
+ | lrwx------ 1 philip philip 64 2009-04-10 15:16 1 -> /dev/tty2 | ||
+ | lrwx------ 1 philip philip 64 2009-04-10 15:16 2 -> /dev/tty2 | ||
+ | lrwx------ 1 philip philip 64 2009-04-10 15:16 255 -> /dev/tty2 | ||
+ | philip@dinky:~$ | ||
+ | |||
+ | there are, of course, redirections to your current interactive shell. '''This is why your software will crash''' receiving a SIGPIPE [[signal]] when you close your terminal while something is running. | ||
+ | |||
+ | So, what do those numbers mean? If software allocates file descriptors (thats what it gets when it opens files for reading or writing) from the operating system, it will receive integer numbers. Thus, the [[PID]] plus the [[file descriptor]] will uniquely identify an open file system wide - a bit like an ip-address + port pair for a TCP connection. | ||
+ | |||
+ | '''0''' is '''standard input''' | ||
+ | '''1''' is '''standard output''' | ||
+ | '''2''' is '''standard error''' | ||
+ | '''everything else''' are filedescriptors received when the software opened other files, like configuration files, log files etc. | ||
+ | '''255''' is just there to mess with my explanation. | ||
+ | |||
+ | The whole "standard X" metaphor is a bit to wrap ones head around, but after a while using them extensively and with deep understanding, one will be able to prevent a lot of shit from happening (processes dying under certain conditions, missing logfiles, mailbombs from cron etc) and finally lead a happy life. | ||
+ | |||
+ | The standard file descriptors are the fds where software will write and receive their user interaction by default. Yet, software does not necessarily need to do that; while libc provides for every running piece of software with those filedescriptors by standard, the software can close them willingly. Software can indeed decide to "unbind" itself from standard input and output by closing the respective file descriptor. Analog to that, the respective symlinks in the proc fs will disappear, and so will the depdendy on those files (like the interactive shell). | ||
+ | |||
+ | By using redirection, you can remap those standard file descriptors in your own way. Consider doing something like this on one terminal: | ||
+ | philip@dinky:~$ cat > /dev/null | ||
+ | and try finding the PID of this cat process, then do something like the following on another shell (without killing the cat of course): | ||
+ | philip@dinky:~$ ls /proc/10519/fd -l | ||
+ | total 0 | ||
+ | lrwx------ 1 philip philip 64 2009-04-10 15:31 0 -> /dev/pts/17 | ||
+ | l-wx------ 1 philip philip 64 2009-04-10 15:31 1 -> /dev/null | ||
+ | lrwx------ 1 philip philip 64 2009-04-10 15:31 2 -> /dev/pts/17 | ||
+ | so, i hope now it starts making sense for you: by redirecting (which is a bash/shell feature by the way), you can remap the file descriptors to an endpoint you chose. This can be useful for logging errors to a file, like in | ||
+ | find . 2>nopermissions.txt | ||
+ | or to pipe all output through a filter, as in | ||
+ | strace echo Hello World 2>&1 |less | ||
+ | the latter will redirect the standard error output into the pipe of the standard output and pipe the resulting combined output through less. | ||
+ | |||
+ | ==== redirect output and error ==== | ||
+ | ls -l 1>normal_out 2>error_out | ||
if you want to have both in one file | if you want to have both in one file | ||
ls -l 1>normal_out 2>&1 | ls -l 1>normal_out 2>&1 |
Revision as of 13:39, 10 April 2009
Contents |
Bash Handling
Job Control
Interactive
Key Combinations
I will not use the ^X notation of key combos since those are a bit misleading imo - after all, we are mostly using lower case letters, also people who modified the ^ or meta keys know how to substitute correctly. And of course the terminal type must be right.
ATTENTION: Bash's key combo mode can be set to editor-alike behaviour, specifically it can be set to behaving like emacs or to behaving like vi. You do this with:
set -o emacs
for the emacs mode, obviously, and
set -o vi
for the vi mode.
Most distributions of the GNU bash come with emacs mode preset.
For the purpose of readability, we will give the key kombinations as follows:
- Ctrl-a/A means: "Ctrl-a in emacs mode, "a" in vi mode"
- Ctrl-r/" means: "Ctrl-r in emacs mode, Ctrl-r in vi mode"
- Ctrl-o/- means: "Ctrl-o in emacs mode, nothing identical in vi mode (or not known to the author ;)"
A word about the vi mode
The vi mode is only "something like" the vi you know. Per default, you are in insert mode, you get to command mode (the mode where key combos will work) by pressing Esc. If you use Ctrl-c, you will be returned to a clear insert mode Prompt again. By pressing "i", you get back to the insert mode. Be aware that some vi key combos do not really behave alike in bash.
Searching in bash history
Of course you know that bash will pull lines from the history into the buffer with the up and down cursor keys.
Ctrl-r/" allows you to recall-by-typing from the bash history.
Executing in bash history
Ctrl-o/-(emacs) alone does nothing, but while pressed when on the line of a history entry (pulled with Ctrl - r or cursor keys), it will execute the bash history entry and when it exists, put the next command in history into the shell buffer.
Example:
philip@dinky:~$ vi test.c philip@dinky:~$ make test cc test.c -o test philip@dinky:~$ ./test Hello World! philip@dinky:~$ (press Ctrl-r vi here) (reverse-i-search)`vi': vi test.c (press enter here) philip@dinky:~$ vi test.c philip@dinky:~$ (press Ctrl-r mak here) (reverse-i-search)`mak': make test (press Ctrl-o here) philip@dinky:~$ make test cc test.c -o test philip@dinky:~$ ./test
Moving about the prompt/command line
There are some slight differences between the emacs and the vi versions, mostly about whats considered to be in the copy buffer, where the cursor offset will be etc.
- ctrl-a/0 move the cursor to the beginning of the bash prompt/line
- ctrl-e/A move the cursor to the end of line
- ctrl-w/- cut the word previous to the cursor (including intermittent whitespace)
- ctrl-u/" cut everything from line beginning to, but not including the cursor position
- ctrl-k/", /D cut everything from, and including the cursor position to end of line
- ctrl-_/u undo
- alt-b/- move the cursor back one word
- alt-f/- move the cursor forward one word
- ctrl-y/p paste the last thing cut
- -/h move cursor left
- -/l move cursor right
- -/a start inserting after current cursor position
- -/D cut everything from, and not including
- -/dd delete line (save it for pasting)
- ctrl-c/" reset input buffer (do not save for pasting)
- arrow-up/j move up through history
- arrow-down/k move down through history
Bash Programming
Bash is the most common shell form on linux, and most other unix systems.
Just a small reminder. Before you start writing a long bash script, think if you can do this in perl^W python.
Redirection
Standard Input, Output and Error
Digression for a better understanding of file descriptors
If you are on a system with a /proc filesystem (and may god have mercy on your soul if you are not), you will find that there are interesting symlinks in /proc/<pid>/fd. Let me elaborate.
Consider the following: you are logged on, have an interactive shell like bash, and you are the owner of your terminal (tty). Now, find out your PID:
philip@dinky:~$ w |head -3 15:15:04 up 72 days, 17 min, 35 users, load average: 0.15, 0.11, 0.09 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT philip tty2 - 11Feb09 58days 0.10s 0.00s /bin/login -- philip@dinky:~$ ps waux |grep bash |grep tty2 philip 22008 0.0 0.0 21396 1372 tty2 S Feb11 0:00 -bash philip@dinky:~$
now, with that information, let's have a peek into our filedescriptors:
philip@dinky:~$ ls -l /proc/22008/fd total 0 lrwx------ 1 philip philip 64 2009-04-10 15:16 0 -> /dev/tty2 lrwx------ 1 philip philip 64 2009-04-10 15:16 1 -> /dev/tty2 lrwx------ 1 philip philip 64 2009-04-10 15:16 2 -> /dev/tty2 lrwx------ 1 philip philip 64 2009-04-10 15:16 255 -> /dev/tty2 philip@dinky:~$
there are, of course, redirections to your current interactive shell. This is why your software will crash receiving a SIGPIPE signal when you close your terminal while something is running.
So, what do those numbers mean? If software allocates file descriptors (thats what it gets when it opens files for reading or writing) from the operating system, it will receive integer numbers. Thus, the PID plus the file descriptor will uniquely identify an open file system wide - a bit like an ip-address + port pair for a TCP connection.
0 is standard input 1 is standard output 2 is standard error everything else are filedescriptors received when the software opened other files, like configuration files, log files etc. 255 is just there to mess with my explanation.
The whole "standard X" metaphor is a bit to wrap ones head around, but after a while using them extensively and with deep understanding, one will be able to prevent a lot of shit from happening (processes dying under certain conditions, missing logfiles, mailbombs from cron etc) and finally lead a happy life.
The standard file descriptors are the fds where software will write and receive their user interaction by default. Yet, software does not necessarily need to do that; while libc provides for every running piece of software with those filedescriptors by standard, the software can close them willingly. Software can indeed decide to "unbind" itself from standard input and output by closing the respective file descriptor. Analog to that, the respective symlinks in the proc fs will disappear, and so will the depdendy on those files (like the interactive shell).
By using redirection, you can remap those standard file descriptors in your own way. Consider doing something like this on one terminal:
philip@dinky:~$ cat > /dev/null
and try finding the PID of this cat process, then do something like the following on another shell (without killing the cat of course):
philip@dinky:~$ ls /proc/10519/fd -l total 0 lrwx------ 1 philip philip 64 2009-04-10 15:31 0 -> /dev/pts/17 l-wx------ 1 philip philip 64 2009-04-10 15:31 1 -> /dev/null lrwx------ 1 philip philip 64 2009-04-10 15:31 2 -> /dev/pts/17
so, i hope now it starts making sense for you: by redirecting (which is a bash/shell feature by the way), you can remap the file descriptors to an endpoint you chose. This can be useful for logging errors to a file, like in
find . 2>nopermissions.txt
or to pipe all output through a filter, as in
strace echo Hello World 2>&1 |less
the latter will redirect the standard error output into the pipe of the standard output and pipe the resulting combined output through less.
redirect output and error
ls -l 1>normal_out 2>error_out
if you want to have both in one file
ls -l 1>normal_out 2>&1
Questions and Solutions
Below some examples for bash problems that might come up
Arrays in Bash:
foo[0]=1; foo[1]=2; foo[2]=3; # loop for (( i=0; i<${#foo[@]}; i++ )); do echo ${foo[$i]}; done; # more like a foreach for i in ${foo[@]}; do echo $i; done;
Variable Variables
foobar=5; bar="foobar"; echo $foobar; echo $bar; echo ${!bar};
Variable Variables in Arrays
foo[0]=1; foo[1]=2; foo[2]=3; foo_tcp[0]="AT"; foo_upd[0]="AU"; foo_tcp[1]="BT"; foo_upd[1]="BU"; foo_tcp[2]="CT"; foo_upd[2]="CU"; # loop for (( i=0; i<${#foo[@]}; i++ )); do echo ${foo[$i]}; for k in upd tcp; do data=foo_${k}[$i]; echo ${!data}; done; done;
More Documentation
The best advanced guide for bash scripting: http://www.tldp.org/LDP/abs/html/