Personal tools
User menu

Knowledge Base/Unix/General

From Thalesians

Jump to: navigation, search

Contents

A summary of some useful Linux/Unix commands

chown -R apache:apache /path/to/directory

Change ownership of /path/to/directory recursively to user apache, group apache (first user, then group).

tar czf /path/to/output.tar.gz /path/to/input/directory/

Create a *.tar.gz archive file from a directory.

du -a /path/to/directory | sort -n -r | head -n 10

List the top 10 files and directories by size in a directory tree. By default, the units are 1024 bytes (1 K).

mysqladmin -u root -p status

MySQL server status. You will be prompted for the password (-p).

mysql -u root -p
mysql> show databases

List the databases on a MySQL server. You will be prompted for the password (-p).

mysqldump -u root -p database_name > ~/path/to/output/database_name.sql

Backup a MySQL database database_name to a file. You will be promoted for a password (-p).

Finding out which platform / configuration you are using

Use

uname --all

This will "print certain system information". For example,

CYGWIN_NT-6.0-WOW64 PB-MacBookPro 1.5.25(0.156/4/2) 2008-06-12 19:34 i686 Cygwin

Finding out which shell you are using

Inspect the SHELL environment variable. The command

echo $SHELL

will display its value. Look at the last part of the pathname:

  • /bin/sh - Bourne shell
  • /bin/bash - Bourne Again SHell
  • /bin/csh - C shell
  • /bin/ksh - Korn shell
  • /bin/tcsh - TC shell
  • /bin/zsh - Z shell

Which shell is best for me?

The following has been reproduced from unix-faq/shell/shell-differences ver. 1.17:

Feature sh csh ksh bash tcsh zsh rc es
Job control N Y Y Y Y Y N N
Aliases N Y Y Y Y Y N N
Shell functions Y(1) N Y Y N Y Y Y
"Sensible" Input/Output redirection Y N Y Y N Y Y Y
Directory stack N Y Y Y Y Y F F
Command history N Y Y Y Y Y L L
Command line editing N N Y Y Y Y L L
Vi Command line editing N N Y Y Y(3) Y L L
Emacs Command line editing N N Y Y Y Y L L
Rebindable Command line editing N N N Y Y Y L L
User name look up N Y Y Y Y Y L L
Login/Logout watching N N N N Y Y F F
Filename completion N Y(1) Y Y Y Y L L
Username completion N Y(2) Y Y Y Y L L
Hostname completion N Y(2) Y Y Y Y L L
History completion N N N Y Y Y L L
Fully programmable Completion N N N N Y Y N N
Mh Mailbox completion N N N N(4) N(6) N(6) N N
Co Processes N N Y N N Y N N
Builtin artithmetic evaluation N Y Y Y Y Y N N
Can follow symbolic links invisibly N N Y Y Y Y N N
Periodic command execution N N N N Y Y N N
Custom Prompt (easily) N N Y Y Y Y Y Y
Sun Keyboard Hack N N N N N Y N N
Spelling Correction N N N N Y Y N N
Process Substitution N N N Y(2) N Y Y Y
Underlying Syntax sh csh sh sh csh sh rc rc
Freely Available N N N(5) Y Y Y Y Y
Checks Mailbox N Y Y Y Y Y F F
Tty Sanity Checking N N N N Y Y N N
Can cope with large argument lists Y N Y Y Y Y Y Y
Has non-interactive startup file N Y Y(7) Y(7) Y Y N N
Has non-login startup file N Y Y(7) Y Y Y N N
Can avoid user startup files N Y N Y N Y Y Y
Can specify startup file N N Y Y N N N N
Low level command redefinition N N N N N N N Y
Has anonymous functions N N N N N N Y Y
List Variables N Y Y N Y Y Y Y
Full signal trap handling Y N Y Y N Y Y Y
File no clobber ability N Y Y Y Y Y N F
Local variables N N Y Y N Y Y Y
Lexically scoped variables N N N N N N N Y
Exceptions N N N N N N N Y

Key to the table above

  • Y - Feature can be done using this shell.
  • N - Feature is not present in the shell.
  • F - Feature can only be done by using the shells function mechanism.
  • L - The readline library must be linked into the shell to enable this Feature.

Notes to the table above

  1. This feature was not in the orginal version, but has since become almost standard.
  2. This feature is fairly new and so is often not found on many versions of the shell, it is gradually making its way into standard distribution.
  3. The Vi emulation of this shell is thought by many to be incomplete.
  4. This feature is not standard but unoffical patches exist to perform this.
  5. A version called 'pdksh' is freely available, but does not have the full functionality of the AT&T version.
  6. This can be done via the shells programmable completion mechanism.
  7. Only by specifing a file via the ENV environment variable.

Customising your working environment

When you log into the system, it runs the system profile file, /etc/profile. Usually this file is readable by all users, writable exclusively by the administrator.

After the system profile, the user profile file(s) is/are run from the user's home directory. These are shell-specific. They are essentially shell scripts that you can edit to customise your environment, e.g. set the environment variables. Here is a list of user profile scripts for different shells:

  • Bourne shell: .profile
  • Bourne Again SHell: .bash_profile
  • C shell: .login and .cshrc
  • Korn shell: .profile
  • TC shell: .login, .tcshrc and .cshrc
  • Z shell: .zlogin and .zshrc

What's the difference between the multiple user profile files?

.cshrc (.tcshrc, .zshrc) is executed when the user first logs into the system and each time a sub shell is spawned (a shell script is executed).

TC shell looks for .tcshrc first. If one is not found, it looks for and executes .cshrc.

.login (.zlogin) is executed when the user logs into the system immediately after .cshrc (.tcshrc, .zshrc). However, it is not executed when a sub shell is spawned.

.login (.zlogin) should contain commands which only need to happen at login time, not for each shell invocation. Thus it would be a bad idea to redefine your path variable in .login (.zlogin) and not in .cshrc (.tcshrc, .zshrc) because sub shells (including X Windows other than your initial login window) would not be aware of this new path.

In general, you will customise .cshrc for C shell, .tcshrc for TC shell, .zshrc for Z shell.

Notes

  1. C shell and TC shell will also run /etc/csh.cshrc, /etc/csh.login at login and /etc/csh.logout. These are likewise global (affect all users) and are edited by the administrator.
  2. Such "dot"-files are not normally displayed by ls. To view them, you should use
ls -la

Finding out which groups a given user belongs to

The command is groups [ user ... ]. To discover your own group membership, simply enter

groups

Checking how much space is occupied by a given directory tree

A good command to know is

du -k .

This output something like the following:

9940    ./2008.08.27/MarketDepth
9944    ./2008.08.27
9804    ./2008.08.28/MarketDepth
9808    ./2008.08.28
8452    ./2008.08.29/MarketDepth
8456    ./2008.08.29
8       ./2008.08.30/MarketDepth
12      ./2008.08.30
964     ./2008.08.31/MarketDepth
968     ./2008.08.31
58356   .

Where 58356 is the total size of the directory tree ".".

The parameter -k tells du to use the unit of 1 kilobyte (1024 byte) in its output. A useful alternative is -h which tells it to print sizes in human readable format (e.g., 1K, 234M, 2G).

Finally, if your directory tree is very deep, it may be worth specifying --max-depth=N. This will print the total for a directory only if it is N or fewer levels below the command line argument.

The following form is particularly useful:

du -h --max-depth=1 .

This will print something like

279G    ./Dir1
30G     ./SomeOtherDir
69G     ./Foo
624G    ./Bar

Giving the totals for the first-level subdirectories.

Checking how much free space is available in a given directory (on a given mount)

For the current directory, use

df -b .

This should give you something like

Filesystem              avail
somesys123:/vol/apps_123/blahblahblah/user 319407932

Where 319407932 is the number of bytes free.

Creating a symbolic link

Somehow the syntax for this command is easily forgotten. It is:

ln -s [TARGET DIRECTORY OR FILE] ./[SHORTCUT]

For example:

ln -s /usr/local/apache/logs ./logs

Checking if a process is listening on a specific port

E.g. if you are worried about port 19010, you can do the following:

netstat -ltnup | grep 19010

You may see something like this:

tcp        0      0 0.0.0.0:19010               0.0.0.0:*                   LISTEN      5060/q

Here 5060 is the process ID.

Listing the contents of a *.tar or *.tar.gz file

tar tvf filename.tar

Extracting a *.tar or *.tar.gz file

gzip -d filename.tar.gz
tar xvf filename.tar

Using sort to make sense of the output of other commands, such as ps and ls

sort does exactly what it is supposed to do: sorts. Consider this example:

ps aml | sort -r -k 7 | more

Here the "process status" command (ps) lists all processes with memory information in long format, and standard output from this command is piped into sort, where it is reverse sorted (-r) on the seventh column (-k 7), and then is "piped" into more for easy viewing by the user.

Along the same lines,

ls -al | sort -g -r -k 5 | head -10

will report the 10 largest files, sorted by size, in your current dirrectory. (-g tells it to use the general numeric sort rather than the dictionary order sort.)

How can I find out the user ID of the currently logged in user? Is there a difference between who am i and whoami?

The answer to the first part of the question follows from the question itself: use who am i or whoami.

The answer to the second part of the question is, yes:

who am i

displays

bilokon  pts/2        Jan 26 14:42 (foo.bar.baz.com)

whereas

whoami

displays simply

bilokon

The latter may be more useful in backticks:

echo I am `whoami`

displays

I am bilokon

I want to create a directory foo/bar/baz but I don't know whether foo and foo/bar exist

Use

mkdir -p foo/bar/baz

This will create foo, foo/bar, and then foo/bar/baz and do nothing (without complaining) if they already exist.

Obtaining shell script's full path

Can a shell script determine the full path to itself? Yes, as follows:

#!/bin/csh

echo "My path:"
echo "--------"

echo `readlink -f $0`

Or, if you want to save this path,

set DIR_THIS_SCRIPT=`readlink -f $0`
echo $DIR_THIS_SCRIPT

What if we want the path to the containing directory?

set PATH_THIS_SCRIPT=`readlink -f $0`
echo $PATH_THIS_SCRIPT

set PATH_THIS_SCRIPT_DIR=`dirname $PATH_THIS_SCRIPT`
echo $PATH_THIS_SCRIPT_DIR

Finding all files in a given directory subtree that match a specified pattern

You can use something like this:

find -name "*.gz"

If you are in /foo, this will look inside /foo, /foo/bar, /foo/bar/baz, etc.

If you want to count the files, use

find -name "*.gz" | wc -l

Listing directory contents one file per line

ls -1

Finding out how much memory there is on a machine

vmstat -s -S k

This will display something like

     16807600 k total memory
     16743662 k used memory
      1967300 k active memory
      8966222 k inactive memory
        63938 k free memory
       117428 k buffer memory
      8899985 k swap cache
     18227212 k total swap
          212 k used swap
     18227000 k free swap
     11707612 non-nice user cpu ticks
         2363 nice user cpu ticks
     20877755 system cpu ticks
   6209150782 idle cpu ticks
      6218002 IO-wait cpu ticks
       136901 IRQ cpu ticks
       797432 softirq cpu ticks
     20214961 pages paged in
    987134283 pages paged out
            0 pages swapped in
           52 pages swapped out
   1313489012 interrupts
   3283339599 CPU context switches
   1217830213 boot time
     99998296 forks

where the units are 1000's (NOT 1024's!) or bytes.

Finding out how much memory is used by a process

Assuming that you know the PID (process ID), which you can obtain by running ps aux, you can use

pmap 12570

where you should replace 12570 with your process ID.

Finding all files whose name matches a particular pattern and which contain a given text string

You can use the following:

find . -name "*.sh" -exec grep -q "FOO" '{}' \; -print

This will list the paths of all text files whose name matches "*.sh" and which contain the string "FOO". The output of the command looks as follows:

./file1.sh
./file2.sh
./somedir/yet_another_file.sh

It's obvious from the presence of somedir in the output above that this command will process the current directory and all its subdirectories. If you want to process the current directory alone, use

find . -maxdepth 1 -name "*.sh" -exec grep -q "FOO" '{}' \; -print

If you don't care about the file name but only care about the requirement that the file contain the string "FOO" you should omit -name "*.sh":

find . -exec grep -q "FOO" '{}' \; -print

Finding files that contain both pattern 1 and pattern 2

If your pattern 1 is "FOO" and your pattern 2 is "BAR", then you can use this:

grep -l FOO *.sh | xargs grep -l BAR

Inspecting open files

Note: lsof may be under /usr/sbin, so you may have to invoke it as /usr/sbin/lsof.

Listing all open files

lsof

Listing files open for a particular user

lsof -u somelogin

where somelogin is the user's login.

Listing the users that have a particular file open

lsof /path/to/file

Nailing a process to a particular CPU

There is a useful command called taskset which retrieves or sets a process's CPU affinity.

CPU affinity is a scheduler property that "bonds" a process to a given set of CPUs on the system. The Linux scheduler will honour the given CPU affinity and the process will not run on any other CPUs. Note that the Linux scheduler also supports natural CPU affinity: the scheduler attempts to keep processes on the same CPU as long as practical for performance reasons.

The CPU affinity is represented as a bitmask, with the lowest order bit corresponding to the first logical CPU and the highest order bit corresponding to the last logical CPU. Not all CPUs may exist on a given system but a mask may specify more CPUs than are present. For example,

  • 0x00000001 is processor #0
  • 0x00000003 is processors #0 and #1
  • 0xFFFFFFFF is all processors (#0 through #31)

To find out a process's (say PID 4211) current affinity mask:

taskset -p 4211

This will return something like

pid 4211's current affinity mask: f

(Meaning none has been set.)

To set it:

taskset -p 0x00000003 4211

You can also use taskset with a mask to launch a new command:

taskset 0x00000003 du -k

You should understand that although taskset will nail your process to particular cores it doesn't do anything to block other tasks from using those cores.

You may want to look at all the heavy-hitter processes on the box and taskset them appropriately too.

Learning more about a particular process by its PID

First, you can discover the PID of a pricess by running something like

ps aux

or

top

Once you have the PID, it may be a good idea to explore the /proc hierarchy. It is a veritual file system, sometimes referred to as a process information pseudo-file system. It doesn't contain "real" files but runtime information. Many system utilities are simply calls to files in this directory. E.g. lsmod is the same as cat /proc/modules.

Anyway, assuming your PID is 4211,

cd /proc/4211
ls

You will see a number of entries:

attr     cwd      fd        mem         numa_maps  stat    task
auxv     environ  loginuid  mounts      root       statm   wchan
cmdline  exe      maps      mountstats  smaps      status

The most important of these are:

  • /proc/PID/cmdline — command line arguments.
  • /proc/PID/cwd — link to the current working directory.
  • /proc/PID/environ — values of environment variables.
  • /proc/PID/exe — link to the executable of this process.
  • /proc/PID/fd — directory, which contains all file descriptors.
  • /proc/PID/mem — memory held by this process.
  • /proc/PID/status — process status in human readable form.

So you can simply

cat /proc/4211/status

And you will see something like this:

Name:   runRemoteScript
State:  R (running)
SleepAVG:       0%
Tgid:   4211
Pid:    4211
PPid:   1
TracerPid:      0
Uid:    59873   59873   59873   59873
Gid:    2198    2198    2198    2198
FDSize: 256
Groups: 2198
VmSize:    19484 kB
VmLck:         0 kB
VmRSS:     11228 kB
VmData:    12244 kB
VmStk:        28 kB
VmExe:       768 kB
VmLib:     10684 kB
StaBrk: 08112000 kB
Brk:    088e1000 kB
StaStk: ffffd620 kB
Threads:        1
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000000006
SigCgt: 0000000000000000
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000

Basic user and group management

To list all users:

cat /etc/passwd

You will see "proper" users, like

paul:x:500:10::/home/paul:/bin/bash
saeed:x:501:10::/home/saeed:/bin/bash
thomas:x:502:10::/home/thomas:/bin/bash

but also "system" users:

apache:x:48:48:Apache:/var/www:/sbin/nologin
mysql:x:27:27:MySQL Server:/var/lib/mysql:/bin/bash

To list all groups:

groups

You will see something like

wheel thalesians_team

Create a new user:

sudo /usr/sbin/adduser nikolai

We have just created a user named nikolai.

Set the user's password:

sudo passwd nikolai

Adding an (existing) user to an (existing) group:

sudo /usr/sbin/usermod -a -G thalesians_team nikolai

The user nikolai has been added to the group named thalesians_team.

To list the user's groups:

groups nikolai

You may see something like this:

nikolai : nikolai thalesians_team

Notice that by default nikolai's primary group is nikolai

To change the user's primary group:

sudo /usr/sbin/usermod -g thalesians_team

Now if you issue the command

groups nikolai

you will see

nikolai : thalesians_team

The first group listed is the primary group.

To delete a user:

sudo /usr/sbin/userdel -r nikolai

-r tells userdel to remove the home directory and mail spool, too.

Replacing a string in a binary file

If you need to replace pattern with its replacement, whether in a text or binary file, you can use perl:

perl -pi -e 's/pattern/replacement/g' file1

Output readability tricks

We shall list information about the files in the current directory, sorting them by modification time. This is done by

ls -t

Chances are this has produced too much output if there are too many files. We may need to filter the results.

If we want to display the first 10 results, we apply

ls -t | head -n 10

This will give us the 10 files that were modified most recently.

If we want to display the last 10 results, we apply

ls -t | tail -n 10

This will give us the 10 files that were modified least recently.

Returning the context of grep's match

Suppose that we have a file example.txt:

Monday
Tuesday
Wednesday
Thursday
Friday
Saturday
Sunday

Let's see what happens if we grep it for Thu:

$ grep Thu example.txt
Thursday

We have found Thursday, but this may not be enough for us. Suppose that we want to see the context too, i.e. a line or two that surround the matching line.

$ grep -C 1 Thu example.txt
Wednesday
Thursday
Friday

This has returned the matching line as well as the one before and the one that follows. Similarly

$ grep -C 2 Thu example.txt
Tuesday
Wednesday
Thursday
Friday
Saturday

What does the output look like if we are processing multiple files? In order to test this, we create a copy of example.txt:

$ cp example.txt example_copy.txt

Then

$ grep -C 1 example*.txt

produces this output:

example_copy.txt-Wednesday
example_copy.txt:Thursday
example_copy.txt-Friday
--
example.txt-Wednesday
example.txt:Thursday
example.txt-Friday

As you can see, the filename appears on each line and a line containing "--" is placed between contiguous groups of matches.

Manipulating the output of Unix commands

Suppose you would like to add up the sizes of several directories:

$ du -k ./2009.02.26/*

which produces the output

12      ./2009.02.26/Foo
12      ./2009.02.26/Bar
8       ./2009.02.26/Baz

We need to add up the numbers in the first column. But first, how do we get the first column? We can use cut:

$ du -k ./2009.02.26/* | cut -f 1

which gives

12
12
8

We can now use awk (a simple pattern scanning and processing language) to add these up:

$ du -k ./2009.02.26/* | cut -f 1 | awk '{a+=$1}END{print a}'

This prints

32

su, sudo and sudoers

su enables users to run a shell with substitute user and group IDs. In other words, it enables the users to work as someone else:

su

and I magically become root (but I'll have to enter the root's password).

su -l matthew

And I start working as matthew:

[paul@Thalesians ~]$ su -l matthew
Password:
[matthew@Thalesians ~]$

until I do

[matthew@Thalesians ~]$ exit
[paul@Thalesians ~]$

sudo also offers a lot of power. I can remain paul while issuing root commands. All I need to do is prefix them with sudo.

E.g. I would need to be logged in as a root to do this:

/usr/sbin/userdel -r nikolai

However, I could remain paul and do

sudo /usr/sbin/userdel -r nikolai

But that's provided that I'm allowed, i.e. that I'm a sudoer. Not everyone is a sudoer. Also, some sudoers will have to enter the root's password anyway, others won't. This is all determined by the sudoers file, which you can edit safely with

sudo /usr/sbin/visudo

Adding a password to Apache

sudo htpasswd -b /etc/httpd/passwd/passwords jayson xxxxxxxx

You will see

Adding password for user jayson

Killing a process

There is more than one way to kill a process.

Some users are accustomed to

kill -9 pid

where pid is the process ID. However, this may not be a great idea. Here is why.

kill sends a signal to a process. In *nix there are many different types of signals that are sent to processes. They serve various pusposes. The process may choose to ignore the signal or trap it and execute a signal handler. Untrapped signals have a default action which may be "do nothing" or "exit". There are two signals that are untrappable: SIGKILL and SIGSTOP.

You can examine the list of all the signals by issuing

kill -l

This will show you something like

HUP INT QUIT ILL TRAP ABRT BUS FPE KILL USR1 SEGV USR2 PIPE ALRM TERM STKFLT CHLD CONT STOP TSTP TTIN
TTOU URG XCPU XFSZ VTALRM PROF WINCH POLL PWR SYS RTMIN RTMIN+1 RTMIN+2 RTMIN+3 RTMAX-3 RTMAX-2 RTMAX-1
RTMAX

The most commonly used signals are:

  • SIGTERM (15, terminate)
  • SIGINT (2, interrupt)
  • SIGKILL (9, kill &mdash untrappable!)

Most programs perform various cleanup steps on exit (e.g. delete temporary files, shut down sockets, remove shared memory segments, close open files). They typically trap SIGTERM and SIGINT to do their cleanup duties. However, they cannot trap SIGKILL.

kill -9 on MySQL will corrupt your database. Similarly, your sockets probably won't be properly shut down.

To avoid data corruption, you should follow the following routine to shut down a process:

Check that the process is running and get its PID:

ps aux | grep processname

where processname is the name of the process.

kill pid

where pid is will send SIGTERM. Wait 5 seconds.

kill pid

try again. Wait 5 seconds.

kill -INT pid

will send SIGINT. Wait 5 seconds.

kill -INT pid

try again. Wait 5 seconds.

kill -KILL pid

(this is the same as kill -9 pid). If this doesn't work, something is very, very wrong. You may try

kill -KILL pid

again.

Acknowledgement: This advice is based on http://speculation.org/garrick/kill-9.html by Garrick Staples.

Colours in the shell prompt

It is possible to use colours in your shell prompt.

Here is a quick recipe. For more information, please see http://tldp.org/HOWTO/Bash-Prompt-HOWTO/x405.html and http://www.cyberciti.biz/faq/bash-shell-change-the-color-of-my-shell-prompt-under-linux-or-unix/

echo "* Configuring shell prompt"

BLUE=`tput setf 1`
GREEN=`tput setf 2`
CYAN=`tput setf 3`
RED=`tput setf 4`
MAGENTA=`tput setf 5`
YELLOW=`tput setf 6`
WHITE=`tput setf 7`

PS1="\[$MAGENTA\]\@ \[$RED\]\u@\h \[$CYAN\]\w \[$YELLOW\]\! \[$WHITE\]$ "

Your shell prompt will look like this:

  • 09:36 AM paul@host01 ~/dev 963 $ emacs
  • This page was last modified on 12 June 2010, at 12:21.
  • This page has been accessed 71,875 times.