Friday, May 30, 2014

Linux: Compilation of 31 Bash tips and tricks - Part 2 (16-31)

The next part of tips.

These are picked from various sources and usually are helpful to me in my day to day work, so you do not have to read the man pages everytime. I do not remember all the sources, so I will quote the source wherever I remember. If you know the source, please let me know.

I know I have written it in a very clumsy way, without too much explaining of the underlying context and theory or any references. But I hope to make it beginner friendly.

Here we go:

16. [Bash usage tip/security]
Enable a 15 minute timeout for bash. Helps in security best practices.

Let's say you want to auto-logout of your bash shell after 15 minutes of inactivity. This is sometimes an important security requirement as well. You can set this code in the global /etc/profile or for specific user in ~/.bash_profile. This piece essentially creates a readonly environment variable when a user logs in

#Add this in /etc/profile, tested in SUSE
readonly TMOUT
export TMOUT

17. [Bash usage tip/security]
Disable command execution in Less.

Well if you don't know this, you can execute commands in less, vi etc
To disable this in less, you need to set an environment variable called LESSSECURE.


18. [Bash usage tip/security]
Executing bash commands in vi, less, and more:

in vi -> :!bash
in less -> !bash
in more -> !bash

19. [Bash usage tip/security]
Setting an environment variable as READONLY:

readonly TMOUT
export TMOUT

20. [Bash usage tip/security]
Disable bash builtins using enable. This might help if you are trying something like a restricted shell. I must warn you, its risky.

enable -n <builtin_name>

21. [Bash usage tip]
Useful commands in vi

:set list -special chars
:set nu -line numbers
:.! ls - Add a . before ! during command execution and it will dump the output in the current screen.
:r! <cmd> -same thing, dump the cmd output
:%!xxd - Turn vim into a hexeditor, :%xxd -r to reverse.
q: -command history
:%TOhtml -create an html file body

22. [Shell scripting tip]
Command execution in a subshell. Shell scripting tip.

$(command) is the same as `command`

$(ls) gives you the output of ls
so does `ls`.

23. [Bash usage tip]
env and export -p

Use the env (or export -p) command to see only those variables that have been exported and would be available to a subshell.

24. [Bash usage tip ]
set command:

Use the set command to see the value of all variables and function definitions in the current shell. The list produced by env is a subset of the list produced by set, since not all variables are exported.

25. [Shell scripting tip]
Looping over vars with spaces like "My Folder".

for file in "$@"
    chmod 0750 "$file"

26. [Shell scripting tip]
Difference between: "$*" and "$@".

for file in "$*" will expand to:
for file in "file1 file2 file3 My File.txt"

The above will not help if the filename has spaces, like My File.txt, bash would treat it as two files, My and File.txt, and thereafter producing an error like My not found.

for file in "$@"
will expand to:
for file in "file1" "file2" "file3" "My File.txt"

27. [Shell scripting tip]
Number of args can be accessed by ${#}. 

28. [Shell scripting tip]
 Quick sed handy examples, when I read those examples I recall the logic, otherwise the theory confuses me.

Replace password hash in shadow file, if you use -i it will replace in the original file, so be careful:

sed -e '/^user:/s/:[^:]*:/:newpassword:/' /etc/shadow

Change the param value to 3 in sshd_config file

sed -i "s/\(\#MaxAuthTries.*\)/MaxAuthTries 3/g" /etc/ssh/sshd_config

Replace all digits

sed -e 's/[[:digit:]]//g'

Replace all other than digits (Use ^ to negate)

sed -e 's/[^[:digit:]]//g'
Replace all alpha-numeric

sed -e 's/[[:alnum:]]//g'
Replace all other than alphanumeric (special chars)

sed -e 's/[^[:alnum:]]//g'
29. [Performance monitoring tip]
 Listing Apache httpd processes and threads.

List httpd processes:
ps -elf | grep httpd

List httpd worker threads:
ps -elfT | grep httpd

30. [Shell scripting tip]
Using readlink and dirname in shell scripts to get absolute path and directory name.

If you want to read the absolute path for a file use:
readlink -f
$ readlink -f ./file.txt would return

For only the directory:
dirname /etc/passwd returns /etc
31. [ Bash usage tip]
Use CTRL-R to go through the history of commands.

1. Ctrl-R and then type command, it gives the most recent one. Press Ctrl -R more times.
2. Exit anytime using Ctrl-C
3. Edit using arrow keys

Thursday, May 22, 2014

Linux: Compilation of 31 Bash tips and tricks - Part 1 (1-15)

I thought I will start capturing all the personal favorite/useful/bombastic/flamboyant  tips that I use frequently and that I forget regularly. Basically if I have to revise all my bash tricks, I would quickly walk over these tips that I collected over a period of time. BTW, I am adding the tips in parts, and I have added part 2 here:

These are picked from various sources and usually are helpful to me in my day to day work, so you do not have to read the man pages everytime. I do not remember all the sources, so I will quote the source wherever I remember. If you know the source, please let me know.

I know I have written it in a very clumsy way, without too much explaining of the underlying context and theory or any references. But I hope to make it beginner friendly.

Differences between bash and sh:

Bash documentation home:

Here we go:

1. [Bash usage tip]
Text navigation shortcuts (to make you look like a pro).
These shortcuts are pretty handy and save a lot of your time when you have remembered them. In the beginning I struggled, but later after some practice I find them very easy to use.:

Ctrl - A --- Start
Ctrl - E ---- End
Ctrl - U ---- Cut before the cursor
Ctrl - K ---- Cut after the cursor
Ctrl - Y ---- Paste
Ctrl - T ---- Swap chars before cursor
Ctrl - W ---- Delete word left top the cursor
Ctrl - L ---- Clean the screen
Esc- f/Esc - Right arrow ---- Jump 1 word fwd
Esc-b/Esc - Left arrow ---- Jump 1 word backward

2. [Bash usage tip]
Delete Control M or crlf chars in a text file transferred from windows.
 So basically when you transfer text files to and from a *Nix machine. The transfer tool auto-detects that it is a text file and performs an EOL conversion. However, this does not happen 'automatically' if you have explicitly set the transfer mode to "Binary", or your text files are inside a binary file like zip, or tar.gz.:

When you try to execute a shell script having CRLF chars, you get an error of sort:
# ./
-bash: ./ /bin/sh^M: bad interpreter: No such file or directory
You can remove them by the simple use of sed. However, the trick is to type in Ctrl-M character.
sed -i 's/^M//' <filename>

Windows uses CR-LF (carriage and return) for line endings, while *nix uses only return (LF). Type Ctrl - m like this:

Ctrl -V then Ctrl M.

 Print/check for Ctrl M chars in a file using cat:

cat -v <filename>
# cat -v
echo "Hello world!"^M

3.[Bash usage tip]
 Quickly setting date and time:

date -s "8 DEC 2013 18:30:00"
Errors: date: invalid date"
4. [Bash usage tip]
 Size of a directory:

du -sh /root
       17G /root

5. [Bash usage tip]
 View ports tcp (t),udp (u) and  LISTENing (l), along with their corresponding processes (p) and use numbers (n)  (netstat hyphen TOO-LP-N):

netstat -tulpn

6. [Bash usage/Shell scripting tip]
 Cut a field correctly, by use of translate and squeez (tr) to squeez the tab/space formatting. e.g. the following returns the pid.
tr for translate and cut are very important tools for parsing a command line output. The -s option of tr followed by the whitespace character " ", squeezes the whitespace characters (including tabs) and reduces its occurrence to a single whitespace. If we do not use tr, then cut will have some problems identifying the correct field due to multiple occurrence of spaces and tabs.:
ps -ef | grep -i | tr -s " " | cut -d" " -f2

7. [Bash usage/Shell scripting tip]
 Redirect output to a file and to standard output at the same time using tee:
 You wanted to save the output of netstat in a file using redirection operator '>' but at the same time wanted to see it on the screen. Use tee and |
netstat -tnlp | tee aaa.txt

8. [Shell scripting tip]
 Set -e file to exit upon error (useful in shell scripts):
 This is quite useful if you have a shell script which has commands that depend on the success of the previous command. For e.g. login to ssh and read a remote file. Using set -e, would make sure that the script exits execution if any of the commands return an error.

#Exit immediately if you see an error.
set -e

9. [Shell scripting tip]
 Set -x to see debug output (useful in shell scripts):

#Prints a lot of debugging output
set -x

10. [Bash usage tip]
 Use screen to detach, reattach or share the terminal: 
This will help you to run a command that runs overnight, disconnect the remote session and go home. Then come back later next day to re attach to the screen and see how it went.

screen (to simply start a screen, see help for detailed options)
Ctrl -D to detach from the screen
screen -r to re attach
screen -x to attach to an existing screen.
If you are unable to locate screen in your linux, perhaps you need to install it, which isnt very difficult.
11. [Bash usage tip]
 Install open source xming from sourcefourge to setup XWindows display:

You need this when you are running a program that requires a GUI window to be displayed, but if the display variable is not set correctly it fails to start the GUI screen.
For e.g. when you run the weblogic patch utility through putty or a remote terminal. You will get an error of sort:
"No X11 DISPLAY variable was set, but this program performed an operation which requires it."

I should write a separate article on how to setup Xming and display correctly with putty. I know I struggled a lot for the first time. :/
Here is some rough information on how it works: What basically happens is that when you install and start Xming on your windows box, it starts an X11 server which listens for incoming X11 information. Then on your remote linux prompt you set up the DISPLAY information to point to your windows box ip. After that when you start a GUI based program, the X11/GUI information is thrown to the ip set in DISPLAY and the listening server on your windows grabs it and displays the GUI to you.

And BTW, you can also avoid this problem by directly logging into the Desktop environment (if installed) in your linux machine through the console.

12. [Bash/Linux usage tip]
 Setup a chrooted ssh sftp account. Yes, you can do it! (Tested on Suse) 

Add a user with a home directory:

useradd -d /home/bobuser -m bobuser

#Sftp/chroot Settings for bobuser in /etc/ssh/sshd_config
#Change LogLevel to debug and check errors (if any) in /var/log/messages
Subsystem sftp internal-sftp

#Sftp/chroot Settings for bobuser
Match User bobuser
   X11Forwarding no
   AllowTcpForwarding no
   ForceCommand internal-sftp
   ChrootDirectory /home/bobuser
Now restart the ssh service. And try connecting.

r00ter127:~ # service sshd restart
Shutting down SSH daemon done
Starting SSH daemon done
r00ter127:~ # sftp bobuser@localhost
Connecting to localhost...
Read from remote host localhost: Connection reset by peer
Couldn't read packet: Connection reset by peer
Ouch..We need to read the errors in /var/log/messages, we had already set it to debug level. There are some requirements expected by the ssh daemon

Jan 25 11:30:27 r00ter127 sshd[10220]: debug1: PAM: establishing credentials
Jan 25 11:30:27 r00ter127 sshd[10220]: fatal: bad ownership or modes for chroot directory "/home/bobuser"
Set the ownership of the home and parent directories to root. That's a requirement. chown root:root /home/bobuser

r00ter127:~ # sftp bobuser@localhost
Connecting to localhost...
subsystem request failed on channel 0
Couldn't read packet: Connection reset by peer
If you get the above error, then it means there is some problem invoking the sftp server. And the ssh logs are not very helpful in this regard. Make sure you are using the internal-sftp:

Subsystem sftp internal-sftp
   ForceCommand internal-sftp
And then.. you are done.

r00ter127:~ # sftp bobuser@localhost
Connecting to localhost...
sftp> pwd
Remote working directory: /

13. [Bash usage/Security tip]
 Audacious use of history to read a file, e.g. read the /etc/passwd file using history:

history -r /etc/passwd

14.[Bash usage tip]
 Use 'which' and 'type' to differentiate if a command is a binary command or a shell builtin.:

which history
type history

15. [Bash usage/Security tip]
What is the hashing algorithm used in my /etc/shadow:
Well, this could be useful if someone asks you whats the hashing algorithm being used to secure the OS passwords. Higher the number, more secure the algorithm. This tip is incomplete actually. You must also know what algorithms are supported by your Linux distro, and how to change the algo to a stronger one. You will also have to change the passwords so that they are hashed with the new algorithm.

$1 -> md5
$2a -> Blowfish
$5 -> Sha256
$6 -> Sha512

Go to part 2: