Thursday, May 22, 2014

Linux: Compilation of 31 Bash tips and tricks - Part 1 (1-15)

I thought I will start capturing all the personal favorite/useful/bombastic/flamboyant  tips that I use frequently and that I forget regularly. Basically if I have to revise all my bash tricks, I would quickly walk over these tips that I collected over a period of time. BTW, I am adding the tips in parts, and I have added part 2 here:

These are picked from various sources and usually are helpful to me in my day to day work, so you do not have to read the man pages everytime. I do not remember all the sources, so I will quote the source wherever I remember. If you know the source, please let me know.

I know I have written it in a very clumsy way, without too much explaining of the underlying context and theory or any references. But I hope to make it beginner friendly.

Differences between bash and sh:

Bash documentation home:

Here we go:

1. [Bash usage tip]
Text navigation shortcuts (to make you look like a pro).
These shortcuts are pretty handy and save a lot of your time when you have remembered them. In the beginning I struggled, but later after some practice I find them very easy to use.:

Ctrl - A --- Start
Ctrl - E ---- End
Ctrl - U ---- Cut before the cursor
Ctrl - K ---- Cut after the cursor
Ctrl - Y ---- Paste
Ctrl - T ---- Swap chars before cursor
Ctrl - W ---- Delete word left top the cursor
Ctrl - L ---- Clean the screen
Esc- f/Esc - Right arrow ---- Jump 1 word fwd
Esc-b/Esc - Left arrow ---- Jump 1 word backward

2. [Bash usage tip]
Delete Control M or crlf chars in a text file transferred from windows.
 So basically when you transfer text files to and from a *Nix machine. The transfer tool auto-detects that it is a text file and performs an EOL conversion. However, this does not happen 'automatically' if you have explicitly set the transfer mode to "Binary", or your text files are inside a binary file like zip, or tar.gz.:

When you try to execute a shell script having CRLF chars, you get an error of sort:
# ./
-bash: ./ /bin/sh^M: bad interpreter: No such file or directory
You can remove them by the simple use of sed. However, the trick is to type in Ctrl-M character.
sed -i 's/^M//' <filename>

Windows uses CR-LF (carriage and return) for line endings, while *nix uses only return (LF). Type Ctrl - m like this:

Ctrl -V then Ctrl M.

 Print/check for Ctrl M chars in a file using cat:

cat -v <filename>
# cat -v
echo "Hello world!"^M

3.[Bash usage tip]
 Quickly setting date and time:

date -s "8 DEC 2013 18:30:00"
Errors: date: invalid date"
4. [Bash usage tip]
 Size of a directory:

du -sh /root
       17G /root

5. [Bash usage tip]
 View ports tcp (t),udp (u) and  LISTENing (l), along with their corresponding processes (p) and use numbers (n)  (netstat hyphen TOO-LP-N):

netstat -tulpn

6. [Bash usage/Shell scripting tip]
 Cut a field correctly, by use of translate and squeez (tr) to squeez the tab/space formatting. e.g. the following returns the pid.
tr for translate and cut are very important tools for parsing a command line output. The -s option of tr followed by the whitespace character " ", squeezes the whitespace characters (including tabs) and reduces its occurrence to a single whitespace. If we do not use tr, then cut will have some problems identifying the correct field due to multiple occurrence of spaces and tabs.:
ps -ef | grep -i | tr -s " " | cut -d" " -f2

7. [Bash usage/Shell scripting tip]
 Redirect output to a file and to standard output at the same time using tee:
 You wanted to save the output of netstat in a file using redirection operator '>' but at the same time wanted to see it on the screen. Use tee and |
netstat -tnlp | tee aaa.txt

8. [Shell scripting tip]
 Set -e file to exit upon error (useful in shell scripts):
 This is quite useful if you have a shell script which has commands that depend on the success of the previous command. For e.g. login to ssh and read a remote file. Using set -e, would make sure that the script exits execution if any of the commands return an error.

#Exit immediately if you see an error.
set -e

9. [Shell scripting tip]
 Set -x to see debug output (useful in shell scripts):

#Prints a lot of debugging output
set -x

10. [Bash usage tip]
 Use screen to detach, reattach or share the terminal: 
This will help you to run a command that runs overnight, disconnect the remote session and go home. Then come back later next day to re attach to the screen and see how it went.

screen (to simply start a screen, see help for detailed options)
Ctrl -D to detach from the screen
screen -r to re attach
screen -x to attach to an existing screen.
If you are unable to locate screen in your linux, perhaps you need to install it, which isnt very difficult.
11. [Bash usage tip]
 Install open source xming from sourcefourge to setup XWindows display:

You need this when you are running a program that requires a GUI window to be displayed, but if the display variable is not set correctly it fails to start the GUI screen.
For e.g. when you run the weblogic patch utility through putty or a remote terminal. You will get an error of sort:
"No X11 DISPLAY variable was set, but this program performed an operation which requires it."

I should write a separate article on how to setup Xming and display correctly with putty. I know I struggled a lot for the first time. :/
Here is some rough information on how it works: What basically happens is that when you install and start Xming on your windows box, it starts an X11 server which listens for incoming X11 information. Then on your remote linux prompt you set up the DISPLAY information to point to your windows box ip. After that when you start a GUI based program, the X11/GUI information is thrown to the ip set in DISPLAY and the listening server on your windows grabs it and displays the GUI to you.

And BTW, you can also avoid this problem by directly logging into the Desktop environment (if installed) in your linux machine through the console.

12. [Bash/Linux usage tip]
 Setup a chrooted ssh sftp account. Yes, you can do it! (Tested on Suse) 

Add a user with a home directory:

useradd -d /home/bobuser -m bobuser

#Sftp/chroot Settings for bobuser in /etc/ssh/sshd_config
#Change LogLevel to debug and check errors (if any) in /var/log/messages
Subsystem sftp internal-sftp

#Sftp/chroot Settings for bobuser
Match User bobuser
   X11Forwarding no
   AllowTcpForwarding no
   ForceCommand internal-sftp
   ChrootDirectory /home/bobuser
Now restart the ssh service. And try connecting.

r00ter127:~ # service sshd restart
Shutting down SSH daemon done
Starting SSH daemon done
r00ter127:~ # sftp bobuser@localhost
Connecting to localhost...
Read from remote host localhost: Connection reset by peer
Couldn't read packet: Connection reset by peer
Ouch..We need to read the errors in /var/log/messages, we had already set it to debug level. There are some requirements expected by the ssh daemon

Jan 25 11:30:27 r00ter127 sshd[10220]: debug1: PAM: establishing credentials
Jan 25 11:30:27 r00ter127 sshd[10220]: fatal: bad ownership or modes for chroot directory "/home/bobuser"
Set the ownership of the home and parent directories to root. That's a requirement. chown root:root /home/bobuser

r00ter127:~ # sftp bobuser@localhost
Connecting to localhost...
subsystem request failed on channel 0
Couldn't read packet: Connection reset by peer
If you get the above error, then it means there is some problem invoking the sftp server. And the ssh logs are not very helpful in this regard. Make sure you are using the internal-sftp:

Subsystem sftp internal-sftp
   ForceCommand internal-sftp
And then.. you are done.

r00ter127:~ # sftp bobuser@localhost
Connecting to localhost...
sftp> pwd
Remote working directory: /

13. [Bash usage/Security tip]
 Audacious use of history to read a file, e.g. read the /etc/passwd file using history:

history -r /etc/passwd

14.[Bash usage tip]
 Use 'which' and 'type' to differentiate if a command is a binary command or a shell builtin.:

which history
type history

15. [Bash usage/Security tip]
What is the hashing algorithm used in my /etc/shadow:
Well, this could be useful if someone asks you whats the hashing algorithm being used to secure the OS passwords. Higher the number, more secure the algorithm. This tip is incomplete actually. You must also know what algorithms are supported by your Linux distro, and how to change the algo to a stronger one. You will also have to change the passwords so that they are hashed with the new algorithm.

$1 -> md5
$2a -> Blowfish
$5 -> Sha256
$6 -> Sha512

Go to part 2:

No comments:

Post a Comment