Getting Hadoop Up and Running on Ubuntu

Hadoop is a framework written in Java, that allows processing of large datasets in a distributed manner across large clusters of commodity hardware.

Hadoop was created by Doug Cutting and he got the inspiration for Hadoop after reading Google papers "The Google File System" and "MapReduce: Simplified Data Processing on Large Clusters" published in 2003 and 2004 respectively.

As per the project main page, Hadoop includes four modules; Hadoop Common, Hadoop Distributed File System(HDFS), Hadoop YARN and Hadoop MapReduce. There is a whole ecosystem built around Hadoop; including projects like Apache Hive, Apache Pig, Apache ZooKeeper ect...

There are three modes to start a Hadoop cluster.

1. Local (Standalone) Mode
2. Pseudo-Distributed Mode
3. Fully-Distributed Mode

In this post my aim is to get Hadoop up and running on a Ubuntu host using Local (Standalone) Mode and on Pseudo-Distributed Mode.

Linux is supported as a development and a production platform by Hadoop. Since Hadoop runs on any Linux distribution, my choice of platform being Ubuntu will not have any effect to the readers who are using different Linux distributions. Note that environment configurations can be vary across different distributions.

For this post I'll be using Ubuntu 14.04 LTS and Apache Hadoop 2.5.1.

Prerequisites

Since Hadoop is written in Java, Java should be installed in your Ubuntu host. Refer this link for recommended Java versions. Perform following command in command line to check if you have already installed Java on your machine.

 
$ javac
$ java -version

This link provides a good resource in case you have not already installed Java.

Once Java is installed, you should set JAVA_HOME/bin to your PATH, to ensure java is available from the command line. To save the JAVA_HOME environment variable persistently, open up ~/.profile file using following command.

 
$ gedit ~/.profile

Append following lines to it and save.

export JAVA_HOME=/usr/lib/jvm/java-7-oracle
export PATH=$JAVA_HOME/bin

Note that after editing, you should re-login in order to initialize the variables, but you could use following command and use the variable without re-login. Also there are many different ways you can save an environment variables in Ubuntu, refer this link to find out what they are.

 
$ source ~/.profile

Downloading Hadoop

1. Download the latest stable Hadoop release from this link. You'll most likely to download a file named like; hadoop-2.5.1.tar.gz(This is the latest version of Hadoop at the time of this writing.)
2. I prefer Hadoop being installed in /usr/local directory. Decompress the downloaded file using the following command.

    
   $ tar -xf hadoop-2.5.1.tar.gz -C /usr/local/
   

3. Add $HADOOP_PREFIX/bin directory to your PATH, to ensure Hadoop is available from the command line. Follow the same steps we followed for adding JAVA_HOME variable, except you should append following in .profile file.
   
   export HADOOP_PREFIX=/usr/local/hadoop-2.5.0
   export PATH=$HADOOP_PREFIX/bin:$PATH
4. Define following parameters in etc/hadoop/hadoop-env.sh file.

 
$ export JAVA_HOME=/usr/lib/jvm/java-7-oracle
$ export HADOOP_PREFIX=/usr/local/hadoop-2.5.1

5. If the execution of the following command displays the usage documentation for Hadoop script, you are good to go to start your Hadoop cluster in one of the above mentioned three modes.

    
   $ hadoop
   

Standalone Mode

Hadoop by default is configured to run as a single Java process, which runs in a non distributed mode. Standalone mode is usually useful in development phase since it is easy to test and debug. Also, Hadoop daemons are not started in this mode. Since Hadoop's default properties are set to standalone mode and there are no Hadoop daemons to run, there are no additional steps to carry out here.

Pseudo-Distributed Mode

This mode simulates a small scale cluster, with Hadoop daemons running on a local machine. Each Hadoop daemon is run on a separate Java process. Pseudo-Distributed Mode is a special case of Fully distributed mode.

To enable Pseudo-Distributed Mode, you should edit following two XML files. These XML files contain multiple property elements within a single configuration element. Property elements contain name and value elements.

1. etc/hadoop/core-site.xml
2. etc/hadoop/hdfs-site.xml

Edit the core-site.xml and modify the following properties. fs.defaultFS property holds the locations of the NameNode.

 
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

Edit the hdfs-site.xml and modify the following properties. dfs.replication property holds the number of times each HDFS block should be replicated.

 
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

Setting up SSH

In a typical cluster, the Hadoop user should be able to execute commands on machines in the cluster, without having to enter a password for every single log in. If we use a password to log in to machines in the cluster, we will have to go on to each individual machine and start all the processes there.

Like I mentioned earlier, in Pseudo-Distributed Mode, we need to start Hadoop daemons. The host(single) being localhost, we need a way to log in to localhost without entering a password and start Hadoop daemons there. To do that we need ssh, we need to make sure we can ssh to localhost without giving a password. To allow password-less log in, we need to create a ssh key pair that has an empty password.

[ssh provides a way to securely log onto remote systems without using a password, using Key-based authentication. Key-based authentication creates a pair of keys; a private key and a public key. The private key will be kept as a secret at the client machine. The public key can be placed on any server you wish to access. Briefly what happens when a client tries to connect to a server is, server will generate a message to client using client's public key and only client can read it using it's private key. Based on the response server will get from client, server can decide if client is authorized or not.]

1. ssh is pre-packaged with Ubuntu, but we need to install ssh first to start sshd server. Use the following command to install ssh and sshd.

    
   $ sudo apt-get install ssh
   

   Verify installation using following commands.

    
   $ which ssh
   ## Should print '/usr/bin/ssh'
   
   $ which sshd
   ## Should print '/usr/bin/sshd'
   

2. Check if you can ssh to the localhost without a password.

    
   $ ssh localhost
   

   Note that if you try o ssh to the localhost without installing ssh first, an error message will be printed saying 'ssh: connect to host localhost port 22: Connection refused'. So be sure to install ssh first.
3. If you cannot SSH to the localhost without a password create a ssh key pair using the following command.

    
   $ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
   

4. Now the key pair has been created, note that id_rsa is the private key and id_rsa.pub is the public key are in .ssh directory. We need to include the new public key to the list of authorized keys using the following command.

    
   $ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
   

5. Now connect to the localhost and check if you can ssh to the localhost without a password.

    
   $ ssh localhost
   

Configuring the base HDFS directory

hadoop.tmp.dir property within core-site.xml file holds the location to the base HDFS directory. Note that this property configuration doesn't depend on the mode Hadoop runs on. The default value for hadoop.tmp.dir property is /tmp/hadoop-${user.name}, and there is a risk that some linux distributions might discard the contents of the /tmp directory in the local file system on each reboot, and leads to data loss within the local file system, hence to be on the safer side, it makes sense to change the location of the base directory to a much reliable one.

Carry out following steps to change the location of the base HDFS directory.

1. Create a directory for Hadoop to store its data locally and change its permissions to be writable by any user.

    
   $ mkdir /app/hadoop/tmp
   $ chmod 777 /app/hadoop/tmp
   

2. Edit the core-site.xml and modify the following property.

    
   <configuration>
       <property>
           <name>hadoop.tmp.dir</name>
           <value>/app/hadoop/tmp</value>
       </property>
   </configuration>
   

Formatting the HDFS filesystem

We need to format the HDFS file system, before starting Hadoop cluster in Pseudo-Distributed Mode for the first time. Note that formatting the file system multiple times will result deleting the existing file system data.

Execute the following command on command line to format the HDFS file system.

 
$ hdfs namenode -format

Starting NameNode daemon and DataNode daemon

 
$ $HADOOP_PREFIX/sbin/start-dfs.sh

Now you can access the name node web interface at http://localhost:50070/.

You can use the following command to see the Java Processes which are currently running.

 
$ jps

Creating the home directory

In Hadoop the home directories for each user are stored under /user directory.

If there is no /user directory, create that using the following command. Note that even though you would skip this step, /user directory will be automatically created by Hadoop later, if necessary.

 
$ hdfs dfs -mkdir /user

Then create your home directory using the following command.

 
$ hdfs dfs -mkdir /user/hadoop

Note that this is an explicit step, even if you don't carry out this step, a home directory will be automatically created later by Hadoop by the name you are logged into your local machine. For an example if I logged into my local machine as a user called pavithra, my home directory in HDFS will be /user/pavithra. So in order to utilize the previous step you should log in to your local machine as a user called hadoop.

Starting a MapReduce job

1. Use following command to create an input directory in HDFS.

    
   $ hdfs dfs -mkdir input
   

2. Use following command to copy input files into HDFS.

    
   $ hdfs dfs -put $HADOOP_PREFIX/etc/hadoop/*.xml input
   

3. Use the following command to run the MapReduced job, provided.

    
   $ hadoop jar $HADOOP_PREFIX/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.1.jar grep input output 'dfs[a-z.]+'
   

   Hadoop itself creates the output directory you have specified in this command. If by any chance you specify a directory which already exists in HDFS, Hadoop will throw an exception saying "Output directory already exists". With this Hadoop ensures data from previous jobs will not get replaced with the data from the current job.
4. Use following commands to copy the output files from HDFS to local file system and view them.

    
   $ hdfs dfs -get output output
   $ cat output/*
   

   OR you can use following command to View the output files on HDFS itself.

    
   $ bin/hdfs dfs -cat output/*
   

   Note that result files inside the output directory follows a naming convention of part-nnnnn.
5. Use the following command to stop the daemons.

 
$ $HADOOP_PREFIX/sbin/stop-dfs.sh

RESTful Web Services with Java

REST stands for REpresentational State Transfer, was first introduced by Roy Fielding in his thesis "Architectural Styles and the Design of Network-based Software Architectures" in year 2000.

REST is an architectural style. HTTP is a protocol which contains the set of REST architectural constraints.

REST fundamentals

• Everything in REST is considered as a resource.
• Every resource is identified by an URI.
• Uses uniform interfaces. Resources are handled using POST, GET, PUT, DELETE operations which are similar to Create, Read, update and Delete(CRUD) operations.
• Be stateless. Every request is an independent request. Each request from client to server must contain all the information necessary to understand the request.
• Communications are done via representations. E.g. XML, JSON

RESTful Web Services

RESTful Web Services have embraced by large service providers across the web as an alternative to SOAP based Web Services due to its simplicity. This post will demonstrate how to create a RESTful Web Service and client using Jersey framework which extends JAX-RS API. Examples are done using Eclipse IDE and Java SE 6.

Creating RESTful Web Service

• In Eclipse, create a new dynamic web project called "RESTfulWS"

• Download Jersey zip bundle from here. Jersey version used in these examples is 1.17.1. Once you unzip it you'll have a directory called "jersey-archive-1.17.1". Inside it find the lib directory. Copy following jars from there and paste them inside WEB-INF -> lib folder in your project. Once you've done that, add those jars to your project build path as well.
  1. asm-3.1.jar
  2. jersey-client-1.17.1.jar
  3. jersey-core-1.17.1.jar
  4. jersey-server-1.17.1.jar
  5. jersey-servlet-1.17.1.jar
  6. jsr311-api-1.1.1.jar


• In your project, inside Java Resources -> src create a new package called "com.eviac.blog.restws". Inside it create a new java class called "UserInfo". Also include the given web.xml file inside WEB-INF folder.
UserInfo.java

 
package com.eviac.blog.restws;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;

/**
 * 
 * @author pavithra
 * 
 */

// @Path here defines class level path. Identifies the URI path that 
// a resource class will serve requests for.
@Path("UserInfoService")
public class UserInfo {

 // @GET here defines, this method will method will process HTTP GET
 // requests.
 @GET
 // @Path here defines method level path. Identifies the URI path that a
 // resource class method will serve requests for.
 @Path("/name/{i}")
 // @Produces here defines the media type(s) that the methods
 // of a resource class can produce.
 @Produces(MediaType.TEXT_XML)
 // @PathParam injects the value of URI parameter that defined in @Path
 // expression, into the method.
 public String userName(@PathParam("i") String i) {

  String name = i;
  return "<User>" + "<Name>" + name + "</Name>" + "</User>";
 }
 
 @GET 
 @Path("/age/{j}") 
 @Produces(MediaType.TEXT_XML)
 public String userAge(@PathParam("j") int j) {

  int age = j;
  return "<User>" + "<Age>" + age + "</Age>" + "</User>";
 }
}

web.xml

<?xml version="1.0" encoding="UTF-8"?>  
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" id="WebApp_ID" version="2.5">  
  <display-name>RESTfulWS</display-name>  
  <servlet>  
    <servlet-name>Jersey REST Service</servlet-name>  
    <servlet-class>com.sun.jersey.spi.container.servlet.ServletContainer</servlet-class>  
    <init-param>  
      <param-name>com.sun.jersey.config.property.packages</param-name>  
      <param-value>com.eviac.blog.restws</param-value>  
    </init-param>  
    <load-on-startup>1</load-on-startup>  
  </servlet>  
  <servlet-mapping>  
    <servlet-name>Jersey REST Service</servlet-name>  
    <url-pattern>/rest/*</url-pattern>  
  </servlet-mapping>  
</web-app>

• To run the project, right click on it and click on run as ->run on server.
• Execute the following URL in your browser and you'll see the output.

  http://localhost:8080/RESTfulWS/rest/UserInfoService/name/Pavithra
  


output

Creating Client

• Create a package called "com.eviac.blog.restclient". Inside it create a java class called "UserInfoClient".
UserInfoClient.java

package com.eviac.blog.restclient;

import javax.ws.rs.core.MediaType;

import com.sun.jersey.api.client.Client;
import com.sun.jersey.api.client.ClientResponse;
import com.sun.jersey.api.client.WebResource;
import com.sun.jersey.api.client.config.ClientConfig;
import com.sun.jersey.api.client.config.DefaultClientConfig;

/**
 * 
 * @author pavithra
 * 
 */
public class UserInfoClient {

 public static final String BASE_URI = "http://localhost:8080/RESTfulWS";
 public static final String PATH_NAME = "/UserInfoService/name/";
 public static final String PATH_AGE = "/UserInfoService/age/";

 public static void main(String[] args) {

  String name = "Pavithra";
  int age = 25;

  ClientConfig config = new DefaultClientConfig();
  Client client = Client.create(config);
  WebResource resource = client.resource(BASE_URI);

  WebResource nameResource = resource.path("rest").path(PATH_NAME + name);
  System.out.println("Client Response \n"
    + getClientResponse(nameResource));
  System.out.println("Response \n" + getResponse(nameResource) + "\n\n");

  WebResource ageResource = resource.path("rest").path(PATH_AGE + age);
  System.out.println("Client Response \n"
    + getClientResponse(ageResource));
  System.out.println("Response \n" + getResponse(ageResource));
 }

 /**
  * Returns client response.
  * e.g : 
  * GET http://localhost:8080/RESTfulWS/rest/UserInfoService/name/Pavithra 
  * returned a response status of 200 OK
  *
  * @param service
  * @return
  */
 private static String getClientResponse(WebResource resource) {
  return resource.accept(MediaType.TEXT_XML).get(ClientResponse.class)
    .toString();
 }

 /**
  * Returns the response as XML
  * e.g : <User><Name>Pavithra</Name></User> 
  * 
  * @param service
  * @return
  */
 private static String getResponse(WebResource resource) {
  return resource.accept(MediaType.TEXT_XML).get(String.class);
 }
}

• Once you run the client program, you'll get following output.

Client Response 
GET http://localhost:8080/RESTfulWS/rest/UserInfoService/name/Pavithra returned a response status of 200 OK
Response 
<User><Name>Pavithra</Name></User>


Client Response 
GET http://localhost:8080/RESTfulWS/rest/UserInfoService/age/25 returned a response status of 200 OK
Response 
<User><Age>25</Age></User>

Enjoy!

Web Server in C

I implemented a web server in C language using only the standard libraries and thought it would be useful for you guys if I share the code.

The server runs on Linux and includes features like handling HTTP GET request, handling content types(txt, html, jpg, zip. rar, pdf, php etc.), sending proper HTTP error codes, serving the files from a web root, change in web root in a config file, zero copy optimization using sendfile method and php file handling. A port number should be provided as a command line argument.

After the server is up and running you can request for files using a web browser like Firefox.

For an example assume port number is "9000" and if you want to request a file called "test.php" which is in the webroot, use

http://localhost:9000/test.php 

WebServer.c

/*
 * WebServer.c
 *
 *  Created on: Nov 3, 2012
 *      Author: pavithra
 *
 * A web server in C language using only the standard libraries.
 * The port number is passed as an argument.
 *
 */

#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <fcntl.h>
#include <errno.h>

#define EOL "\r\n"
#define EOL_SIZE 2

typedef struct {
 char *ext;
 char *mediatype;
} extn;

//Possible media types
extn extensions[] ={
 {"gif", "image/gif" },
 {"txt", "text/plain" },
 {"jpg", "image/jpg" },
 {"jpeg","image/jpeg"},
 {"png", "image/png" },
 {"ico", "image/ico" },
 {"zip", "image/zip" },
 {"gz",  "image/gz"  },
 {"tar", "image/tar" },
 {"htm", "text/html" },
 {"html","text/html" },
 {"php", "text/html" },
 {"pdf","application/pdf"},
 {"zip","application/octet-stream"},
 {"rar","application/octet-stream"},
 {0,0} };

/*
 A helper function
 */
void error(const char *msg) {
 perror(msg);
 exit(1);
}

/*
 A helper function
 */
int get_file_size(int fd) {
 struct stat stat_struct;
 if (fstat(fd, &stat_struct) == -1)
  return (1);
 return (int) stat_struct.st_size;
}

/*
 A helper function
 */
void send_new(int fd, char *msg) {
 int len = strlen(msg);
 if (send(fd, msg, len, 0) == -1) {
  printf("Error in send\n");
 }
}

/*
 This function recieves the buffer
 until an "End of line(EOL)" byte is recieved
 */
int recv_new(int fd, char *buffer) {
 char *p = buffer; // Use of a pointer to the buffer rather than dealing with the buffer directly
 int eol_matched = 0; // Use to check whether the recieved byte is matched with the buffer byte or not
 while (recv(fd, p, 1, 0) != 0) // Start receiving 1 byte at a time
 {
  if (*p == EOL[eol_matched]) // if the byte matches with the first eol byte that is '\r'
    {
   ++eol_matched;
   if (eol_matched == EOL_SIZE) // if both the bytes matches with the EOL
   {
    *(p + 1 - EOL_SIZE) = '\0'; // End the string
    return (strlen(buffer)); // Return the bytes recieved
   }
  } else {
   eol_matched = 0;
  }
  p++; // Increment the pointer to receive next byte
 }
 return (0);
}

/*
 A helper function: Returns the
 web root location.
 */
char* webroot() {
 // open the file "conf" for reading
 FILE *in = fopen("conf", "rt");
 // read the first line from the file
 char buff[1000];
 fgets(buff, 1000, in);
 // close the stream
 fclose(in);
 char* nl_ptr = strrchr(buff, '\n');
 if (nl_ptr != NULL)
  *nl_ptr = '\0';
 return strdup(buff);
}

/*
 Handles php requests
 */
void php_cgi(char* script_path, int fd) {
 send_new(fd, "HTTP/1.1 200 OK\n Server: Web Server in C\n Connection: close\n");
 dup2(fd, STDOUT_FILENO);
 char script[500];
 strcpy(script, "SCRIPT_FILENAME=");
 strcat(script, script_path);
 putenv("GATEWAY_INTERFACE=CGI/1.1");
 putenv(script);
 putenv("QUERY_STRING=");
 putenv("REQUEST_METHOD=GET");
 putenv("REDIRECT_STATUS=true");
 putenv("SERVER_PROTOCOL=HTTP/1.1");
 putenv("REMOTE_HOST=127.0.0.1");
 execl("/usr/bin/php-cgi", "php-cgi", NULL);
}

/*
 This function parses the HTTP requests,
 arrange resource locations,
 check for supported media types,
 serves files in a web root,
 sends the HTTP error codes.
 */
int connection(int fd) {
 char request[500], resource[500], *ptr;
 int fd1, length;
 if (recv_new(fd, request) == 0) {
  printf("Recieve Failed\n");
 }
 printf("%s\n", request);
 // Check for a valid browser request
 ptr = strstr(request, " HTTP/");
 if (ptr == NULL) {
  printf("NOT HTTP !\n");
 } else {
  *ptr = 0;
  ptr = NULL;

  if (strncmp(request, "GET ", 4) == 0) {
   ptr = request + 4;
  }
  if (ptr == NULL) {
   printf("Unknown Request ! \n");
  } else {
   if (ptr[strlen(ptr) - 1] == '/') {
    strcat(ptr, "index.html");
   }
   strcpy(resource, webroot());
   strcat(resource, ptr);
   char* s = strchr(ptr, '.');
   int i;
   for (i = 0; extensions[i].ext != NULL; i++) {
    if (strcmp(s + 1, extensions[i].ext) == 0) {
     fd1 = open(resource, O_RDONLY, 0);
     printf("Opening \"%s\"\n", resource);
     if (fd1 == -1) {
      printf("404 File not found Error\n");
      send_new(fd, "HTTP/1.1 404 Not Found\r\n");
      send_new(fd, "Server : Web Server in C\r\n\r\n");
      send_new(fd, "<html><head><title>404 Not Found</head></title>");
      send_new(fd, "<body><p>404 Not Found: The requested resource could not be found!</p></body></html>");
      //Handling php requests
     } else if (strcmp(extensions[i].ext, "php") == 0) {
      php_cgi(resource, fd);
      sleep(1);
      close(fd);
      exit(1);
     } else {
      printf("200 OK, Content-Type: %s\n\n",
        extensions[i].mediatype);
      send_new(fd, "HTTP/1.1 200 OK\r\n");
      send_new(fd, "Server : Web Server in C\r\n\r\n");
      if (ptr == request + 4) // if it is a GET request
        {
       if ((length = get_file_size(fd1)) == -1)
        printf("Error in getting size !\n");
       size_t total_bytes_sent = 0;
       ssize_t bytes_sent;
       while (total_bytes_sent < length) {
        //Zero copy optimization
        if ((bytes_sent = sendfile(fd, fd1, 0,
          length - total_bytes_sent)) <= 0) {
         if (errno == EINTR || errno == EAGAIN) {
          continue;
         }
         perror("sendfile");
         return -1;
        }
        total_bytes_sent += bytes_sent;
       }

      }
     }
     break;
    }
    int size = sizeof(extensions) / sizeof(extensions[0]);
    if (i == size - 2) {
     printf("415 Unsupported Media Type\n");
     send_new(fd, "HTTP/1.1 415 Unsupported Media Type\r\n");
     send_new(fd, "Server : Web Server in C\r\n\r\n");
     send_new(fd, "<html><head><title>415 Unsupported Media Type</head></title>");
     send_new(fd, "<body><p>415 Unsupported Media Type!</p></body></html>");
    }
   }

   close(fd);
  }
 }
 shutdown(fd, SHUT_RDWR);
}

int main(int argc, char *argv[]) {
 int sockfd, newsockfd, portno, pid;
 socklen_t clilen;
 struct sockaddr_in serv_addr, cli_addr;

 if (argc < 2) {
  fprintf(stderr, "ERROR, no port provided\n");
  exit(1);
 }
 sockfd = socket(AF_INET, SOCK_STREAM, 0);
 if (sockfd < 0)
  error("ERROR opening socket");
 bzero((char *) &serv_addr, sizeof(serv_addr));
 portno = atoi(argv[1]);
 serv_addr.sin_family = AF_INET;
 serv_addr.sin_addr.s_addr = INADDR_ANY;
 serv_addr.sin_port = htons(portno);
 if (bind(sockfd, (struct sockaddr *) &serv_addr, sizeof(serv_addr)) < 0)
  error("ERROR on binding");
 listen(sockfd, 5);
 clilen = sizeof(cli_addr);
 /*
  Server runs forever, forking off a separate
  process for each connection.
  */
 while (1) {
  newsockfd = accept(sockfd, (struct sockaddr *) &cli_addr, &clilen);
  if (newsockfd < 0)
   error("ERROR on accept");
  pid = fork();
  if (pid < 0)
   error("ERROR on fork");
  if (pid == 0) {
   close(sockfd);
   connection(newsockfd);
   exit(0);
  } else
   close(newsockfd);
 } /* end of while */
 close(sockfd);
 return 0; /* we never get here */
}

Make sure you have installed php and there's exist a "conf" file consisted of the webroot before running this.

Enjoy!

NS-2 Simulation

Installing NS2 on Ubuntu 11.04

sudo apt-get install ns2 nam

Here I have used NS-2 to simulate a network with the topology as shown below.

I have written tcl scripts to create the above topology and to show,

1. When two TCP flows compete for the bandwidth they fairly share the bandwidth
2. When a TCP flow and a UDP flow compete for the bandwidth there is no fair sharing

Also I have plotted the throughput at the receivers against time using xgraph.

Installing xgraph on Ubuntu 11.04

sudo apt-get install xgraph

To start the simulation

ns XXX.tcl

How two TCP flows compete for the bandwidth

tcp_tcp.tcl

This script simulates two TCP flows; one between n0 and n4 and one between n1 and n5. Also the bandwidth of L2 has selected in such a way that it becomes the bottleneck link and the "record" procedure has used to measure the throughput at the receivers. .

#Create a simulator object
set ns [new Simulator]

#Define different colors for data flows (for NAM)
$ns color 1 Blue
$ns color 2 Red

#Open the trace files outX.tr for Xgraph and out.nam for nam
set f0 [open out_tcp0.tr w]
set f1 [open out_tcp1.tr w]

#Open the NAM trace file
set nf [open out.nam w]
$ns namtrace-all $nf

#Define a 'finish' procedure
proc finish {} {
 global ns nf f0 f1
 $ns flush-trace
#Close the NAM trace file
 close $nf
#Close the output files
 close $f0
 close $f1
#Execute xgraph to display the results
 exec xgraph out_tcp0.tr out_tcp1.tr -geometry 600x400 &
#Execute NAM on the trace file
 exec nam out.nam &
 exit 0
}

#Create five nodes
set n0 [$ns node]
set n1 [$ns node]
set n2 [$ns node]
set n3 [$ns node]
set n4 [$ns node]
set n5 [$ns node]

#Create links between the nodes
$ns duplex-link $n0 $n2 2Mb 10ms DropTail
$ns duplex-link $n1 $n2 2Mb 10ms DropTail
$ns duplex-link $n2 $n3 1.7Mb 20ms DropTail
$ns duplex-link $n3 $n4 2Mb 10ms DropTail
$ns duplex-link $n3 $n5 2Mb 10ms DropTail

#Set Queue Size of link (n2-n3) to 20
$ns queue-limit $n2 $n3 20

#Give node position (for NAM)
$ns duplex-link-op $n0 $n2 orient right-down
$ns duplex-link-op $n1 $n2 orient right-up
$ns duplex-link-op $n2 $n3 orient right
$ns duplex-link-op $n3 $n4 orient right-up
$ns duplex-link-op $n3 $n5 orient right-down

#record procedure
proc record {} {
 global sink sink1 f0 f1
#Get an instance of the simulator
 set ns [Simulator instance]

#Set the time after which the procedure should be called again
 set time 0.5

#How many bytes have been received by the traffic sinks?
 set bw0 [$sink set bytes_]
 set bw1 [$sink1 set bytes_]

#Get the current time
 set now [$ns now]

#Calculate the bandwidth (in MBit/s) and write it to the files
 puts $f0 "$now [expr $bw0/$time*8/1000000]"
 puts $f1 "$now [expr $bw1/$time*8/1000000]"

#Reset the bytes_ values on the traffic sinks
 $sink set bytes_ 0
 $sink1 set bytes_ 0

#Re-schedule the procedure
 $ns at [expr $now+$time] "record"
}

#Setup a TCP connection
set tcp [new Agent/TCP]
$tcp set class_ 2
$ns attach-agent $n0 $tcp
set sink [new Agent/TCPSink]
$ns attach-agent $n4 $sink
$ns connect $tcp $sink
$tcp set fid_ 1

#Setup a FTP over TCP connection
set ftp [new Application/FTP]
$ftp attach-agent $tcp
$ftp set type_ FTP

#Setup a TCP connection
set tcp1 [new Agent/TCP]
$tcp1 set class_ 2
$ns attach-agent $n1 $tcp1
set sink1 [new Agent/TCPSink]
$ns attach-agent $n5 $sink1
$ns connect $tcp1 $sink1
$tcp1 set fid_ 2

#Setup a FTP over TCP connection
set ftp1 [new Application/FTP]
$ftp1 attach-agent $tcp1
$ftp1 set type_ FTP

#Start logging the received bandwidth
$ns at 0.0 "record"

#Schedule events for the FTP agents
$ns at 0.1 "$ftp start"
$ns at 0.8 "$ftp1 start"
$ns at 4.0 "$ftp1 stop"
$ns at 4.8 "$ftp stop"

#Call the finish procedure after 5 seconds of simulation time
$ns at 5.0 "finish"

#Run the simulation
$ns run

Graph

Above graph visualization clearly shows how two TCP flows fairly share the bandwidth.

How a TCP flow and a UDP flow compete for the bandwidth

tcp_udp.tcl

This script simulates one TCP flow (between n0 and n4) and one UDP flow (between n1 and n5). Again the bandwidth of L2 has selected in such a way that it becomes the bottleneck link and the "record" procedure has used to measure the throughput at the receivers.

#Create a simulator object
set ns [new Simulator]

#Define different colors for data flows (for NAM)
$ns color 1 Blue
$ns color 2 Red

#Open the trace files outX.tr for Xgraph and out.nam for nam
set f0 [open out_tcp.tr w]
set f1 [open out_udp.tr w]

#Open the NAM trace file
set nf [open out_udptcp.nam w]
$ns namtrace-all $nf

#Define a 'finish' procedure
proc finish {} {
 global ns nf f0 f1
 $ns flush-trace
#Close the NAM trace file
 close $nf
#Close the output files
 close $f0
 close $f1
#Execute xgraph to display the results
 exec xgraph out_tcp.tr out_udp.tr -geometry 600x400 &
#Execute NAM on the trace file
 exec nam out_udptcp.nam &
 exit 0
}

#Create five nodes
set n0 [$ns node]
set n1 [$ns node]
set n2 [$ns node]
set n3 [$ns node]
set n4 [$ns node]
set n5 [$ns node]

#Create links between the nodes
$ns duplex-link $n0 $n2 2Mb 10ms DropTail
$ns duplex-link $n1 $n2 2Mb 10ms DropTail
$ns duplex-link $n2 $n3 1.7Mb 20ms DropTail
$ns duplex-link $n3 $n4 2Mb 10ms DropTail
$ns duplex-link $n3 $n5 2Mb 10ms DropTail

#Set Queue Size of link (n2-n3) to 20
$ns queue-limit $n2 $n3 20

#Give node position (for NAM)
$ns duplex-link-op $n0 $n2 orient right-down
$ns duplex-link-op $n1 $n2 orient right-up
$ns duplex-link-op $n2 $n3 orient right
$ns duplex-link-op $n3 $n4 orient right-up
$ns duplex-link-op $n3 $n5 orient right-down

#record procedure
proc record {} {
 global sink sink1 f0 f1
#Get an instance of the simulator
 set ns [Simulator instance]

#Set the time after which the procedure should be called again
 set time 0.5

#How many bytes have been received by the traffic sinks?
 set bw0 [$sink set bytes_]
 set bw1 [$sink1 set bytes_]

#Get the current time
 set now [$ns now]

#Calculate the bandwidth (in MBit/s) and write it to the files
 puts $f0 "$now [expr $bw0/$time*8/1000000]"
 puts $f1 "$now [expr $bw1/$time*8/1000000]"

#Reset the bytes_ values on the traffic sinks
 $sink set bytes_ 0
 $sink1 set bytes_ 0

#Re-schedule the procedure
 $ns at [expr $now+$time] "record"
}

#Setup a TCP connection
set tcp [new Agent/TCP]
$tcp set class_ 2
$ns attach-agent $n0 $tcp
set sink [new Agent/TCPSink]
$ns attach-agent $n4 $sink
$ns connect $tcp $sink
$tcp set fid_ 1

#Setup a FTP over TCP connection
set ftp [new Application/FTP]
$ftp attach-agent $tcp
$ftp set type_ FTP

#Setup a UDP connection
set udp [new Agent/UDP]
$ns attach-agent $n1 $udp
set sink1 [new Agent/LossMonitor]
$ns attach-agent $n5 $sink1
$ns connect $udp $sink1
$udp set fid_ 2

#Setup a CBR over UDP connection
set cbr [new Application/Traffic/CBR]
$cbr attach-agent $udp
$cbr set type_ CBR
$cbr set packet_size_ 1000
$cbr set rate_ 2mb
$cbr set random_ false

#Start logging the received bandwidth
$ns at 0.0 "record"

#Schedule events for the CBR and FTP agents
$ns at 0.1 "$cbr start"
$ns at 0.8 "$ftp start"
$ns at 4.0 "$ftp stop"
$ns at 4.8 "$cbr stop"

#Call the finish procedure after 5 seconds of simulation time
$ns at 5.0 "finish"

#Run the simulation
$ns run

Graph

Above graph visualization clearly shows, when it comes to a TCP flow and a UDP flow there is no fair sharing the bandwidth, UDP gets the most of it.

How to deploy a war file using GlassFish

Before trying this out make sure you have installed GlassFish in your machine. Installation is just easy as downloading the zip archive from here and unzipping it to a desired location.

Creating a new domain

Open up a terminal, change directory to the GlassFish installation directory and run following.

bin/asadmin

This will enable you to use asadmin tool. Now execute the following command to create a new domain. after running this command you will probably have to give admin password and master password.

create-domain --adminport 5000 --profile developer --user admin domain2

Now open up another terminal and change directory to [glassfish-installation]/domains/ and you will see the newly created domain2 has appeared there.

Open up a browser and go to http://localhost:5000/. This will bring you the GlassFish GUI admin console. This is one place you can deploy your war file. But in this post I'm not much focusing on it, instead I will show you how to do this using the command line. But GUI lovers, for any consolation I have put some screenshots which you can follow if you prefer the GUI way.

In case you want to delete a domain use the following command.

delete-domain domain2

Starting the domain

To start domain2 run following command.

start-domain domain2

Deploying a war file

Use the following command to deploy your war file.

deploy --port 5000 --host localhost /home/pavithra/workspace/NewDemoService/WebServicesJaxWs/NewDemoService.war 

After deploying the war file I can access the WSDL file I want to access using the URL http://localhost:8080/NewDemoService/NewDemoService?WSDL

To change default 8080 (HTTP) port which specifies where the web application context roots are available for a Web browser to connect to, you can use the --instanceport parameter when creating the domain. See the following command.

create-domain --adminport 5000 --profile developer --user admin --instanceport 9000 domain2

Undeploying a war file

To undeploy NewDemoService.war file you need to use the following command. Note that here you don't have to use the full name but literal "NewDemoService".

undeploy --port 5000 --host localhost NewDemoService

Stopping a domain

To stop the domain "domain2" use the following command.

stop-domain domain2

After this if you try to deploy to this particular domain, it will complain.

Auto Deploy

To perform auto deploy, copy NewDemoService.war file in to [glassfish-installation-directory]/domains/domain2/autodeploy directory. If autodeploy succeeds, you will see NewDemoService.war_deployed has created. This will deploy your war file automatically.

Getting started with JAX-WS

JAX-WS stands for Java API for XML Web Services. It is a Java programming language API for creating web services and clients that communicate using XML. This post is a quick start for JAX-WS.

Prerequisites
GlassFish integrated with Eclipse.

Creating the JAX-WS Web Service

1. In Eclipse create a Dynamic Web Project called "com.eviac.blog.jaxwsproj". Make GlassFish as the Target Runtime.
   
2. Create a new class called "SampleWS" in the created project. This will be the implementation class of the web service.
   
   SampleWS.java

   package com.eviac.blog.jaxws.service;
   
   import javax.jws.WebMethod;
   import javax.jws.WebService;
   
   @WebService
   public class SampleWS {
   
    @WebMethod
    public int sum(int a, int b) {
     return a + b;
    }
   
    @WebMethod
    public int multiply(int a, int b) {
     return a * b;
    }
   
   }
   
   

3. Open a terminal and navigate to the root of the project directory. Create a directory called wsdl inside WebContent/WEB-INF/. Use the following command to create web service artifacts. Make sure your JAVA_ HOME is set properly or this command will not work. Also make sure to build the project before running this command or it will complain class not found.

   wsgen -classpath build/classes/ -wsdl -r WebContent/WEB-INF/wsdl -s src -d build/classes/ com.eviac.blog.jaxws.service.SampleWS
   

   
4. Refresh the project to discover created artifacts. Open the created WSDL-file inside wsdl folder. Search for REPLACE_WITH_ACTUAL_URL and replace it with the web service URL: http://localhost:8080/com.eviac.blog.jaxwsproj/SampleWSService, and save the file.
   
   
   
5. Deploy the project in Glassfish by right-clicking the project, click Run As -> Run on Server and select the Glassfish server.
   
   

Creating the JAX-WS client

1. Create a Java project in eclipse called "com.eviac.blog.jaxwsclientproj". Open up a new terminal and go to the project root. Use the following command to generate the classes you need to access the web service. Here you will need to use the URL of the WSDL file.

   wsimport -s src -d bin http://localhost:8080/com.eviac.blog.jaxwsproj/SampleWSService?wsdl
   

   
2. Create a new class called "SampleWSClient" in the project.
   
   SampleWSClient.java

   package com.eviac.blog.jaxws.client;
   
   import javax.xml.ws.WebServiceRef;
   
   import com.eviac.blog.jaxws.service.SampleWS;
   import com.eviac.blog.jaxws.service.SampleWSService;
   
   public class SampleWSClient {
   
    @WebServiceRef(wsdlLocation = "http://localhost:8080/com.eviac.blog.jaxwsproj/SampleWSService?wsdl")
    private static SampleWSService Samplews;
   
    public static void main(String[] args) {
     SampleWSClient wsClient = new SampleWSClient();
     wsClient.run();
    }
   
    public void run() {
     Samplews = new SampleWSService();
     SampleWS port = Samplews.getSampleWSPort();
     System.out.println("multiplication Result= "+ port.multiply(10, 20));
     System.out.println("Addition Result= "+port.sum(10, 20));
    }
   
   }
   

3. Right click on the project and click on Run As -> Java Application. This will result following.

   multiplication Result= 200
   Addition Result= 30
   

Integrating GlassFish with Eclipse 3.7

This post will guide you to integrate GlassFish 3.0.1 with Eclipse 3.7.

1. Download the zip archive for GlassFish from here.
2. Unzip it using the following command.

   unzip glassfish-3.0.1.zip 
   

3. Move GlassFish installation directory to a more suitable place.

   sudo mv glassfishv3 /opt
   

4. Open Eclipse.
   Navigate to New -> Server and click on download additional server adapters link and install Oracle GlassFish server tools. Restart Eclipse, when the installation is finished.
   
   
   After restarting Eclipse, navigate to New -> server.
   
   Select GlassFish Server Open Source Edition 3 (Java EE 6) and click next.
   
   Select a working JRE environment, select the installation path of your Glassfish application server and click next.
   
   In this screen keep the default settings and click next.
   
   In this screen it lets you select existing projects to be added to the newly configured server adapter, just click Finish.
   
   After a successful configuration, server adapter will be appeared in the servers tab. Start the server by right-clicking on it.