Configure HTTP Access Logging in WSO2 products

If you want to analyze your application's usage activities such as who is accessing your page, number of hits, errors ect, you can use the HTTP access log files monitor and analyze the above measures. WSO2 products uses embedded tomcat as it's primary runtime therefore, you can use apache tomcat access logger to monitor your activity and performance of the server as well as any errors that may be occurring. This post explain how you can configure HTTP access logging in WSO2 products in order to get efficient monitoring.

In WSO2 products you can customize the http access log configuration by editing the catalina-server.xml which is located {CARBON_HOME}/repository/conf/tomcat directory, which is the server descripter file for the embedded tomcat integration. In the catalina-server.xml, under valves you have the HTTP access log configuration as shown below

<Valve className="org.apache.catalina.valves.AccessLogValve" directory="${carbon.home}/repository/logs"

               prefix="http_access_" suffix=".log"

               pattern="combined" />

The server access log records all requests processed by the server. You can modify the what to log and what not to log by customizing the pattern attribute.

In the pattern attribute we can define formatting layouts. A formatting layout identifying the various information fields from the request and response to be logged, or the word "common" or "combined" to select a standard format.

Values for the pattern attribute are made up of literal text strings, combined with pattern identifiers prefixed by the "%" character to cause replacement by the corresponding variable value from the current request and response. The following pattern codes are supported:

%a - Remote IP address

%A - Local IP address

%b - Bytes sent, excluding HTTP headers, or '-' if no bytes were sent

%B - Bytes sent, excluding HTTP headers

%h - Remote host name

%H - Request protocol

%l - Remote logical username from identd (always returns '-')

%m - Request method

%p - Local port

%q - Query string (prepended with a '?' if it exists, otherwise an empty string

%r - First line of the request

%s - HTTP status code of the response

%S - User session ID

%t - Date and time, in Common Log Format format

%u - Remote user that was authenticated

%U - Requested URL path

%v - Local server name

%D - Time taken to process the request, in millis

%T - Time taken to process the request, in seconds

%I - current request thread name (can compare later with stacktraces) 

In addition, the caller can specify one of the following aliases for commonly utilized patterns:

common - %h %l %u %t "%r" %s %b

combined - %h %l %u %t "%r" %s %b "%{Referer}i" "%{User-Agent}i"

Please note that the optimized access does only support common and combined as the value for this attribute.

If you want to change the format you can modify the as shown below

 <Valve className="org.apache.catalina.valves.AccessLogValve" directory="${carbon.home}/repository/logs"

               prefix="http_access_" suffix=".log"

               pattern="%h %l %u %t '%r' %s %b" />

You can further change the log file location by changing the directory, prefix is the name of the log file. By default logs are rotated daily and the date is appended to the log file name. However, you can disable the log rotation by adding the following attribute rotatable="false". If you want to not have the current date appended in the log file name however, need the date to the rotated log file you can add renameOnRotate="true" attribute the the valve configuration. Please refer Access Log Valve Attributes for the full list of supported attributes. 

How to enable tenant aware service logging using WSO2 ESB

Unlike in other WSO2 products (WSO2 Appserver, WSO2 Data Services,  ect) we will not be able to see the application/service level logs  in ESB by default in ESB. Mainly because in the current carbon logging framework is  tenantaware and synapse is not multi tenanted and it runs on its own worker thread pool and does not know aboout the carbon context. Since it uses an independent thread pool from the carbon context logging does not work as expected with NHTTP. Therefore, to overcome this we wrote a handler call TenantDomainSetter which will set the tenant infomation/application infomation in the carbon context of each request.

To enable this you need to go to CARBON_HOME/repository/conf/axis2/axis2.xml and add the TenantDomainSetter in the prefetch of the request path (PreDispatch).

<handler name="TenantDomainSetter" class="org.wso2.carbon.utils.logging.handler.TenantDomainSetter"/>

Once you do that restart WSO2 ESB then you will be able to see service specific logs in LogViewer (monitor -> View Application Logs) which will show the logs of ESB proxy service logs. However application log viewer DOES NOT filter out Sequances, APIs, which are components inside a proxy service ... it will ONLY list down the proxy services.

Application Logs

System Logs

Please note since we are setting the tenant domain for every request there will be an aditional overhead in ESB.. if you want to minimize the overhead its recomended to switch of TenantDomainSetter and use ESB.

Kick start on Openstack with Devstack {Folsom}

Recently I was told to install openstack controller-compute node set ... and my first impression on it was piece of cake!! because you get couple of scripts and all you need to just run the script.. !! I WAS SOO WRONG!! yes it is just couple of script but you need to know how things work and there are soo many places where it can totally messed up your machine and your set up. Meaning you can easily get messed up!!! If you don't go in the right path it will take ages .. and you might even have to format your machine several times to get things clean and start all over again which is very time consuming.. So I ended up installing openstack for almost a week spending day and night on my precious development time!!!

So I decided to write a post on focusing on HOW TO INSTALL OPENSTACK? So other people want do the same mistake I did or take ages to get things done. If you know what you are doing then installing devstack only take one hour's time all you need to do is make your configuration right and run the scrip!!!

So before we begin I would like to give a small introduction to OpenStack to make things more clearer.

What is OpenStack ?

In simple terms OpenStack is an opensource cloud operating system ... It is like the fabric of the cloud. OpenStack was initially founded by rackspace and NASA in 2010 and its being growing ever since. The best way to get what OpenStack is by reading their mission which is very simple and direct

"To produce the ubiquitous Open Source Cloud Computing platform that will meet the needs of public and private clouds regardless of size, by being simple to implement and massively scalable."

OpenStack is written in purely written in python, true it use other systems like MYSQL and rabbitMQ but the core is written in python which makes things very efficient and fast. And also initially it had two main component 1) Nova - which is the compute side of it and the other one 2) swift - the data storage side of it.

Current OpenStack stable version {Folsom} have few core components

• Openstack object storage (swift)
• OpenStack image registry and delivery (Glance)
• Openstack compute (nova)
• Horizon (web UI)
• Keystone (authentication)
• Quantum/Melange (Network)

In future OpenStack is going to provide other compoents such Cellometer (Billing service), Cinder (block storage) and load balancer services. So that's enough about OpenStack lets see how we can install it. I will be installing OpenStack using devstack..

What is devstack ?

Well devstack is openstack development project (there are quite a few :) ) but I think devstack is the easiest :).  Devstack provides a documented shell script (which is written in bash) to build a complete OpenStack development environments. Originally created by Rackspace now maintained by OpenStack community. This supports both ubuntu and fedora (but today we will be focusing how we can install openstack in ubuntu 12.04).

Installing OpenStack  

In this installation I will be explaining how we can install OpenStack in Multi-Node environment.

Prerequisites 

Install ubuntu 12.04 and

Install git
     apt-get install -y git sudo

Network setting on each node

    You need to have a stack ip and also decide on what your Floating IP range (public IP range ) and Fixed IP range (private IP range) are.

This is my network configurations

  Static IP (controller ) - 10.100.0.30

  Floating (Public) IP range - 10.100.0.128/25

  Fixed (Private) IP range  - 192.168.16.0/25 

Go to /etc/network/interfaces and configure your network accordingly.

Controller node 

auto eth0
iface eth0 inet static
address 10.100.0.30
netmask 255.255.252.0
gateway 10.100.1.254

Compute Node

auto eth0
iface eth0 inet static
address 10.100.0.29
netmask 255.255.252.0
gateway 10.100.1.254



Before we begin we need to create  separate user to do things in a cleaner way.


groupadd stack
useradd -g stack -s /bin/bash -d /opt/stack -m stack
echo "stack ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers

From here onward we will be doing all the operation from the stack user.

Once you create the user log out and login from the the stack user and download openstack using git. When you are downloading make sure you download the stable version (in my case its folsom) so you will not have any surprises. :)


sudo su stack # login from the user
cd # go to user home
git clone git://github.com/openstack-dev/devstack.git -b stable/folsom #download devstack stable version

Once you download devstack other things are pretty straight foward. Go to devstack folder and create localrc  file and add the configuration accordingly in each node.

Configuring and Installing OpenStack in Controller Node

vi localrc

HOST_IP=10.100.0.30
FLAT_INTERFACE=eth0
FIXED_RANGE=192.168.16.0/25 
FIXED_NETWORK_SIZE=126
FLOATING_RANGE=10.100.0.128/25
MULTI_HOST=1
LOGFILE=/opt/stack/logs/stack.sh.log
ADMIN_PASSWORD=openstack
MYSQL_PASSWORD=root
RABBIT_PASSWORD=rabbitmq
SERVICE_PASSWORD=password
SERVICE_TOKEN=poohdedoo

Once you create the file you can run the setup script using the following command.

./stack.sh

After a successful  installation you will get the following output


Horizon is now available at http://10.100.0.30/
Keystone is serving at http://10.100.0.30:5000/v2.0/
Examples on using novaclient command line is in exercise.sh
The default users are: admin and demo
The password: openstack
This is your host ip: 10.100.0.30
stack.sh completed in 3443 seconds.

Configuring and Installing OpenStack in Compute Node

Just like the controller node you need to have the basic infrastructure in order to install the computer. Only change is there is the content of the localrc file in the compute node.


vi localrc

SERVICE_HOST=10.100.0.30
HOST_IP=10.100.0.29
FLAT_INTERFACE=eth0
FIXED_RANGE=192.168.16.0/25 
FIXED_NETWORK_SIZE=126
FLOATING_RANGE=10.100.0.128/25
MULTI_HOST=1
LOGFILE=/opt/stack/logs/stack.sh.log
ADMIN_PASSWORD=openstack
MYSQL_PASSWORD=root
RABBIT_PASSWORD=rabbitmq
SERVICE_PASSWORD=password
SERVICE_TOKEN=poohdedoo
MYSQL_HOST=$SERVICE_HOST
RABBIT_HOST=$SERVICE_HOST
GLANCE_HOSTPORT=$SERVICE_HOST:9292
Q_HOST=$SERVICE_HOST
ENABLED_SERVICES=n-cpu,n-net,n-api,c-sch,c-api,c-vol,rabbit


That's it !!. Just like the controller node you need to run the setup script to execute the installation.

./stack.sh

Once you successfully install openstack in both nodes you can login to openstack using the horizon dashboard

 http://10.100.0.30/ and login according to the user credentials you have provided in your setup configuration.

You can run the exercise.sh run provided by devstack in order to test your set up which includes uploading/launching instances and creating user groups/keypairs and user rules. 

If you want to upload images manually using the script you can use the following commands

glance image-create --name mysql-cartridge-amd64  --is-public true --container-format ami --disk-format ami < /home/wso2/wso2s2-openstack-1.0.0/cartridges/mysql-cartridge-amd64.img

To see what you have uploaded 

source openrc demo demo

glance image-list

You can also see these images using the horizon dashboard and further do task like launching the instances, assigning public IPs create key pairs and assign rules to security groups ect. I am not going to details since its a very straight forward dashboard and you can easily do everything by following the dashboard.

But I would like to proudly attach my openstack dashboad :) for references sake :)

On a final note restarting openstack is pretty tricky however, you can use the devstack script to restart openstack.

Restarting OpenStack using Devstack

Run the following in order to own the shell


script /dev/null 

And execute the following script
./rejoin-stack.sh

inside devstack folder in order to restart openStack.

To get out of the screen use ctrl-a , ctrl-d

Thats it folks !!
Once you set it up its like having a data center in your basement you can test load balancers and you can do all those caching stuff and experiment new stuff its really a cool thing to have. All the things you do which was in a private cloud and you can point it to a public cloud very easily!!!

Carbon Logging framework - Conversion Characters for TenantAwarePatternLayout

Usually when users use a logging framework one of their primary requirement is to customize the log output format. In Log4j framework this is done using the layout components.

There are couple of predefined layouts comming from the log4j api however, if you want more customize layout with extra set of attributes you need to write your own custom layout extending the org.apache.log4j.PatternLayout class.

In WSO2 Products we do not use the default PatternLyout we have written our own layout to make sure our log format is service aware and multi-tenanted. Therefore we have our own set of conversion characters defined in our TenantAwarePatternLayout class.

Below table explains characters used in the TenantAwarePatternLayout and all other characters (which are comming from PatternLayout) which you can use in your custom pattern.

Conversion Character	Description
Coming from custom carbon layout
D	used to output the name of the tenant domain which is currently logged in to the server (super tenant domain is carbon.super)
T	used to output the tenant id - to uniquely identify each tenant tenant id can be used in the layout (super tenant id is 0)
S	used to output the server key - to uniquely identify each carbon server a server key can be used (ie as,esb,dss )
U	used to output the user name - current user who is logged into the server
A	used to output the application name - service name or web application name of the current running applicaiton
H	used to output the host name
I	used to output the carbon instance id- uuid which is generated for each carbon instance to uniquely identify each instance
Coming from the log4j patternlayout
c	used to output category(logger); a.b.c → %c{2} = a.b (this can be also used as fully qualified name)
d	used to output the date of the logging event. for example, %d{hh:mm:ss,sss} or %d{dd mmm yyyy hh:mm:ss,sss}.
F	used to output the file name where the logging request was issued.
l	used to output location information of the caller which generated the logging event.
L	used to output the line number from where the logging request was issued.
m	used to output the application supplied message associated with the logging event.
M	used to output the method name where the logging request was issued.
n	used to output the platform dependent line separator character or characters.
p	used to output the priority of the logging event.
r	used to output the number of milliseconds elapsed from the construction of the layout until the creation of the logging event.
t	used to output the name of the thread that generated the logging event.

Please note %C, %F, %l, %L, %M slow down program run!

To create your own ConversionPattern you can go to the log4j.properties file and edit ConversionPattern of the appender of your choice.

log4j.appender.CARBON_LOGFILE.layout.ConversionPattern=TID: [%T] [%S] [%d] %P%5p {%c} - %x %m {%c}%n

Relational Storage Solution using WSO2 Storage as a Service

 Lets say you want to create a relational database on the cloud and you want to expose these data to other services?
Its very easy simple and very straight forward. All you need to have is a stratoslive account and then login to WSO2 Storage Server.

WSO2 Stroage Server provides almost all the data administrator level functionality when it comes to creating and manipulating data storage. Basically it provides features such as creating and manipulating data storage for relational data bases, Cassandra data storage and hadoop/hdfs file system.

Wondering How to create databases in the cloud ??

It only takes 4 clicks ... to Create and secure your database on the Cloud

In this article I am going to demonstrate on how you can create a database, and assign users and special privileges to that data base within few clicks

Step 1 - Creating Users

Assuming you already have a stratoslive account and logged in to storage as a server, lets first try to create a user for the database we are going to create.

Under Manage -> RSS Manager (relational storage service manager), you have the three basic functions to manipulate relational data bases.

To Create users click on Database Users. This will show the list of users (if you have not created users before you will not see any)  along with  "Add Database User " 

Select Database Server Instance Name as WSO2_RSS (which is the default server), and give the user credentials as your choice and click save to save the user.

Step 2 - Creating Privilege Group

Once you create the user you can create a template which separate out set of functions which can be performed on a given database,  so you can assign roles.

Go to RSS Manager ->  Privilege Templates -> Add Database Privilege Template. 

Lets say you only want this template to have   INSERT UPDATE DELETE CREATE DROP. Select the above fields along with a suitable name and save.

Step 3 - Creating Database

Now lets create a database in the cloud and assign the created users and privilege template to that database.

Go to RSS Manager -> Databases  -> Add Database 

Give the RSS instance name,  database name and save.

Once you create the database it will go under Databases list, where you have the option to manage databases by adding users and privileges to that database.

Go to RSS Manager -> Databases  -> database name -> Manage -> Attach Database User

Assign the user and role we created in step1 and step2 to the database as shown below.

Once you attach this user and role you can use this database with the user and the jdbc url shown in the databases page.

That's It .. now you have created your very own database in the cloud!!!

If you want view this database using a mysql client.. you can use WSO2 Data Services Server DB explorer.

Go to  WSO2 Data Services Server -> Tools -> Database Explorer.

Give the database information accordingly along with our newly created database and user.  And log in to your database.

And you can create your tables and manage data accordingly

If you want to expose your data as services you can use the following guide which explains how to expose your data as a service within few clicks.

How to setup Cassandra Cluster using WSO2 Storage Server

In my previous article I have explain how to configure multi-tenant logging in WSO2 stratos, in this post I am going to explain how we can create an external carbonized cluster and point it to BAM so that our logs will be stored in an external cassandra cluster. If you are having a production deployment and if you want to store big data for example daily logs of all servers, then you need to have an external cassandra cluster for high available and for high perfomance.

So to have a carbonized cassandra cluster.. we are going to use WSO2 storage server. WSO2 Storage server provides rich set of tools to create and manipulate storages such as relational data storages, cassandra and hdfs file systems.

To start you need to download WSO2 Storage Server.

Before we begin you need to have a basic understanding on cassandra in order to get a clear understanding on what we are trying to do. Unlike most master-slave deployments .. cassandra does  not use master/slave architecture, it uses a peer-to-peer implementation which avoids the pitfalls, latency problems, single point of failure issues and perfomance hits assosiated with master/slave setups. Which makes cassandra more high available and efficient.

So basically what happens when we write to cassandra is client write to any node in the cassandra cluster controller node replicate to  nodes and zones and nodes return acknowledgement to coordinator. Then coordinator return ack to client and data is written to internal commit log disk. If a node goes offline hinted handoff completes the write when the nodes come back up/

So lets begin to configure carbonized cassandra ..

Storage Server Management node deployment steps

cassandra-component.xml - 

cassandra-component.xml point the backend Cassandra cluster. 

<Cassandra>   

  <Cluster>

        <Name>SSCluster</Name>

        <DefaultPort>9160</DefaultPort>

        <Nodes>node0:9160,node1:9160,node2:9160,node3:9160</Nodes>

        <AutoDiscovery disable="true" delay="1000"/>

    </Cluster>

</Cassandra>

cassandra-auth.xml  

User has to create a system user with admin privilege to communicate with the cassandra backend and

configure the cassandra-auth.xml with that user and the remote shared key publisher service URL.

<Cassandra>  

<EPR>https://cassandra.cluster.backend.ip:9443/services/CassandraSharedKeyPublisher</EPR>

   <User>admin</User>

   <Password>admin</Password>

</Cassandra>

For the EPR you can give the IP of the first backend server node, you also need to consider the offset when providing the port as it will not automatically change it.

Since we are doing many hdfs/hadoop functions (to complete the bam story) using relational data storage we also need to configure rss-config.xml

rss-config.xml

There you need to give the datasource properties (jdbc url of the mysql server and user credentials) accordingly.

<dataSourceProps>          

 <property name="URL">jdbc:mysql://mysql.stratos-local.wso2.com:3306/rss_db</property>

            <property name="user">root</property>

            <property name="password">root</property>

         </dataSourceProps>

Storage Server Cassandra cluster deployment steps

All the nodes in the Storage Server Cassandra cluster should configure to use common carbon user base as any other carbon server. User has to update the user-mgt.xml and registry.xml with correct
configurations.

Node Configuration

cassandra.yaml

Start the node with default seed configuration

- seeds: "127.0.0.1"

Edit the cluster listening address 

listen_address:cassandra.node.ip

Edit the thrift listening address 

rpc_address::cassandra.node.ip

Both these ip addresses are the ip address of the machine it self which cassandra recedes on.

If you are  having another node as the seed node, then you can seed node to boostrap

ie 

rpc_address:ip.of.cassandra.api

Configuring BAM to talk to External Cassandra

Once we configure the external carbonized cassandra we need to tell the BAM we are no longer using the internal cassandra inbuilt in BAM but to use the external cassandra. In order to do that you need to start the BAM server with the  following system property.

-Ddisable.cassandra.server.startup=true

And also you need to configure the following configuration files.

cassandra­component.xml


 <Cassandra>
    <Cluster>
       <Name>ClusterOne</Name> <!--This is not important-->
<Nodes>node0.cassandra.com:9160,node1.cassandra.com:9160,node2.cassandra.com:9160</Nodes> <!--Host and port of the first node of the backend cluster. Refer the diagram to identify the first node. Need to consider offset of the backend node when providing the port here-->
       <AutoDiscovery disable="false" delay="1000"/>
    </Cluster>
</Cassandra>

cassandra-auth.xml  

 <Cassandra> <EPR>https://cassandra.cluster.backend.ip:9443/services/CassandraSharedKeyPublisher</EPR>

   <User>admin</User>

   <Password>admin</Password>

</Cassandra>

rss-config.xml

</dataSourceProps>     

  <property name="URL">jdbc:mysql://mysql.stratos-local.wso2.com:3306/rss_db</property>

            <property name="user">root</property>

            <property name="password">root</property>

         </dataSourceProps>

Also if you are using BAM analytics you need to configure the hive-site.xml as well

hive-site.xml

<property> 

<name>javax.jdo.option.ConnectionURL</name>

 <value>jdbc:mysql://mysql.server.url/hive_db</value>

 <description>JDBC connect string for a JDBC metastore</description>

</property>

<property>

    <name>fs.default.name</name>

    <value>hdfs://hdfs.url:hdfs_port</value> <!--normally the port is 9000 -->

 </property>

 <property>

    <name>mapred.job.tracker</name>

    <value>hdfs.url:hdfs_job_tracker_port</value> <!--normally the port is 9001 -->

 </property>

Introducing Leamo - To LEarn And MOre

Today I thought of introducing a cool cutting edge technology, which can help young minds to learn educate and have fun. Leamo, is an interactive learning platform which uses an iphone and bunch of flash cards and brings a magical environment to home/pre schools for kids under all ages.

Why not traditional learning ?

Most traditional education tools/applications uses the sit and learn approach, where educators have to talk and interact with kids while kids sit and listen. Most kids get soo bored with this traditional learning approach, and sometimes they get tired of the whole education system. Kids are more energetic than adults they need to run around poke things to nurish their curious minds. They cant just listen to you all the time and agree on what you say rather they like to experiment with new things and then learn how things work.

Why Leamo ? 

Kids like to play with toys, flash cards, play-dough and also interact with things without just listening and watching. They like to touch and feel things rather than look and listen. In the modern days kids are more attracted to digital screens, such as iphone, ipads, tablets and smartphones. Even though these ipad/ipones apps are very attractive for kids, there are major draw backs when it comes to kids education. I agree there are good educational applications provided by these gadgematics, however, it will not make your child develop the basic skills such as  creativity,  motor skills and hand-eye coordination ect. Most of these skills are develop when they interact with physical objects. So the solution is to mix the digital world with the physical world and give a hybrid educational platform for kids. So they wont get bored, or get distracted while learning and give a fulfilled learning experience.

This is the base of leamo, leamo provides a novel way of educating your child, using flashcards and digital surface. Where there are several education apps focusing mathematics, science and language skills along with history,arts music which will give them a balanced and rich education and develop both sides of their brain.

How Leamo Works?

Leamo runs on ipad, iphones and ipod (at the moment however, it will improve for android devices as well), all you need to do is connect your apple device to an LCD screen or a projector, and run the application. Leamo will come with bunch of flash cards and physical toys so the kids can interact with the digital world through the physical objects, by placing them in the correct place at the correct time. Leamo will guide you with all the kid's activities, parental supervision is not needed, kids can learn independently at their own space. Leamo will provide bunch of educational applications such as number games, language, fun with arts and music and story telling which will take the kids to a magical world where learning becomes addictive and fun. This is the best educational tool for energetic kids where they can run around, talk with it and make imagination come to reality.

Educating young minds ....

From my experience as a technology for education enthusiastic and a technology provider for kids, there are major advantages using Leamo.One of them is making them eager and addicted to learning new things through technology. And it improves their creative thinking skills, psycho-motor skills and reactive skills compared to other learning tools. Last but not least you don't need much parental guidance with leamo, you can easily set it up and make them learn independently.

 For more information on Leamo, please visit get-leamo and be amazed!!!!. Leamo is a proud product of Kreyonic,   An Interactive Digital Media Solutions provider specialize in providing interactive experiences, mobile solutions, that augment your perception of reality to create a new dimension in multimedia experience, taking digital media to the next level. 

Introduction to GIT with GIT commands

Recently I had some experiance working with git and I thought of sharing most basic and mostly used set of git commands. Before we begin I will give a quick introduction to "git".

What is "git"? 

Git is an open source distributed version control system designed to manange source code, system which designed for speed and efficiency. So the important factor here to remember is git is a distributed version control system unlike the old fasion centralized version control system such as SVN. In a centralized VCS you have a server which has all the source code and clients, and these two parties can be distringuishly identified. If a client wants to use the code to  do any modification, they have to checkout the source from the server and comit back to the server. So the entire code base stays in a single data store.If you loose the data store then you are screwed!!!. Git on the other hand is a distributed version control system, what it means is you dont check out a version of a project to start working but you clone it. So you just clone it to the local file system, this way is far efficient and you can work offline (You dont have to be online for most operations) no network issues,you can push and pull directly to peers. This means almost everything is local, so its very fast, every clone is a backup (so everyone working in that project has a backup for the system), and you can work offline.

Why use "git"?

I asked a friend why he thinks a project/company should adopt "git", his simple answer was less build breaks :D (which of course a good enough answer) and you can give restricted commit rights to team members so no careless glitches :). But of course there's more to that story, So I listed down few advantages which I could think of "git" and Why projects should move to "git"

• Its very fast (since almost everything is local)

• You want lose your code (Every clone is a backup to the system)

• You can work offline to perfom each of the mostly used task such as:

                 performing a diff,

                 viewing file history, 

                 committing changes, 

                merging branches,

                obtaining other revision of a file, 

                switching branches 

Which means you can work anywhere in the world even when you are up in the sky !!!!

• Its imutable (it never removes data) -"git" will not re write your history, it will always write a new history (you will have a pointer to your new history) you can always go back so you want lose data.

Before I begin I should warn you, if you are used to old fasion version control system like SVN you are going to hate "git". You will start hating everything about "git" and get tired of it very easily. Because its very differant from the centralized version control systems for example most version control systems are file based delta storage its mainly file based operations on the other hand "git" thinks about data as snapshots,it looks at the content (ignores the filename) and put that content in the database as key value pairs and return the key.That is, instead of thinking about and storing commit points as file based patches or changes, it stores it as a simple snapshot of what your project looked like when you committed.. So the easiest way to get you hands on these cool stuff, you need to foget all you know about centralized version control systems, youve been using for the past years and start thinking differantly and it will blow your mind :)

Before I bore you off with the conceptual infomation on "git" I will start giving the most commonly used "git" commands to start off with "git" and I will continue explaining interesting stuff about "git" in my next blog posts.

init -  This will initialize a brand new git repository in a project directory.

git init

clone - This will clone an exact copy of an existing project.

git clone http://git.stratos.com/amani.com/poo

add - Adding files

To add a single file:

 git add info.php

To add multiple files:

 git add info.php README.txt

To add all the files in the directory:

git add .

Status - This tells you what files have been modified since the last time they were committed.

git status

commit - Commit changes to head (but not yet to the remote repository):

git commit -m "Committing my changes" 

Push - Send changes to the master branch of your remote repository

git push origin master

Blame - To check who screwed it up :)

git blame hello.java

If you screwed it up :( ?

reset - Revert the uncommited changes from last commit

git reset --hard HEAD

checkout - Undo local changes 

git checkout -- myFile.txt

* If you mess up, you can replace the changes in your working tree with the last content in head:

Changes already added to the index, as well as new files, will be kept.

fetch - fetch the latest history from the server and point your local master branch at it.

git fetch origin

git reset --hard origin/master

Grep - Search the working directory for isService():

git grep "isService()"

Thats it for now to get a quick start on git try GitHub. GitHub made git more easy,  you can try it out and boost up on git with GitHub. :)

How to configure a hadoop cluster

Prerequisites

Before we begin there are couple of softwares you need to install  along with the hadoop user, before installing hadoop .

Java - Install Java into a location where all the user groups can access.
Eg: opt/java/jdk-1.6_29

rsync - Install rsync using apt-get (This is to copy the Hadoop distribution of Name Node across all the other nodes)

Create hadoop user -  Navigate to /home. Create the user “hadoop”

To login as user hadoop using the command.

su - hadoop

bash

PS: Above steps need to be performed (software should be installed) on all the other Hadoop nodes as well

Setup public key login from master to slave nodes

Create ssh public keys for each user (ssh-keygen -t rsa -b 2048) and added the public key (*.pub) to the authorized_keys file in master and slave nodes.

Key Exchange for Passphraseless SSH

1. We need to have password / passphraseless SSH to communicate with other Hadoop nodes in the
cluster.

Try to SSH to another node
          ssh hadoop@amani26.poohdedoo.com

2. Generate a key for the Name Node using the following command.

                ssh-keygen

This will generate an output similar to below.

Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
8207:9e:1e:60:37:28:03:a6:18:b3:b6:f1:e4:2f:ef hadoop@bam01

3. This will create a .ssh directory inside the ‘hadoop’ user account. Navigate and to .ssh directory. It will contain a file with the generated public key. Inspect the public key stored in the ‘id_rsa.pub’ file

with the command,
cat id_rsa.pub

It will display the public keys …

4. This public key of the Name Node should appended to the ‘authorized_keys’ file in the other Data
Nodes. Execute the following command and copy the id_rsa.pub file into the other nodes.

                scp id_rsa.pub root@amani276.poohdedoo.com:/root

5. Login to the second Hadoop node’s ‘hadoop’ user account. Try to SSH to another node from this.

                ssh hadoop@amani26.poohdedoo.com

This will create the .ssh directory in the hadoop account.

6. Append the copied public key to the ‘authorized_key’ file in the hadoop account of this Data node.
Execute the following commands.

               cat /root/id_rsa.pub > authorized_keys
        chown hadoop:hadoop authorized_keys
        chmod 600 authorized_keys

7. Now you can ssh to this Data node from the earlier configured Master node. Login to the Master
node. From the hadoop account, login to the Data node with the following command.

ssh -i id_rsa hadoop@amani27.poohdedoo.com
                or
ssh hadoop@amani27.poohdedoo.com

Setup Hadoop

1. Download and extract hadoop (tar xvfz hadoop-x.x.x.tar.gz -C /mnt/)
2. Change the permission of the extracted directory if necessary (chown -R user:user /mnt/hadoop-x.x.x)
3. [optional] If IPv6 is not used disable it.

- add 'net.ipv6.conf.all.disable_ipv6 = 1'

Configure Hadoop

Configuration files $HADOOP_HOME/conf/

• Set JAVA_HOME in $HADOOP_HOME/conf/hadoop-env.sh (Add export JAVA_HOME=/path/to/javahome)   
• eg: export JAVA_HOME=/opt/java/jdk1.6.0_29

• Edit he HADOOP_HOME/conf/core-site.xml as follows:

<configuration>

<property>

<name>fs.default.name</name>

<value>hdfs://hadoop0.poohdedoo.com:9000</value>

</property>

<property>

<name>fs.hdfs.impl</name>

<value>org.apache.hadoop.hdfs.DistributedFileSystem</value>

</property>

<property>

<name>hadoop.tmp.dir</name>

<value>/mnt/hadoop_tmp</value>

</property>

</configuration>

• Edit the $HADOOP_HOME/conf/hdfs-site.xml as follows:

<configuration>

<property>

<name>dfs.replication</name>

<value>1</value>

</property>

<property>

<name>dfs.name.dir</name>

<value>/mnt/hadoop_data/dfs/name</value>

</property>

<property>

<name>dfs.data.dir</name>

<value>/mnt/hadoop_data/dfs/data</value>

</property>

</configuration>

• Edit the $HADOOP_HOME/conf/mapred-site.xml as follows

<configuration>

<property>

<name>mapred.job.tracker</name>

<value>hadoop0.poohdedoo.com:9001</value>

</property>

<property>

<name>mapred.system.dir</name>

<value>/mnt/hadoop_data/mapred/system</value>

</property>

</configuration>

• Edit the $HADOOP_HOME/conf/hadoop-policy.xml

 By default the value for 'security.job.submission.protocol.acl' is *

change it to a user group or a name

<property>

<name>security.job.submission.protocol.acl</name>

<value>adminuser</value>

* Change the 'masters' and 'slaves' files (Master node only; slave machines does not need this configurations)

- $HADOOP_HOME/conf/maseters (masters file contain secondary namenode servers)

hadoop0.poohdedoo.com

- $HADOOP_HOME/conf/slaves (slaves file contain slave servers ; datanodes and task trackers)

hadoop1.poohdedoo.com

hadoop2.poohdedoo.com

Setting up hadoop cluster

Format the namenode before starting the cluster

$HADOOP_HOME/bin/hadoop namenode -format

start the services

$HADOOP_HOME/bin/start-all.sh

It will start namenode,jobtracker, secondarynamenode in master node and datanode and tasktracker on slave nodes. (To check the services run $JAVA_HOME/bin/jps)

stop the services

$HADOOP_HOME/bin/stop-all.sh

It will stop namenode,jobtracker, secondarynamenode in master node and datanode and tasktracker on slave nodes. (To check the services run $JAVA_HOME/bin/jps)