How to setup Cassandra Cluster using WSO2 Storage Server

In my previous article I have explain how to configure multi-tenant logging in WSO2 stratos, in this post I am going to explain how we can create an external carbonized cluster and point it to BAM so that our logs will be stored in an external cassandra cluster. If you are having a production deployment and if you want to store big data for example daily logs of all servers, then you need to have an external cassandra cluster for high available and for high perfomance.

So to have a carbonized cassandra cluster.. we are going to use WSO2 storage server. WSO2 Storage server provides rich set of tools to create and manipulate storages such as relational data storages, cassandra and hdfs file systems.

To start you need to download WSO2 Storage Server.

Before we begin you need to have a basic understanding on cassandra in order to get a clear understanding on what we are trying to do. Unlike most master-slave deployments .. cassandra does  not use master/slave architecture, it uses a peer-to-peer implementation which avoids the pitfalls, latency problems, single point of failure issues and perfomance hits assosiated with master/slave setups. Which makes cassandra more high available and efficient.

So basically what happens when we write to cassandra is client write to any node in the cassandra cluster controller node replicate to  nodes and zones and nodes return acknowledgement to coordinator. Then coordinator return ack to client and data is written to internal commit log disk. If a node goes offline hinted handoff completes the write when the nodes come back up/

So lets begin to configure carbonized cassandra ..

Storage Server Management node deployment steps

cassandra-component.xml - 

cassandra-component.xml point the backend Cassandra cluster. 

<Cassandra>   

  <Cluster>

        <Name>SSCluster</Name>

        <DefaultPort>9160</DefaultPort>

        <Nodes>node0:9160,node1:9160,node2:9160,node3:9160</Nodes>

        <AutoDiscovery disable="true" delay="1000"/>

    </Cluster>

</Cassandra>

cassandra-auth.xml  

User has to create a system user with admin privilege to communicate with the cassandra backend and

configure the cassandra-auth.xml with that user and the remote shared key publisher service URL.

<Cassandra>  

<EPR>https://cassandra.cluster.backend.ip:9443/services/CassandraSharedKeyPublisher</EPR>

   <User>admin</User>

   <Password>admin</Password>

</Cassandra>

For the EPR you can give the IP of the first backend server node, you also need to consider the offset when providing the port as it will not automatically change it.

Since we are doing many hdfs/hadoop functions (to complete the bam story) using relational data storage we also need to configure rss-config.xml

rss-config.xml

There you need to give the datasource properties (jdbc url of the mysql server and user credentials) accordingly.

<dataSourceProps>          

 <property name="URL">jdbc:mysql://mysql.stratos-local.wso2.com:3306/rss_db</property>

            <property name="user">root</property>

            <property name="password">root</property>

         </dataSourceProps>

Storage Server Cassandra cluster deployment steps

All the nodes in the Storage Server Cassandra cluster should configure to use common carbon user base as any other carbon server. User has to update the user-mgt.xml and registry.xml with correct
configurations.

Node Configuration

cassandra.yaml

Start the node with default seed configuration

- seeds: "127.0.0.1"

Edit the cluster listening address 

listen_address:cassandra.node.ip

Edit the thrift listening address 

rpc_address::cassandra.node.ip

Both these ip addresses are the ip address of the machine it self which cassandra recedes on.

If you are  having another node as the seed node, then you can seed node to boostrap

ie 

rpc_address:ip.of.cassandra.api

Configuring BAM to talk to External Cassandra

Once we configure the external carbonized cassandra we need to tell the BAM we are no longer using the internal cassandra inbuilt in BAM but to use the external cassandra. In order to do that you need to start the BAM server with the  following system property.

-Ddisable.cassandra.server.startup=true

And also you need to configure the following configuration files.

cassandra­component.xml


 <Cassandra>
    <Cluster>
       <Name>ClusterOne</Name> <!--This is not important-->
<Nodes>node0.cassandra.com:9160,node1.cassandra.com:9160,node2.cassandra.com:9160</Nodes> <!--Host and port of the first node of the backend cluster. Refer the diagram to identify the first node. Need to consider offset of the backend node when providing the port here-->
       <AutoDiscovery disable="false" delay="1000"/>
    </Cluster>
</Cassandra>

cassandra-auth.xml  

 <Cassandra> <EPR>https://cassandra.cluster.backend.ip:9443/services/CassandraSharedKeyPublisher</EPR>

   <User>admin</User>

   <Password>admin</Password>

</Cassandra>

rss-config.xml

</dataSourceProps>     

  <property name="URL">jdbc:mysql://mysql.stratos-local.wso2.com:3306/rss_db</property>

            <property name="user">root</property>

            <property name="password">root</property>

         </dataSourceProps>

Also if you are using BAM analytics you need to configure the hive-site.xml as well

hive-site.xml

<property> 

<name>javax.jdo.option.ConnectionURL</name>

 <value>jdbc:mysql://mysql.server.url/hive_db</value>

 <description>JDBC connect string for a JDBC metastore</description>

</property>

<property>

    <name>fs.default.name</name>

    <value>hdfs://hdfs.url:hdfs_port</value> <!--normally the port is 9000 -->

 </property>

 <property>

    <name>mapred.job.tracker</name>

    <value>hdfs.url:hdfs_job_tracker_port</value> <!--normally the port is 9001 -->

 </property>

Introducing Leamo - To LEarn And MOre

Today I thought of introducing a cool cutting edge technology, which can help young minds to learn educate and have fun. Leamo, is an interactive learning platform which uses an iphone and bunch of flash cards and brings a magical environment to home/pre schools for kids under all ages.

Why not traditional learning ?

Most traditional education tools/applications uses the sit and learn approach, where educators have to talk and interact with kids while kids sit and listen. Most kids get soo bored with this traditional learning approach, and sometimes they get tired of the whole education system. Kids are more energetic than adults they need to run around poke things to nurish their curious minds. They cant just listen to you all the time and agree on what you say rather they like to experiment with new things and then learn how things work.

Why Leamo ? 

Kids like to play with toys, flash cards, play-dough and also interact with things without just listening and watching. They like to touch and feel things rather than look and listen. In the modern days kids are more attracted to digital screens, such as iphone, ipads, tablets and smartphones. Even though these ipad/ipones apps are very attractive for kids, there are major draw backs when it comes to kids education. I agree there are good educational applications provided by these gadgematics, however, it will not make your child develop the basic skills such as  creativity,  motor skills and hand-eye coordination ect. Most of these skills are develop when they interact with physical objects. So the solution is to mix the digital world with the physical world and give a hybrid educational platform for kids. So they wont get bored, or get distracted while learning and give a fulfilled learning experience.

This is the base of leamo, leamo provides a novel way of educating your child, using flashcards and digital surface. Where there are several education apps focusing mathematics, science and language skills along with history,arts music which will give them a balanced and rich education and develop both sides of their brain.

How Leamo Works?

Leamo runs on ipad, iphones and ipod (at the moment however, it will improve for android devices as well), all you need to do is connect your apple device to an LCD screen or a projector, and run the application. Leamo will come with bunch of flash cards and physical toys so the kids can interact with the digital world through the physical objects, by placing them in the correct place at the correct time. Leamo will guide you with all the kid's activities, parental supervision is not needed, kids can learn independently at their own space. Leamo will provide bunch of educational applications such as number games, language, fun with arts and music and story telling which will take the kids to a magical world where learning becomes addictive and fun. This is the best educational tool for energetic kids where they can run around, talk with it and make imagination come to reality.

Educating young minds ....

From my experience as a technology for education enthusiastic and a technology provider for kids, there are major advantages using Leamo.One of them is making them eager and addicted to learning new things through technology. And it improves their creative thinking skills, psycho-motor skills and reactive skills compared to other learning tools. Last but not least you don't need much parental guidance with leamo, you can easily set it up and make them learn independently.

 For more information on Leamo, please visit get-leamo and be amazed!!!!. Leamo is a proud product of Kreyonic,   An Interactive Digital Media Solutions provider specialize in providing interactive experiences, mobile solutions, that augment your perception of reality to create a new dimension in multimedia experience, taking digital media to the next level.