How to configure MT-Logging with WSO2 Products

Prerequisites

Carbon 4.0.0 and above product
WSO2 BAM 2.0.0 and above
Apache Server installed

Introduction 

Stratos MT-Logging architecture provides a logging frame work to send logs to BAM. Which opens wide variety of possibilities when it comes to monitoring logs. In my previous article I have explained  the architecture of Distributed Logging with WSO2 BAM, this tutorial explains how you can set up logging effectively for any WSO2 Product and how you can analyze and monitor logs effectively.

Architecture

Setting up Hadoop Server to host archived log files

Once the logs are sent to BAM, we analyse the logs daily and send them to a file system. For better performance for archive logs analytic, we send archive logs to HDFS file system. So we can analyze archive logs using map reduce task (big data, long term data analysis).

Please refer How to Configure Hadoop to see how we can configure a hadoop cluster, Once you have hadoop cluster you can give your hdfs information in summarizer-config.xml. So it will automatically analyse your daily logs and send them to HDFS file system

Summarizer Configuration for log archiving.

<cronExpression>0 0 1 ? * * *</cronExpression>
<tmpLogDirectory>/home/usr/temp/logs</tmpLogDirectory>
<hdfsConfig>hdfs://localhost:9000</hdfsConfig>
<archivedLogLocation>/stratos/archivedLogs/</archivedLogLocation>
<bamUserName>admin</bamUserName>
<bamPassword>admin</bamPassword>

cronExpression - The schedule time, that summarizer runs daily

hdfsConfig - hdfs file server intimation
archivedLogLocation - HDFS file patch which the archived logs should be saved

Setting up Log4jAppender - Server Side (AS/ESB/GREG/etc)

To publish log events to BAM, log4j appender should be configured in each server. In order to do that you need to add LogEvent to the root logger and configure the LogEvent credential accordingly.

Add LogEvent to the root logger in log4j

Go to Server_Home/repository/con -> log4j.properties and LOGEVENT to log4j root logger (or replace the following line)  
log4j.rootLogger=INFO, CARBON_CONSOLE, CARBON_LOGFILE, CARBON_MEMORY, CARBON_SYS_LOG,LOGEVENT

Add Data publishing URLs and credentials
Go to  Server_Home/repository/con -> log4j.properties. Modify LOGEVENT appender’s  LOGEVENT.url as BAM Server thrift URL,LOGEVENT.userName, .LOGEVENT.password           

                                                            

log4j.appender.LOGEVENT=org.wso2.carbon.logging.appender.LogEventAppender

log4j.appender.LOGEVENT.url=tcp://localhost:7611

log4j.appender.LOGEVENT.layout=org.wso2.carbon.utils.logging.TenantAwarePatternLayout

log4j.appender.LOGEVENT.columnList=%T,%S,%A,%d,%c,%p,%m,%H,%I,%Stacktrace

log4j.appender.LOGEVENT.userName=admin

log4j.appender.LOGEVENT.password=admin

Enabling the Log Viewer

When the log viewer is not enable to take logs from cassandra the default behaviour of the log viewer to take logs from the carbon memory. It will only display the most recent logs of the carbon server. To get persistence logs (logs which are coming from the current date) you need to enable isLogsFromCassandra true so that you can view persistance logs through the management console of any carbon server (ESB/DSS/AS etc) . And also you need to give the user credentials of the cassandra server as shown below.

Change Logging-Config.xml to View Logs from BAM.

Got to Server_Home/repository/con/etc-> Logging-config.xml

Enable isDataFromCassandra

<isDataFromCassandra>true</isDataFromCassandra>

Give cassandra url of BAM Server

<cassandraHost>localhost:9160</cassandraHost>

Give BAM Server user credentials to access Cassandra Server in BAM

<userName>admin</userName>
<password>admin</password>

Give hadoop hdfs  hosted url for the logs viewer

<archivedHost>hdfs://localhost:9000</archivedHost>
<archivedHDFSPath>/stratos/logs</archivedHDFSPath>

Setting up Logging Analyzer - WSO2 BAM Side

Setting up BAM

Bind IPs for cassandra {This is not logging related, this is just to bind an ip address to cassandra so that cassandra will not start in localhost}

Copy cassandra.yaml from {WSO2_BAM_HOME}/repository/components/features/

org.wso2.carbon.cassandra.server_4.0.1/conf/cassandra.yaml to repository/conf/etc. Change the

IP address (localhost) to the correct ip address of BAM of listen_address and rpc_address

Copy cassandra-component.xml from {WSO2_BAM_HOME}/repository/components/

features/org.wso2.carbon.cassandra.dataaccess_4.0.1/conf/cassandra-component.xml to

repository/conf/etc. Change the IP address (localhost) to the correct ip address of BAM of

192.168.4.148:9160

Installing Logging Summarizer

Download P2 Profile which will contain  Logging summarizer features. Install logging.summarizer feature through Management Console (Go to Configure -> Features and Click on Add Repository). Once you add the repository you will be redirected to a page which contains available features. Select bam summarizer feature and install it. 

Change the logging config.xml

<isDataFromCassandra>true</isDataFromCassandra>

Change log rotation paths, give the log directory as apache log rotation directory, and give BAM username password credentials

<publisherURL>tcp://localhost:7611</publisherURL>
<publisherUser>admin</publisherUser>
<publisherPassword>admin</publisherPassword>

<logDirectory>/home/usr/apache/logs/</logDirectory>

 <tmpLogDirectory>/home/usr/temp/logs</tmpLogDirectory>

Point BAM to external hdfs file server

ow the logging is configured in both publisher and receiver, and you can view your logs by log-in into 
In order to point the analyzers to the hdfs file system you need to update BAM_HOME/repository/conf/advanced/hive-site.xml to point it to your hdfs file system.


<property>

  <name>fs.default.name</name>
  <value>hdfs://localhost:9000</value>
</property>

Now the logging is configured in both publisher and receiver, and you can view your logs by log-in into management console. And view System Logs - This will view the current logs as well as archived logs taken from the apache server.  Logs are daily archived to the apache server through a cron job.

If you want to analyze logs using hive analytics, and display in dashboards you can use bam analytics tools and dashboard tool kits to customize Logging KPIs for system administration.

How Distributed Logging Works in WSO2 Stratos.

Why we need distributed Logging ???????

Stratos is a distributed clustered setup where we have several applications such as ESB Servers,Application Servers, Identity Servers, Governance Servers, Data Services Sever etc  deployed together to work with each other to serve as Platform as a Service. Each of these servers are deployed in a clustered environment, where there will be more than one node for a given server and depending on the need, there will be new nodes spawned dynamically inside this cluster. And all these servers are fronted through an Elastic load balancer and depending on the request the load balancer will send requests to a selected node in a round robin fashion.

What would you do when there is an error occurs in a deployment like above where there are 13 different types of servers running in production and each of these servers are clustered and load balanced across 50+ servers?. This would be a nightmare for the system administrators to log-in into each server and grepping for the logs to identify the exact caused of the error. This is why distributed application deployment's  needs to keep a centralized application logs. These centralize logs should also be kept in a high scalable data storage in an ordered manner with easy access.So that the users (administrators,developers) can easily accesses  logs, whenever something goes unexpected, with the least amount of filtering  to pinpoint the exact cause of the issue.

When designing a logging  system like above, there are several things you need to consider.

1. Capturing the right information inside the LogEvent – You have to make sure all the information you need in order to monitor your logs is aggregated in the LogEvent. For example in a cloud deployment setup you have to make sure not only the basic log details(logger,date,log level) are not enough to  point a critical issue. You further needs tenant information (user/domain), Host information (to identify which node is sending what), Name of the server (from which server you are getting the log) etc. These information is very critical when it comes to analyzing and monitor logs in an efficient way.
2. Send logs to a centralized system in a nonblocking asynchronous manner so that monitoring will not affect the performance of the applications.
3. High availability and Scalability
4. Security – Stratos can be deployed and hosted in public clouds therefore, its important to make sure the logging system is high secured.
5. How to display system/application logs in an efficient way with filtering options along with log rotation.

Those are the 5 main aspects which were mainly concerned when designing the distributed logging architecture. Since Stratos support multitenancy we made sure that logs can be separated by tenants, services, and applications.

MT-Logging with WSO2 BAM 2.0

WSO2 BAM 2.0 provide a rich set of tool for aggregation, analyzing and presentation for large scale data sets and any monitoring scenario can be easily modeled according to the BAM architecture. We selected WSO2 BAM as the backbone of our logging architecture mainly because it provides high performance with non intrusiveness along with high scalability and security. Since those are the crucial factors essential for a distributed logging system WSO2 BAM became the idol candidate for MT-Logging architecture.

Publishing Logs to BAM  

We implemented a Log4JAppender to send LogEvents to bam. There we used BAM Data agents get Log Data across to BAM. BAM data agents send data using thrift protocol which gives us high performance message through put as well as it is non blocking and asynchronous. When publishing Log events to BAM we make sure the Data Stream is created per tenant, per server, per date. When the data stream is initialized there will be a unique column family created per tenant, per server per date and the logs will be stored in that column family in a predefine keyspace in cassandra cluster.

The Data stream defines the set of information which needs to be stored for a particular LogEvent and can be modeled into a Data Model.

Data Model which is used for Log Event.

{'name':'log. tenantId. applicationName.date','version':'1.0.0', 'nickName':'Logs', 'description':'Logging Event',

'metaData':[{'name':'clientType','type':'STRING'} ], 

'payloadData':[

   {'name':'tenantID','type':'STRING'},

   {'name':'serverName','type':'STRING'},

   {'name':'appName','type':'STRING'},

   {'name':'logTime','type':'LONG'},

   {'name':'priority','type':'STRING'},

   {'name':'message','type':'STRING'},

   {'name':'logger','type':'STRING'},

   {'name':'ip','type':'STRING'},

   {'name':'instance','type':'STRING'},

   {'name':'stacktrace','type':'STRING'}

 ] }

We extend org.apache.log4j.PatternLayout a in order to capture tenant information, server information and node information and wrap it with log4j LogEvent.

Log Rotation and Archiving

Once we send the log events to BAM the logs will be saved in a Cassandra cluster. WSO2 BAM provides a rich set of tools to create analytic and schedule task. Therefore, we used these hadoop task to rotate logs daily and archive them and store it in a secure environment. In order to do that we use a hive query which will run daily as a cron job. It will read Cassandra data store, retrieve all the column families per tenant per application and archive them in to gzip format.

The hive Query which is used to rotate logs daily

set logs_column_family = %s;

set file_path= %s;

drop table LogStats;

set mapred.output.compress=true;

set hive.exec.compress.output=true;

set mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec;

set io.compression.codecs=org.apache.hadoop.io.compress.GzipCodec;

CREATE EXTERNAL TABLE IF NOT EXISTS LogStats (key STRING,

payload_tenantID STRING,payload_serverName STRING,

payload_appName STRING,payload_message STRING,

payload_stacktrace STRING,

payload_logger STRING,

payload_priority STRING,payload_logTime BIGINT) 

STORED BY 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler' 

WITH SERDEPROPERTIES ( "cassandra.host" = %s,

"cassandra.port" = %s,"cassandra.ks.name" = %s,

"cassandra.ks.username" = %s,"cassandra.ks.password" = %s,

"cassandra.cf.name" = ${hiveconf:logs_column_family},

"cassandra.columns.mapping" = 

":key,payload_tenantID,

payload_serverName,payload_appName,payload_message,

payload_stacktrace,payload_logger,payload_priority,

payload_logTime" );

INSERT OVERWRITE  DIRECTORY 'file:///${hiveconf:file_path}' 

select 

concat('TID[',payload_tenantID, ']\t',

'Server[',payload_serverName,']\t',

'Application[',payload_appName,']\t',

'Message[',payload_message,']\t',

'Stacktrace ',payload_stacktrace,'\t',

'Logger{',payload_logger,'}\t',

'Priority[',payload_priority,']\t'),

concat('LogTime[',

(from_unixtime(cast(payload_logTime/1000 as BIGINT),'yyyy-MM-dd HH:mm:ss.SSS' )),']\n') as LogTime from LogStats

ORDER BY LogTime

Once we archived the logs we will send these archived logs to HDFS file system. The archived logs can be further analysed using map-reduce jobs, for long term data analytics

Advantages of sending Logs to WSO2 BAM

1. Asynchronous and None Blocking Data publishing
2. Receives and Stores Log Events Cassandra Cluster which is high scalable and a big Data Repository
3. Rich tools set for analytics
4. Can be shared with CEP for real time Log Event analysis.
5. Can provide Logging tool boxes and dashboards for system administrators using WSO2 BAM
6. High performance and non-intrusiveness
7. Big data analysis
1. Daily log information analytic - Analyse cassandra data storage
2. Long term log information - Analyse HDFS file system using map-reduce

Monitoring and Analyzing System Logs 

• Using the Log Viewer
  Both application and system logs can be displayed using the management console of a given product. Simply log-in to Management console and under monitor there are two links 1. System logs which has system logs of the running server 2) Application Logs which has application level logs (this can be services/web applications) for a selected application. This makes it easy for users to filter logs by the application they develop monitor logs up to application level.
• Dashboards and Reports
  System administrators can log-in to WSO2 BAM and create their own dashboards and reports, so the can monitor their logs according to their Key performance Indicators. For example if they want to monitor number of fatal errors occur per given month for a given node.
• SMS Alerts and Emails
  Not just dashboards and Reports ... Combining WSO2 BAM with WSO2 CEP you can get real time alerts like trigger emails, SMS so that System administrators can instantly get to know when your system is going through an unexpected behavior.

View Logs Using the Log Viewer - Current Log

View Logs Using the Log Viewer - Archived Logs

All these rich set of monitoring capabilities can be in built into your deployment using Stratos Distributed Logging system. Where you don’t have to worry about always going to the system administrators for logs whenever something goes wrong in your application :).

Validate Domain Ownership using StratosLive

In order to prove that your company is the owner of the domain, you need to validate the ownership of your domain. In < a href="https://stratoslive.wso2.com/home/index.html">WSO2 StratosLive there are two ways to validate your domain.

• Creating a text file in the domain web root
• Setting up a CNAME entry in your DNS.

From the above two methods creating a text file in the domain web root is the fastest way to validate your domain and you can do it within few seconds. If you proceed with the second option, it'll take up to 48 hours to reflect the added CNAME entry in your DNS. So in this knowledge-base, I'll explains how we can validate the domain ownership by adding a text file in the web root using WSO2 StratosLive.

Domain validation in stratos is easy. First login to WSO2 StratosLive as tenant admin using your account and go to Configurations -> Account -> Domain validation.

Under "Validate Domain Ownership", click on "Validate Now" and it will redirect to the domain validation page.

Then  you will get two options to validate your domain. Use the first option. And create a text file in your domain root call wso2multitenancy.txt. And add the given text content inside that text file you just created.

Click the "Validate" button, after you complete the above step. You will get a success message  for a successful validation. Click on continue to finish the validation process. After a sucessful validation, you will be redirected to Account management page with a message saying "Domain Validation Sucessful".

Expose your cloud data as RDF Resources

Since its all about semantic web 3.0 and RDF Data linking, I am going to explain about RDF data and exposing RDF data in the cloud space in 5 to 10 mins :) Just by using WSO2 Stratos Data Services Server.

The Resource Description Framework (RDF) is one of the most powerful technique to expose and interlink data(knowledge) in the decentralized world. It is also the latest trend in publishing and consuming linked data on the cloud therefore, lets discuss how we can expose our data as a RDF resource in the cloud using WSO2 Stratos Data Services Server.

1) use the RDF data model to publish structured data on the Web

RDF data model consist of set of statements which has a way of publishing link data on the web as triplets (with the use of subject predicate and object). In simple terms RDF model is a way of representing machine understandable data on the web as shown in the diagram below.

2. use RDF links to interlink data from different data sources

All things described by RDF are called resources, RDF links represents the linkage between one resource to another which is mainly done by the use of URIs.

-------------------Simple RDF file --------------

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns# xmlns:cd="http://www.product.fake/cd#">

<rdf:Description rdf:about="http://www.product.fake/cd/S10_1678 ">

<cd:productCode>S10_1678</cd:productCode>

<cd:productName>1969 Harley Davidson Ultimate Chopper</cd:productName>

<cd:productLine>Motorcycles</cd:productLine>

<cd:quantityInStock>7933</cd:quantityInStock>

<cd:buyPrice>48.81</cd:buyPrice>

</rdf:Description>

</rdf:RDF>

Now that we have a brief understanding on RDF and the importance of RDF data, lets see how we can generate RDF data source from a Google spread sheet.

First you need to create a google spread sheet of your choice which has some sensible information. To get the full usage of RDF you need to create several rdf

resource for the linking purposes however, for clarity purposes I will demonstrate how to create a single RDF resource and link it with an existing RDF resources.

Lets expose a google spreadsheet with product information on vehicle sales.

Product – Describe the currently available products in a car sale vendor.

ID	Model	Classification	Qty
S10_1678	1996 Moto Guzzi 1100i	Motorcycles	12
S10_1949	2003 Harley-Davidson Eagle Drag Bike	Classic Cars	23
S10_2016	1972 Alfa Romeo GTA	Motorcycles	18
S10_4698	1962 LanciaA Delta 16V	Motorcycles	15
S10_4757	1968 Ford Mustang	Classic Cars	13
S10_4962	2001 Ferrari Enzo	Classic Cars	12
S12_1099	1968 Ford Mustang	Classic Cars	4
S12_1108	2001 Ferrari Enzo	Classic Cars	10

Lets assume we have another set of RDF resources on product line ( which has information on each product line type) ie http://productLines/car , http://productLines/cycle, http://productLines/bus

Now lets create a data service to expose our Spreadsheet data as a rdf resource. In order to expose these data in the cloud you need to have a stratoslive account. Once

you create your stratos live account you can access set of stratos services such as Enterprise Service Bus, Application Server, Data Services etc (to try out stratos services you can easily create a demo account for free )

After creating your stratos account you can easily logged into your tenant domain and start working in the cloud!!!!

Now lets go back to exposing spreadsheet data as a service ... In order to do that we need to use WSO2 Stratos Data Services Server which provides a

powerful set of feature to expose data as a service and set of service utility methods. To access data services go to stratos live manager home page and click on wso2 stratos Data services.

To create a data service go to the left side menu bar and click on create under webservices->Add->Data Services. Then you will get a wizard as shown below. Give a proper data service name and click on next.

Once you click on next you will be directed to add data source page. And give information regarding the google spreadsheet you created along with your credentials

You can click on test connection to confirm your connection.

Click on next to go to the Query page. Query page describe the extracting algorithm to extract your data from the data source (google spreadsheet). Lets extract ProductID, Model,Classification and Qty.

Since our output is RDF result set, we need to specify our output type as RDF. RDF Base URI is the format of rdf:about URI which uniquely identifies each resource.

We will give RDF base URI as http://www.product/cd/{1}; this takes the Spreadsheet column 1(which is the ID) value for each row and replaces it for the RDF about attribute inside rdf:Description element

Output Type – RDF

RDF Base URI :- http://www.product/cd/{1}

Row namespace :- http://www.product/cd#

To generate the response in RDF format click on "Add New Output Mappings" button. There are two mapping types in RDF Output mapping. 1) as a element, 2) as a resource.When mapping an element as a resource, you need to give the resource URI along with the column name which needs to be mapped in curly brackets as shown below. This way we can link two RDF resources together and create a relationship between each other.

Lets map ID, Model and Qty as elements and Classification as a resouce, Lets link classification column to the productline resouces as i mention earlier ( http://productLines/car , http://productLines/cycle, http://productLines/bus )

Mappings of RDF resource

Resource URI http://productLines/{3} (as you can see we put the column 3 to get each classification type of the product).

Resouce Field Name - Classification

Mappings of RDF element

Following diagram shows the output mappings which we mapped from google spreadsheet to RDF resource.

Once we create the the query click on next to add Resources. Since we are exposing data as RDF resource we need to create a resource to expose the data. Lets give our query information when creating the resource.

Resouce Path – Products

Resource Method – Get

Query ID – RDFQuery

Click on finish to deploy the data service. Once you click on finish you can see your deployed data service under service list as shown below.

Now that we created our RDF resource we can test it by accessing it as a rest call or by using the try its feature.

Rest URL https://data.stratoslive.wso2.com/services/t/amani123.com/RDFDataSource/_getproducts (replace the tenant name amani123.com with your tenant domain)

You can validate this RDF resource by using the online RDF validator by copy pasting the rdf resource (right click on the page and view page source copy paste it inside the validation)

Now we exposed our spreadsheet data in the cloud space just within 10 mins :) you can create more rdf data sources using the same manner with different data sources (csv/excel/rdbms) and expose those data as RDF data sources. I will further explain how we can extract RDF data using SPARQL in my next blog post :)

Kick start on WSO2 Stratoslive

Since we are going to see lots and lots of WSO2 Stratos related demonstrations and tutorials and powerful capabilities of cloud computing and stratos, its very important to know how to get started in WSO2 Stratos .

Getting started easy!!! all you need to do is create a StratosLive account (which is TOTALY FREE) and you can start using wso2 stratos live services right away.

Lets see how we can create a stratoslive account.

First go to the stratoslive homepage.

   

To create a new account click on “Get Started Now for Free! ” . It will redirect to a page where you need to give your Domain Information, Usage plan Information, and Contact Details.

To created a domain (a user account/ tenant) you need to give a unique tenant domuain. You can test

the uniqueness of your domain by clicking “check availability” button.

Once you give your domain name you need to select the usage plan for your domain ... there are four types of usage plans you can click on pricing details to get ll be allocated for you. For exploring StratosLive features and understanding the power of cloud computing you can select the demo version (Which is the free version).

After that you need to give your contact details for the registration purposes. Give your personal information along with your tenant-admin user name password and the verification code.

Once everything is filled properly submit the document and you will be redirected to a success page.

To activate your tenant/domain you will get a email with infomation on how you can activate your account. Once you activate you can login to your account simply by giving the username@tenantdomain and the tenant admin password you gave while registering your tenant.

When you login it will redirect to Stratos Manager home page which has all the links to StratosLive services.

To get more information regarding these services click on help it will give a brief description about each service.

To try out services by clicking on each service and have fun with cloud computing!!