Real time log event analysis with WSO2 BAM(CEP features)

One of the major problems when it comes to managing/monitoring distributed systems is not being able to detect when your system is giving issues all of a sudden. Lets say you have like 100 servers and you need to detect when a fatal error appears in the system and you want to act right away. This is nearly impossible if you do not have proper monitoring tools.
WSO2 BAM with WSO2 CEP features has the perfect mechanism to monitor real time logs and alert it to relevant parties when an expectation/ suspicious error occurs.

In this sample I will be using WSO2 Appserver and WSO2 BAM in order to demonstrate the monitoring and alerting capabilities of WSO2 BAM 2.4.0.

• WSO2 Appserver will be used to send log events to BAM.
• WSO2 BAM will do all the monitoring and alerting when logs are captured.

Send Logs to BAM

To send logs all you need to is go to WSO2_AS_HOME/repository/conf/log4j.properties and add LOGEVENT to the root logger. And start Appserver. (please make sure your BAM server is up and running before starting the Appserver)

To test if the logs are successfully sent to BAM, you can log in to cassandra explorer and see if there is a new column family created under EVENT_KS keyspace

ie log.AS.0.2013.1.12

Configuring WSO2 BAM for REAL TIME analytics

In this demo, I will be sending an email if an ERROR log occurs in WSO2 Appserver.  Since we will be using mail transport in BAM to send email alerts to recipients, we have to enable mail transport in BAM. To do that go to repository->conf->axis2->axis2-client.xml and add email configurations.

<transportSender name="mailto" class="org.apache.axis2.transport.mail.MailTransportSender">

       <parameter name="mail.smtp.from">wso2esb.mail@gmail.com</parameter>

       <parameter name="mail.smtp.user">wso2esb.mail</parameter>

       <parameter name="mail.smtp.password">wso2mail</parameter>

       <parameter name="mail.smtp.host">smtp.gmail.com</parameter>

       <parameter name="mail.smtp.port">587</parameter>

       <parameter name="mail.smtp.starttls.enable">true</parameter>

       <parameter name="mail.smtp.auth">true</parameter>

   </transportSender>

Here you can give your own email configurations. 

Restart/Start the BAM server.

Assuming the logs are getting published to BAM, lets see how we can capture these log events for real time analytics.

Step 1 - Creating Event Adapters

In order to do real time analytics we need to create an execution plan. For that we need two input adapters.

1. Input adapter - to capture log events coming to BAM (in  this case it will be a wso2event).
2. Output adapter - to send emails

To create adapters you need to login to BAM management console. Under configurations, there will be "Event Processor Configs" and add input event adapters and output event adapters as shown below.

Input Event adapter

Output Event Adapter

Step 1 - Creating Stream Definitions

Now that we have created event adapters, we need to create a stream definition to capture the LogEvent in order to do complex event processing. In the appserver log event, these are the attributes we have.

• Meta Data
• clientType {String}
• Payload Data
• tenantID  {String}
• serverName {String}
• appName  {String}
• logTime  {Long}
• priority {String}
• message {String}
• logger {String}
• ip {String}
• instance {String}
• stacktrace {String}

To create the stream definition, go to main tab and under create stream definition you can create the log event stream as shown below.

Step 1 - Creating the Execution Plan

In the execution plan we will be specifying the input stream and writing a CEP Query (SQL Like Query)  for the event stream.

Go to Create Execution Plan, and give suitable name for the execution plan. Select the needed stream and give an alias. Click on import after selecting the stream. In our CEP Query we will analyze events and if an event has an error, we will send it to an output stream.

CEP QUERY

from LogEvents[priority == "ERROR"]

select message,stacktrace,serverName

insert into ExceptionStream

After creating the Query, we need to add the exported stream (which is the stream that we are sending Error logs) according to our CEP querry exported stream name should be ExceptionStream give the value of the exported stream name and select "Create Stream Definition" to create the ExceptionStream. This stream will be auto generated by looking at the CEP as shown below.

Once we create the Exception stream select the Exception stream as exported stream and create a new formatter. This formatter is used to specify the email body and email related information like the subject, to address ect.  Give the output mapping type as text so we can give the content of the email message body inline. 

Email Body

Error Occurred in {{serverName}} – {{message}}

{{stacktrace}} 

In this body we are taking message, stacktrace and server name from the OutputStream (ExceptionStream) and adding a readable message for the email message body.

Add the event formatter and save the execution plan. Now we have successfully created the event trigger to monitory error logs for wso2 appserver. You can test this by invoking a service with an error.

If you want more in depth information on real time log event analytics you can follow the following screen cast for more details

[Screencast] Monitoring system logs with WSO2 Business Activity Monitor

This screencast explain how you can do your enterprise monitoring using BAM and CEP features.

Overview of the screencast

  * Send Log Events to BAM/CEP

  * Create analytics to analyse logs daily( can be scheduled accordingly)

  * Create gadgets with  log statistics (charts/tables)

  * Monitor logs real time with CEP (send alerts to relevant parties when an error log appears.)

Eventing using WSO2 DSS - Send data to a JMS Queue for a given event trigger

If you want to trigger an event depend on an SQL related service call this is a great example on how you can do it using WSO2 Data Service Server.

In this sample I am going to explain how you can handle event triggers in WSO2 Data service server.  In WSO2 DSS, you have two types of event triggers.

1. Input event triggers - which trigger's an event looking at the request.
2. Output event triggers - which trigger's an event looking at the response.

For both of these trigger's you can specify an ex-path expression to trigger the event.

To demonstrate this functionality I  am going to write a small data service which insert Student information. And every time an insertion happen, primary key of the inserted record will be send to a JMS Queue.

Prerequisites

In order to try out this sample you need to have
  -> WSO2 Data Services Server Downloaded
  -> You need to have a database of your choice
  -> Add related drivers to DSS_HOME/repository/component/lib folder
  -> ActiveMQ downloaded and started

Step 1 - Enable JMS senders in WSO2 DSS.

In order to send JMS messages from data services, you need to enable the JMS sender in data services server.
To enable JMSSender -> Go to DSS_HOME/repository/conf/axis2/axis2.xml and un-comment the following line.

<transportSender name="jms" class="org.apache.axis2.transport.jms.JMSSender"/>

and then start the data services instance.

Step 2 - Create data Service.

If you are new on creating data services, please refer the following post on how to create a data service using WSO2 Data Service Server.

I have a small table called Student. Lets create a simple data service for the table student.

describe Student;
+---------+-------------+------+-----+---------+----------------+
| Field   | Type        | Null | Key | Default | Extra          |
+---------+-------------+------+-----+---------+----------------+
| sid     | int(11)     | NO   | PRI | NULL    | auto_increment |
| name    | varchar(90) | YES  |     | NULL    |                |
| address | varchar(75) | YES  |     | NULL    |                |
| country | varchar(75) | YES  |     | NULL    |                |
| phone   | varchar(20) | YES  |     | NULL    |                |
| major   | varchar(50) | YES  |     | NULL    |                |
| gpa     | float       | YES  |     | NULL    |                |
| tutorid | int(11)     | YES  |     | NULL    |                |
+---------+-------------+------+-----+---------+----------------+
8 rows in set (0.00 sec)

Log in to DSS management console -> Create New Data Service. Give the data service name appropriately and click on next.

Click on Add Data Source and give database configuration appropriately and click on next.

Next section we will discuss how we can write a data service Query which can invoke a trigger according to the query result.

In this Query my SQL is simple insert

INSERT INTO Student(name,address,country,phone,major,gpa,tutorid) VALUES(:name,:address,:country,:phone,:major,:gpa,:tutorid)

Click on "Generate Input Mappings", and this will generate the input mappings for the input parameters.

Since we need a response back (ideally it will be the primary key of the newly inserted record) we can click  Return generated key which will auto generate the response.

To add the event trigger, scroll down a little bit. Under Events, click on Manage Events. There, we give the eventing configurations.

Event ID - student_addition_trigger

Xpath - //*[local-name()='ID' and namespace-uri()='http://ws.wso2.org/dataservice']>0

Target Topic - student_insertion_topic

Event Sink URL ( JMS URL according to your activeMQ)

jms:/student_insertion_topic?transport.jms.DestinationType=queue&transport.jms.ContentTypeProperty=Content-Type&java.naming.provider.url=tcp://localhost:61616&java.naming.factory.initial=org.apache.activemq.jndi.ActiveMQInitialContextFactory&transport.jms.ConnectionFactoryType=queue&transport.jms.ConnectionFactoryJNDIName=QueueConnectionFactory

Go to Main Configuration and add the Output trigger as shown below.

Now we have done with our Query Section. Click on Save to save the Query and Click next to add the  operation.
Add New Opperation ->
Operation Name* - InsertStudent
Query ID* - insertQ

Save and finish.

Now we can test this functionality by using the try it function provided by Data Service Server

My Request

   <body> <p:InsertStudent xmlns:p="http://ws.wso2.org/dataservice">
      <p:name>Amani</p:name>
      <p:address>Soysa</p:address>
      <p:country>SL</p:country>
      <p:phone>1923123111</p:phone>
      <p:major>SE</p:major>
      <p:gpa>4.2</p:gpa>
      <p:tutorid>12</p:tutorid>
   </p:InsertStudent>
</body>

My Response

   <GeneratedKeys xmlns="http://ws.wso2.org/dataservice"> <Entry>
      <ID>5</ID>
   </Entry>
</GeneratedKeys>

If you go to your active MQ console, You would see number of pending messages as shown below. You can go inside your Queue and explore more!!!!