Real time log event analysis with WSO2 BAM(CEP features)

One of the major problems when it comes to managing/monitoring distributed systems is not being able to detect when your system is giving issues all of a sudden. Lets say you have like 100 servers and you need to detect when a fatal error appears in the system and you want to act right away. This is nearly impossible if you do not have proper monitoring tools.
WSO2 BAM with WSO2 CEP features has the perfect mechanism to monitor real time logs and alert it to relevant parties when an expectation/ suspicious error occurs.

In this sample I will be using WSO2 Appserver and WSO2 BAM in order to demonstrate the monitoring and alerting capabilities of WSO2 BAM 2.4.0.

• WSO2 Appserver will be used to send log events to BAM.
• WSO2 BAM will do all the monitoring and alerting when logs are captured.

Send Logs to BAM

To send logs all you need to is go to WSO2_AS_HOME/repository/conf/log4j.properties and add LOGEVENT to the root logger. And start Appserver. (please make sure your BAM server is up and running before starting the Appserver)

To test if the logs are successfully sent to BAM, you can log in to cassandra explorer and see if there is a new column family created under EVENT_KS keyspace

ie log.AS.0.2013.1.12

Configuring WSO2 BAM for REAL TIME analytics

In this demo, I will be sending an email if an ERROR log occurs in WSO2 Appserver.  Since we will be using mail transport in BAM to send email alerts to recipients, we have to enable mail transport in BAM. To do that go to repository->conf->axis2->axis2-client.xml and add email configurations.

<transportSender name="mailto" class="org.apache.axis2.transport.mail.MailTransportSender">

       <parameter name="mail.smtp.from">wso2esb.mail@gmail.com</parameter>

       <parameter name="mail.smtp.user">wso2esb.mail</parameter>

       <parameter name="mail.smtp.password">wso2mail</parameter>

       <parameter name="mail.smtp.host">smtp.gmail.com</parameter>

       <parameter name="mail.smtp.port">587</parameter>

       <parameter name="mail.smtp.starttls.enable">true</parameter>

       <parameter name="mail.smtp.auth">true</parameter>

   </transportSender>

Here you can give your own email configurations. 

Restart/Start the BAM server.

Assuming the logs are getting published to BAM, lets see how we can capture these log events for real time analytics.

Step 1 - Creating Event Adapters

In order to do real time analytics we need to create an execution plan. For that we need two input adapters.

1. Input adapter - to capture log events coming to BAM (in  this case it will be a wso2event).
2. Output adapter - to send emails

To create adapters you need to login to BAM management console. Under configurations, there will be "Event Processor Configs" and add input event adapters and output event adapters as shown below.

Input Event adapter

Output Event Adapter

Step 1 - Creating Stream Definitions

Now that we have created event adapters, we need to create a stream definition to capture the LogEvent in order to do complex event processing. In the appserver log event, these are the attributes we have.

• Meta Data
• clientType {String}
• Payload Data
• tenantID  {String}
• serverName {String}
• appName  {String}
• logTime  {Long}
• priority {String}
• message {String}
• logger {String}
• ip {String}
• instance {String}
• stacktrace {String}

To create the stream definition, go to main tab and under create stream definition you can create the log event stream as shown below.

Step 1 - Creating the Execution Plan

In the execution plan we will be specifying the input stream and writing a CEP Query (SQL Like Query)  for the event stream.

Go to Create Execution Plan, and give suitable name for the execution plan. Select the needed stream and give an alias. Click on import after selecting the stream. In our CEP Query we will analyze events and if an event has an error, we will send it to an output stream.

CEP QUERY

from LogEvents[priority == "ERROR"]

select message,stacktrace,serverName

insert into ExceptionStream

After creating the Query, we need to add the exported stream (which is the stream that we are sending Error logs) according to our CEP querry exported stream name should be ExceptionStream give the value of the exported stream name and select "Create Stream Definition" to create the ExceptionStream. This stream will be auto generated by looking at the CEP as shown below.

Once we create the Exception stream select the Exception stream as exported stream and create a new formatter. This formatter is used to specify the email body and email related information like the subject, to address ect.  Give the output mapping type as text so we can give the content of the email message body inline. 

Email Body

Error Occurred in {{serverName}} – {{message}}

{{stacktrace}} 

In this body we are taking message, stacktrace and server name from the OutputStream (ExceptionStream) and adding a readable message for the email message body.

Add the event formatter and save the execution plan. Now we have successfully created the event trigger to monitory error logs for wso2 appserver. You can test this by invoking a service with an error.

If you want more in depth information on real time log event analytics you can follow the following screen cast for more details