The building blocks for Complex Event Processing

By David Worthington

September 11, 2009 —

With so much computation happening in memory, designing a Complex Event Processing (CEP) application can be complex indeed. CEP is up-and-coming as an application use, said Lab49 managing consultant Joe Morrison. It is applied to many areas in financial services and middleware office applications, in addition to keeping tabs on latencies in high performance applications, he said.

A "building block" methodology for application development makes it easier to incrementally build features and expand functionality down the line, said John Morrell, vice president of marketing at Aleri, which sells a CEP platform consisting of authoring tools and middleware. By using a modular approach, Morrell said, developers can keep a standardized, steady data interface between components and adjust components without impacting others.

Aleri details five building blocks for CEP development. The first two deal with creating high-speed data streams and preparing that data for analysis; and integrating event stream data with historical and reference data.

They then progress to performing aggregations and computations on data; identifying business events in analysis and mapping decision to actions, then generating outputs or events for downstream applications and users; and generating multi-dimensional analytic results. Application state management is also overviewed.

"We wanted to tell people that you don't have to be afraid of CEP,” Morrell said. Many customers take approaches that are similar to what they do for producing offline analytics, such as capturing, cleansing and preparing data, he explained. However, there are key differences that developers should know.

Aleri in late July began releasing “best practices” Webinars and white papers for the design of CEP applications.

Planning ahead
A common mistake made by developers is that they start out building applications without knowing the results that they are looking for, said Morrell. "The most successful CEP implementations have been where they understand a business process from the top down, and adjust the process itself to deal with things faster and receive more information in a timely fashion."

It is also important to pick the right tools. Developers should decide whether visual or contextual programming tools are a better solution for their project, said Morrison. "We've found that a visual programming model is nice for straightforward applications, but more-complex applications require contextual languages, which are more powerful and expressive."

Design could also be dictated by whether an application needs to call out to C++, Java or .NET code to integrate with external systems, he said. "Some vendors are more Java-oriented, and some are more C++-oriented."

If an enterprise is already using a business intelligence or data warehousing solution for their data visualization layer, it should consider using pre-integrated CEP solutions that are available, integrated and optimized for that platform, said Forrester senior analyst James Kobielus.

IBM and TIBCO Software offer CEP middleware for their Cognos and Spotfire products, and other companies that do not have CEP capabilities, including SAP, will reference customers to partners, he said.

Another concern is that many CEP vendors use their own proprietary languages to query events that developers have to learn, Kobielus said. "It is important to consider CEP offerings that integrate and present an abstraction layer to leverage SOA and SQL. I would think long and hard about committing to them if they don't have [the abstraction layer]."

An organization should determine whether it really does require sub-second latency before settling on a CEP solution, he said. If you need true sub-second latency, CEP is the way to go, but if you don't, existing business activity monitoring data warehouse middleware can provide near real-time features, he explained.

Many BAM (business activity monitoring) software makers also have some degree of CEP capability embedded into those environments, said Kobielus. "Vendors are considering more in-memory architecture and memory virtualization across disparate platforms. The industry is groping for the right programming model to enable that."

Electronic Data Interchange (EDI), which is computer-to-computer interchange of strictly formatted messages, also addresses some of the same middleware requirements for generating analytics as CEP, Kobielus said.

Avoiding common errors
If a CEP solution is the right solution for your organization, a little knowledge can go a long way toward steering a project away from potential disaster.

CEP applications should be created in an isolated environment and be unit-tested frequently, Lab49's Morrison said. "Control inputs, and have outputs work as expected. It's a lot of work to set up the environments. There should be light testing of the entire applications, end-to-end."

Regression testing should also be planned in advance, he said. He added that particular attention should be paid to failure "around the edges" of the system, such as when a database or message bus becomes available. "How does the system respond to that? People don't think of that, and it's hard to do retroactively."

State management must also be dealt with up front to lessen the risk of data loss, Morrison said. "The old-fashioned, database-oriented way is slow, but it has the virtue of persisting data for shutdown and start, so it is easy to resume."

In contrast, CEP has less transparency because more data is processed in memory, making it more difficult to persist in state. He explained that developers should start out asking what would happen if someone first pulled the plug on a CEP application and later resumed it.

Another common mistake is attempting to do too much in memory. Developers should consider limiting the use of custom operators for joining and matching data to avoid creating bottlenecks in CEP applications or creating "huge" memory usage, Morrison said. "It's nice to call out to custom code, but be careful: It could mess up the program."

Anything that takes a long time to calculate, or joins data across multiple event streams, could also lead to a situation where resources are over-utilized, he added. "It's easy to write a program that looks innocent but at runtime consumes ridiculous amounts of memory."

Sometimes it is preferable to perform long-running calculations outside of a CEP program by sending the message to an output queue and retrieving it after work is done, said Morrison. Another workaround is to introduce an external cache for join operations, he added. "CEP and caching products work well together, and some vendors bundle them together."

Additionally, most SOA vendors will continue to strengthen SOA abstraction layers to enable CEP, said Forrester's Kobielus. The industry is taking the "next step" from storage virtualization to memory virtualization for analytics in "massive parallel environments."

Related Search Term(s): CEP

Share this link: http://www.sdtimes.com/link/33752

TIBCO adds CEP features to BusinessEvents BusinessEvents goes into the mainstream with the addition of various CEP features for editing business rules and querying events. A new UI and a language similar to Object Query Language help facilitate these changes.

TIBCO develops hardware for faster messaging In the first hardware release in company history, TIBCO says it can boost messaging speeds while reducing the number of servers required for messaging infrastructure to 10% of what was previously needed.

Coral8 provides customizable analytics in real time Version 5.5 of the complex event processing software also adds filtering, querying and sharing options for downstream clients. An analyst describes Coral8's product as "third-generation" CEP software because of its mixture of CEP and business intelligence features.

Add comment

Name*
Email*
Country

Comment
Preview

News on Monday
more>>
SharePoint Tech Report
more>>

Download Current Issue
ISSUE 3/15/2010 PDF

Need Back Issues?
DOWNLOAD HERE

Receive the print Edition?

Google Code turns 5
Google Code Turns 5, and adds a Paxos Algorithm to make the system more stable and reliable.
03/17/2010 11:16 AM EST

Test your Visual Studio 2010 know-how
Microsoft is offering free beta certification exams for Visual Studio 2010.
03/17/2010 11:08 AM EST

Microsoft lifts the hood on IE9
Microsoft is previewing IE9.
03/16/2010 01:10 PM EST