Complex Event Processing (CEP) Blog

Location awareness and Twitter

Paul Vincent — Wed, 10 Mar 2010 19:20:49 +0000

Thanks to TIBCO’s Patrick Sapinski for some news on a CEP-driven Twitter application (albeit not public yet) that used location awareness principles to “tweet” people depending on where they were (e.g. passing a certain location). The Twitter side of the story was apparently the easy part. Patrick writes:

“We did not do any Java coding to accomplish this. We just used the TIBCO BusinessWorks HTTP activity to send tweets to Twitter. It was pretty easy once I configured the messages, URL, etc.”

Of course, for corporate applications / private tweets there are also services like Tibbr, which I signed up for this week… meanwhile I’m happy sticking with iGoogle to monitor “the real thing” via RSS feeds!

Up-Scale Your Apps with Distributed Caching

Paul Vincent — Wed, 10 Mar 2010 18:45:53 +0000

… was the subject of today’s Forrester-IBM webinar on distributed cache technology, with both Forrester and IBM citing CEP and EDA as users for this technology, amongst others. The overriding driver for this tech being eXtreme Transaction Processing, which we might just refactor as eXtreme Event Processing for the purposes of this blog!

One minor quibble: John Rymer of Forrester did the introduction and during so classified the cache market as .NET, Java and NoSQL camps, with TIBCO placed in the Java camp. This might seem a fair classification of a complex market area, but of the 2 relevant TIBCO distributed cache offerings:

TIBCO BusinessEvents, although Java-based, is more accurately described as a CEP product that embeds a distributed cache - it wouldn’t normally appear on a vendor list of distributed cache technologies;
TIBCO ActiveSpaces is more accurately described as a data grid, but has .NET, C and Java interfaces. I’m sure other caching / data grid products have similar multiple interfaces - after all the client is just “an interface” to the cache / grid.

Spare a thought in passing, though, for the OODBMS guys. Amongst this buzz about data grids and caching, I notice the Forrester blog is reporting that the Progress guys (disclosure: a TIBCO competitor in some areas) are now considering their ObjectStore OODBMS a “legacy platform”. Plus ça change, perhaps.

How does CEP fit into BPM and SOA environments?

Paul Vincent — Fri, 05 Mar 2010 00:47:39 +0000

There were some customer discussions recently on the conceptual relationships between the CEP and BPM and SOA “software stacks”. This coincided with the announcement of the OMG-backed Event Processing Consortium being set up (alongside the merger of the SOA and BPM consortia), events which themselves can be interpreted as that event processing has some special role to play alongside BPM and SOA.

[Disclosure: note TIBCO is not a member of either the OMG's EP or BPMSOA consortia - they seem to be focused on end-user rather than vendor participation - but is an OMG member and participant in standards development. We currently see the EPTS as the main advocacy group across vendors, academics, analysts and end-users, but will be monitoring the progress of the EP Consortium].

So here are a few simplistic patterns on how CEP (event processing) relates to BPM (processes) and SOA (services)…

CEP Pattern	Example	Diagram
1a. Standalone CEP This is a bit of a misnomer - you don’t identify event patterns without some intent to use them, and at the very least such a use would be in a BAM type role displaying interesting correlations for some business person - who of course is engaged in some kind of business process, which may or may not be managed by BPM…	Monitor a production process to provide an “additional view” or dashboard for the process control manager.
1b. CEP enriching BPM processes and/or SOA services. This is the conventional view of CEP, detecting the complex events that are of interest to, and useful in, appropriate processes and services. Complex events in these cases can be as straightforward as deducing that a deliverable has been completed, or some process truly initiated. Typically the CEP system is transforming source events into business events, for onward use in (the) business processes.	Identifying exception events in a business that need to handed to a workflow or case management system for resolution.
2a. CEP monitoring processes and services This is where the sources of events are the managed processes and services themselves. This process and service monitoring is used to detect exceptions, disparities across systems, and system performance… Note that effectively this pattern is a combination of patterns 1a and 1b above.	Detect when response times exceed some metrics and suggest corrective actions such as reallocating resources.
2b. CEP-based decisions for processes and services This is where I need to make intelligent decisions for the process and service layers, using the CEP layer as a monitoring, shared decision management component. Note that effectively this is a slight extension of pattern 2a above.	A BPMN “rule activity” sends a decision request to the CEP engine to get a valid decision for a process decision point; the CEP engine monitors the decisions made.
3a. Dynamic process and service control This is where the events from processes and services, and external services, are combined to select which processes and services are relevant to use for the current context. In effect, the CEP engine becomes the controlling agent for the business processes and service engine, handling for example dynamic process selection. Note that this pattern is a further evolution of patterns 2a and 2b.	In a complex business process for ever-changing fulfillment problems, CEP-based rules determine which sub-processes are valid based on incoming information.
3b. Embedded processes and services within CEP The final evolution of the above is when you argue that the functions of the BPM and SOA stacks can be subsumed into the CEP layer. In reality this is usually only a partial subsumption, as otherwise the centralization of services into just 1 layer could be perceived as contrary to the very idea of SOA! So this covers things like event-based policy implementations being embedded as CEP rules rather than as external services, but alongside some external services such as an operational database. Indeed, one could argue that in this case the CEP event processing agents are themselves really part of the SOA layer, not the other way round!	A service gateway controlling access to existing services, but embedding decisions, service policies, and business rules.

So, did I miss anything?

Complex Event Processing: a technology evaluation check-list

Paul Vincent — Thu, 04 Mar 2010 19:43:54 +0000

One of the challenges for organisations investigating event processing technologies is “how do we know what is important”? Obviously everyone will have different views on this. Nonetheless, as TIBCO has the most experience in CEP across multiple domains and the largest customer base for a CEP technology tool, it seems to make sense to present a sample evaluation guide that shows the typical considerations and features needed.

First, a few comments. TIBCO deals with enterprises who can have sophisticated requirements around their complex event processing needs. TIBCO customers generally use CEP for operational intelligence applications covering anything from business activity and event monitoring to automated intelligent business processes. Hence TIBCO solutions handle high throughput (not just latency) and payload size (business messages can get very large), whilst providing the fault tolerance and resilience capabilities that 24×7 organizations require. But every class of CEP application has different requirements: for example an algorithmic trading solution might prioritise ultra-low-latency over other features like failover-support and logic and payload complexity. Ergo, for every use case the “example scoring factor” provided below will need to be adjusted to match the feature’s importance to the use case.

Feature	Detail	Reason	Example Scoring Factor
1. Event Channels	A. Specific channel types required	Example: JMS, MQ-Series, HTTP…	10
	B. Custom channels	Example: custom event type for SCADA	3
2. Modelling	A. Event models	Event metadata, hierarchy, inheritance	10
	B. Event object / concept models	Object history, hierarchy, inheritance	10
	C. State and status of complex events	State model and flow	8
3. Processing Elements and Expressions	A. EventConditionAction rules	Basic event filters, aggregations	10
	B. Inference rules	Intelligent event processing	6
	C. Continuous queries	Event streaming aggregations, statistics	6
	D. Temporal regular expressions / logic	Concise pattern recognition	6
	E. Specialist analytics and statistical functions	Capabilities for pattern discovery, possibly in conjunction with developer interface (see 7)	6
	F. Other context support	How to handle context across time, such as GVs etc	6
	G. Extensibility	How to add custom functions / algorithms / analytics?	6
4. Event augmentation	A. Database access	JDBC read/update etc	4
	B. SOA access	Invocation of existing services	6
	C. BPM / workflow access	Invocation of manual steps	3
5. Stateful processing and management	A. Event storage in-process	In-memory support	8
	B. Event storage out-of-process	Low latency cache and/or high latency backing store	8
6. Platform capabilities	A. Scalability via distributed processing agents	For large processing loads (logic and/or event size and/or rate)	8
	B. Scalability via distributed event cache	For supporting event patterns over processes, resilience	8
	C. Failover / resilience features	Hot standby and event handling if agent fails	8
	D. Hot deploy / event-based updates	Hot deploy of new rules, queries or patterns, or dynamic update through events	8
	E. Operations and Management	UI and/or reports to monitor performance and/or adapt capabilities	8
	F. Performance supporting features	Components to minimise latency like preprocessor, algorithms etc	10
	G. Deployment environment support	Desired or preferred hardware/OS/cloud platforms	8
7. Development Environment	A. Standardised GUI tool	Example: Eclipse-based	8
	B. Verification tools	Logic verification such as diagram generation and reports	6
	C. Validation tools	Unit and bulk test capabilities	6
	D. Debugging	Local / remote debugging capabilities	6
	E. Team features	SCCS / project repository / team development	4
8. Business interface	A. Business Event / Rule / Decision Management	Management (editors, impact analysis) for business-controlled event processing aspects	6
	B. User Workflow / control	Control and security for business updates to event processing	6
	C. Event monitoring dashboard	Display of events, and complex events such as KPIs	6
9. Project considerations	A. Track record	Numbers of Customers / users in same domain	6
	B. Availability of SIs and development partners	Available assistance if needed	2

One final checklist or scorecard item cannot be added to the list above, but is at least as important as the technology factors. This is the client-vendor relationship, and I was reminded of this when another CEP vendor tweeted recently about how they had been gaining business partly due to another vendor’s “aggressiveness” in business dealings. So, just like everywhere, caveat emptor…. and maybe add a higher score to 9.A. above!

As usual, comments, feedback and suggestions welcome.

Analytics moving to Real-time via CEP?

Paul Vincent — Tue, 02 Mar 2010 13:33:29 +0000

Two events last month showed indicators of a convergence between the analytics world and CEP world.

Firstly Louis Bajuk-Yorgan from TIBCO Spotfire attended the Predictive Analytics World conference in San Francisco. He reported that:

Three key themes showed up multiple times throughout the talks-the growing importance of text mining, the application of net lift modeling to determine the real results of a marketing campaign (ignoring those people who would have bought anyway), and (most interesting to me) the importance of operationalizing predictive analytics.

In his opening keynote speech, Eric Siegel (the conference chair) saw the most important innovation in the field of Predictive Analytics focused on applying predictive analytics to operational decisions (as opposed to more established application areas such as customer churn & product recommendations). In a later talk, James Taylor of Decision Management Solutions (and co-author of the great book “Smart (Enough) Systems”), echoed Eric’s emphasis on operational results, encapsulated in the phrase “Action support, not just decision support.” James advised building an analytic platform that focused on the end game: the need to operationalize analytic decisions.

This is great validation for us, since operationalizing analytics is at the heart of TIBCO’s vision for its combined platform with Spotfire and S+ (as shown in products like Operations Analytics).

Then a week later Andreas Gerst from the TIBCO BusinessEvents team presented at cepconf in Munich, Germany. Andreas presented on CEP and Data Mining, and in particular how both these complement each other for advanced operational intelligence around customer management. Andreas used TIBCO BusinessEvents and TIBCO Spotfire Miner as his example technologies, mentioning techniques like PMML for moving from analytics to real-time event processing technologies.

Processes = event-driven decisions and actions?

Paul Vincent — Wed, 24 Feb 2010 21:08:03 +0000

In a previous post we commented on IBM’s Conceptual Model for Event Processing Systems, but failed to observe that IBM’s rule engine group, Ilog, was noticably absent from its contributor list. So perhaps it was both fortuitous and timely that Ilog’s Daniel Selman recently added another “viewpoint” on the position of event processing (aka the rules viewpoint).

Daniel’s take on the “primary users (sic) of rules technology” are the (use cases for) automating decisions, event responses, processes and inferences. I think these might be better classified by renaming then as decision processing, event processing, and business process processing. But inferencing, which Daniel notes as being a technology to support Artificial Intelligence, is not so much a (user or) use case, but a means of providing knowledge-based reasoning to support any of decisions, processes, etc - “AI” is not (or should not be) a means unto itself.

Standard Event Processing Design Pattern

Indeed, one could probably argue that:

all business processes are driven by events of some kind, and involve decisions and (re)actions;
all processes, decisions and event processing are context (i.e. state) driven; however, some processes and algorithms are used in a subordinate fashion to a stateful process (consider a typical decision service that is subordinate to its application server and database layers);
inferencing can be used to enhance any part of the event-handling process as a form of declarative rule control.

In TIBCO’s experience, an event driven rule engine (like TIBCO BusinessEvents) can be used to provide dynamic business processes, event-driven decisions, and real-time control mechanisms - and often all 3 - exploiting and building on the fundamentals of complex event processing.

So, on the utility of rule engines, we totally agree with Daniel!

IBM’s Conceptual Model for Event Processing Systems

Paul Vincent — Mon, 22 Feb 2010 15:14:14 +0000

Nice to see a paper on IBM Developworks on some of the theory of “event processing”, authored (amongst others) by the Chair of the Event Processing Technical Society, IBM Labs’ Opher Etzion, and fellow EPTS Reference Architecture team member Catherine Moxey. The paper is of course somewhat of a “broad brush” view of event processing, as indeed you might expect from a team representing such a diverse range of event-handling software as Tivoli, CICS, MQ and so forth from the broad (and, from a TIBCO perspective at least, heavyweight ) IBM software stack.

Some thoughts:

The paper implies a need to model, in any significantly large event processing system, the types and roles of the event processing agents and how they interact. Some such agents will be tightly coupled (such as co-operative rulesets in a TIBCO BusinessEvents inference agent), whereas others will have more identifiable interfaces and roles (such as separate inference and query agents in a distributed TIBCO BusinessEvents application). Tables 6 and 7 for example are almost metamodel definitions for an event processing agent.
Possibly a better name for the paper would be “A Conceptual Model for Event Processing Networks“, given the emphasis on event pathways through processing agents.
IBM Developerworks (like this blog) has an unattributable “scoring mechanism”, and shows today that one reader was unimpressed enough to profer a low score - but why? I also can’t explain why this is described as aimed at an “intermediate” audience, as the principles seem basic and well explained…

TUCON 2010 update: Reliance Communications and SouthWest Airlines

Paul Vincent — Fri, 19 Feb 2010 19:07:02 +0000

Interesting to see 2 of the 4 keynotes at TUCON this year (May 10-13, Vegas) are complex event processing customers: Dr. Sumit Chowdhury, CIO, Reliance Communications and Jan Marshall, VP of Technology and CIO, Southwest Airlines. The TUCON theme this year is “Discover the technology of Now”, but perhaps it should have been “Operational Excellence via Operational Intelligence”!

More support for the Event-based Business Process

Paul Vincent — Thu, 18 Feb 2010 21:12:02 +0000

I enjoyed the CIO UK article (mis)titled “Why BPM Should Be On The CIO’s Agenda in 2010″, where Dr Giles Nelson of Apama points out the benefits of event processing technology to “detect situations” and aid business processes. Giles’ article is of course part of a post-BPM-acquisition marketing campaign, and would be more accurately titled “Why Event Processing should be on BPM’s Agenda in 2010″, but he makes some valid points nonetheless:

…we need to take a different view of BPM technology and try to see how it can be used to make knowledge-based business more ‘operationally responsive’…

…the potential for creating real business value by bringing together the two disciplines of event processing and BPM is substantial…

…BPM will get really interesting when it is combined with event processing and can therefore detect situations that are occurring and automatically begin a process that takes action on that information…

…It’s about delivering real-time insight into what’s going on in processes across the organisation and, using that insight, to drive efficiencies…

All very true. Many BPM folk are of course still mostly concerned with defining what happens today, not what needs to happen tomorrow: in some ways event processing is just another step up the real-world business process maturity model.

And an example of Operational Non-Intelligence…

Paul Vincent — Wed, 17 Feb 2010 17:46:51 +0000

What you dont want to see in a presentation

While presenting to TIBCO CEP customer Societe Generale today, I had to make excuses for the example of “unintelligent operations” appearing in my presentation. To wit, whilst in the TIBCO Paris office earlier I had connected to the network to check emails, after which I put the PC into “standby” to facilitate a “quick start” for our presentation.

Big mistake: when I started the presentation, I got the infamous “Restart Windows” dialog. Dang! Unbeknown to me, when I had connected the PC (running a corporate XP set-up with compulsory Windows Update set to “on”) in the office it had gone ahead and downloaded a Windows update in the background. It promptly finished its install and requested a reboot when I started our presentation…

The intelligent behavior here might be to coordinate background downloads and installs with the PC user’s calendar. Other intelligent behavior, which we are seeing more of in TIBCO, would be to voluntarily bypass such corporate policies through the use of a Mac, Linux, or maybe run the “corporate PC settings” in a virtual Windows session

PS: one solution to keep up one’s sleeve is described here: however, this must be done “per session” as the TIBCO IT guys, not easily outsmarted, do things like thwart VPN access when certain services are disabled! The reboot message was just annoying; the total loss of PC responsiveness when the corporate antivirus scan kicks-in would have been disasterous!