Of course, “real” ESP and CEP engines also support continuous queries - TIBCO BusinessEvents included; hence at least one answer to Hans Gilde’s recently asked question on why rule-based (monitoring) systems do not include some SQL-based Event Processing Language  is: actually some already do).

Notes.

[1] Probably, at some point, EPTS will need to publish a comparison of all the different CQL implementations and their semantics. After that, we can discuss”standardization”…  one gets the impression that Oracle would prefer to claim that their implementation *is* the de facto standard, but unlike their SQL implementation they likely don’t have anywhere near the installed base for CQL to justify such a claim.

[2] Note that a continuous query is not the same as CEP. A continuous query running against a high-latency database, for example, may just buy you a more efficient trigger representation.

[3] Presumably, not literally…

[4] I saw these cartoons in a blog recently and thought their similarity to CQL / database issues to good to pass up!

Opher, I noticed, used the term “Event Processing Agent” to describe a logical event processing task, perhaps made up of several EP operations. Hmmm, I’ve seen the term “Agent” in CEP context before…  such as last week when Jim Odell presented at the OMG meeting on the OMG Agent Metamodel and Profile for agent-based programming (listing CEP amongst its uses). We at TIBCO talk about (large granularity) Event Processing Agents (which collaborate as multiple distributed processes, possibly using differing techniques such as rules, rules + states, queries, etc), but otherwise our agents have similarities to Jim’s definition. Oracle, in the following tutorial, seemed to share the same usage as TIBCO. So perhaps the taxonomy / terminology should look something like:

• Event Processing (EP) Operator is-used-in EP Element
  where Element is a logical CEP task consisting of 1 or more CEP operations
  for example, Operators might be  a query part, a rule condition, a SNOOP operator
  for example, Elements might be a rule, ruleset, query, set of queries
  • EP Element is-used-in EP Agent
    where Agent is an autonomous or semi-autonomous process containing EP Elements
    for example, an Agent might be a set of EP Elements together with event interfaces (and any necessary controls) to define a processing unit
    • (and maybe EP Agent is-executed-in EP Engine)
      where Engine is an executable deployment container for EP Agents
      for example, the appropriate CEP executable used to execute 1 (or more) EP Agents
      and where a System may include a set of distributed Engines and their associated Event infrastructure

Although “EP Agents” were defined by Opher, I don’t recall “distributed EP agents” being discussed (although “distributed EP” is somewhat orthogonal to “event processing element”, with the caveat that latency modeling might be needed for highly distributed CEP systems). Distributed CEP requirements are already a subject of discussion (for example, see rule engine advocate James Owen’s recent comment on the need for distributed processing for Homeland Security problems). Such requirements are also, of course, already being addressed, such as by some high-scalability rule-driven CEP vendors…

Notes

[1] Opher had to use someone else’s laptop, with the effect that his PPT displayed logical operators using some very pretty Microsoft symbol font symbols. The effect made his notation look much more interesting though!

[2] … although it may well be the case that BPDM will need “tuning” to accomodate continuous event processing and “non-traditional BPM” concepts. In addition, this may or may not be an ultimately useful exercise (except for those who need holistic views of their enterprise process models).

[3] <Rant> I was searching for a suitable link for OMG BPDM that would make sense to the CEP community, but didn’t fiind any that expressed my view (/hope) of what BPDM represents, or suffered inaccuracies like equivalence to XPDL, that SBVR business notation equates to process rules, or that BPDM is used (implication: directly) by business users. Of course, existing BPDM descriptions are focused on selling the idea to the BPM community (although the RFP implies a wider audience including the EP community). </Rant>

Sharma Chakravarthy from the University of Texas kicked the tutorials day off by covering the Snoop representation for event operations. This seemed a neat way of modeling (some) aspects of event processing, including the fact that a composite event can represent multiple events which define a duration, making temporal operations tuple-based (i.e. with a start and end timestamp). This leads to a specific event-notation (with maybe some similarities to ORM), and “interval-based semantics” for composite event operators (which look like they might be overly complex for typical use cases and CEP users). Snoop seems to have been around for years (see this 1993 reference!). I have to say that the references to Active Databases made for a good history lesson, but also made the tutorial seem outdated (e.g. as an exercise, consider how you might relate the terms “active database” and “distributed event”).

The Snoop info was interesting (for anyone who hasn’t seen it before), but I was hoping for more on patterns and uses for combining ESP with CEP, per the tutorial title (”Events and streams, harnessing and unleashing their synergy”). Indeed the session started with the basic concept of processing streams before combining their resulting high level events - certainly a CEP design pattern which is of particular interest to TIBCO BusinessEvents users.

Cruel relief was provided during Sharma’s “EP operations are not database operations” comments [*2], which for some reason seemed to annoy the heck out of the Oracle guys…

Notes

[1] DEBS included a number of industry presentations, but many of these represented “R&D” updates from vendors. For a more commercial conference, but with less technical detail, consider the Gartner event later in the year. Many CEP and EP vendors were present (NetZero, Streambase, Oracle-the-database-storing-events, Oracle-the-Weblogic-app-server-version-for-events, WareLite), as well as vendors from other technologies such as BRMS/BRE (ILOG), ERP (SAP), RDF-store (Franz), etc.

PS: googling for DEBS2008 or DEBS08 will likely hit a number of, er, unrelated and possibly work-unfriendly sites. Use the search term DEBS 2008 CEP for a better success rate.

[2] Sharma had an excellent comparison slide on “DBMS” vs “data stream management systems”. This went through their respective attributes, which looked approximately like:

1. persistent relations vs transient streams
2. ad hoc vs continuous queries
3. random vs sequential access
4. unbounded disk store vs bounded memory store
5. only current state vs history / arrival order critical
6. low update rate vs high data rate
7. no real-time services vs “real-time” requirements
8. precise results vs approx results
9. transaction mgmt vs no transaction mgmt (e.g. rollback)

To show how current CEP tools might fare in this comparison, lets see which attribute best applies to TIBCO BusinessEvents: 1. both, 2. both, 3. both, 4. both, 5. both, 6. high data rate, 7. “real-time”, 8. precise results, 9. no “transaction management”. Not bad (and you could argue that transactions are just managed “differently” in an EDA).

Notes

[1] Interestingly, one of the keynotes at TUCON (by Margam Sundararajan from Citigroup) explained how a CEP application was used to help justify the SOA strategy. One of their applications, using TIBCO BusinessEvents, was deployed first in order to get a quick business ROI.

[2] The AllState application discussed in this podcast by Anthony Abbattista is also a TIBCO BusinessEvents CEP application.

From a CEP perspective, BPMN represents “simple-event flows”, and could in theory be used as the starting point for a set of standard semantics for continuous event processing (and, by extension, complex event processing or CEP). But simple “process flows” are to business processes like “decision trees” are to business rules - a very useful representation, but not in any way the only, nor necessarily the best, way to represent every process. There is a good reason why BPMN is commonly associated with “workflow” (modelling human-oriented business processes) [*2]. There is also a good reason why BPDM is potentially a very useful cross-process (including CEP) metamodel (and ergo why it should not be directly tied / restricted to just BPMN) [*3]. Conrad Bock’s (NIST) BPDM tutorial at the OMG meeting was well attended, and indicated that BPDM, as intended, shows scope to be a generic process metamodel that could be used to relate different process modeling styles, including continuous and complex event processing. Definitely something for EPTS / OMG / CEP researchers [*4] to look into in future.

Meanwhile, in the real world, we are finding (1) an event-driven approach to modeling enterprise (inter- and intra-departmental as well as B2B etc) processes and (2) the use of executable event-driven models to drive workflows (as commented earlier) are being recognised as very useful capabilities. At TIBCO, for example, the use of standard modeling constructs (concepts / classes, state models, production / inference rules [*4] and queries) and best practices (event-awareness, decision management) with a high-performance distributed execution engine allows for dynamic process definitions that can effectively drive BPMN/BPM (iProcess etc) workflows.

As a heads-up, the relationship between CEP and BPM is a round table topic at the BPMI Think Tank later this year [*4].

Notes

[1] For the record, TIBCO’s BPM team already supports the defacto BPMN persistence mechanism (WfMC’s XPDL), and is a member of the BPMN2 submission team that includes IBM, SAP, Oracle, etc.

[2] Consider a large decision tree where you need to insert a new rule that affects more than 1 decision path. Oops, you now have a maintenance problem (did you update all necessary decision paths? when this rule changes, will you find all the necessary paths? etc). BPMN and orchestration diagrams / activity diagrams all suffer from the same representation scalability problem, which is why declarative forms of behavior are often needed (e.g. alongside or to augment orchestrations).

[3] BPDM was designed, originally, to handle all types of processes, not just BPMN orchestrations, and including event-driven ones. It may yet be considered by EPTS for potential CEP metamodel standardization …

[4] It seems that BPDM might be compatible with existing and proposed CEP and CEP-relevant standards (such as PRR), and even Opher’s proposal for an Event Processing (EP) meta-language (research topic / future standard). BPDM provides a general structure for organizing components of processes, including an emphasis on events, but it remains to be seen if it can handle (or be readily extended to handle) declarative process models, continuous process models, etc.

[5] Disclaimer: TIBCO are chairing said round table.

Meanwhile, someone mentioned (on the separate topic of Enterprise Process modelling) that TIBCO’s standards-based CEP toolset, TIBCO BusinessEvents, is effectively an instance of “executable UML”. For sure, it does not include all (indeed it is very much a subset) of UML 2.0 (e.g. nothing to do with UML pins…) , and the production rules part of UML is still in beta (PRR), but nonetheless the concept (aka class) model and state model are standard UML components. Of course, one uses the Java-like BE rule/action language instead of action semantics script, and the application is generated (as opposed to “interpreted”), but nonetheless it can certainly be described as an event-driven, high-performance, automatic-persistence, executable UML-subset tool…

Notes

[1] However, OMG has diversified into more topical standardization issues such as BPM orchestration flows (BPMN), ontology mappings (ODM), business architecture (a new Special Interest Group), business statements (SBVR), and so forth.

• cross-sell-upsell during a customer interaction
• detect opportunities as they occur
• detect harmful events before they affect the business

Note these all have a “real time” aspect: waiting a few months for the Analytics Department [*2] to get round to discovering some trend or pattern is not going to wash (although would be better than not discovering anything at all, of course). These are (for TIBCO at least) existing CEP customer use cases, too.

Not surprisingly, the author recognises the relevance of CEP to these sorts of “BI requirements” - the last page has Technology predictions [*3] listed as:

• data integration from multiple sources
  [… for example a ESB + CEP + real-time high-performance event / data store ?]
• complex data models with business rules
  [… could be handled by model-driven concept models with associated behavioral production rules… sounds familiar…]
• Complex Event Processing
• Ontologies and semantic search
• “Predictive algorithms”
  [… which presumably I’d want to run under the CEP system against real-time data … ]

As an analyst report, this doesn’t prove anything in particular, but indicates a trend: more people are recognizing (and analysts are finding noteworthy) the role CEP plays across the intelligent enterprise.

Notes

[1] The article is presumably a precis of the full, sponsored, report. 31% of respondants apparently view CEP as a long-term requirement for them to handle future BI needs.

[2] Check out this marketing-post on EbizQ. The first sentance reads:<<[traditional Predictive Analytics tool vendor] has been used for decades by federal agencies, but the need for real-time information sharing, situational awareness and accurate decision making are both a golden opportunity for the company and an opportunity to serve the public good.>> This seems to imply that event traditional analytics tool vendors recognize the need for new approaches (e.g. CEP) in domains like Government.

[3] Probably 3 of these “future BI technologies” could be described as being included in TIBCO’s Spotfire Operations Analytics, by the way…

1. DEBS’08
   Where: Rome, Italy
   When: July 1-4, 2008
   CEP content: Tutorials [*1][*2], Presentations
   Cost: €400 + discounts
2. OMG’s Workshop on Distributed Object Computing for Real-time and Embedded Systems
   Where: Washington, D.C. USA
   When: July 14-16, 2008
   CEP content: Tutorial, Presentations (by TIBCO Software, Naval Surface Warfare Center, RTI) [*3]
   Cost: US$795 + discounts

Further out we have the Gartner Event Processing Summit in Sept as well as various CEP topics at Business Rules Forum / RuleML 08 in Oct (kudos to the organisers for co-hosting, allowing both business and R&D interests to be shared), and the CEP-related (via Event-Decision Architecture) Real Time Decision Making Conference in Dec …

Notes:

[*1] The tutorial list is pretty interesting:

• Oracle has a marketing session on their proprietary extensions to SQL for event stream processing (CQL). Does this presentation imply that the CQL team has triumphed over the BEA-Esper team to be the Oracle CEP tool in the ongoing Oracle-BEA merger?
• Opher Etzion covers CEP architectures and and CEP patterns, which should be lively given all the various opinions and options on these topics …
• Prof Sharma Chakravarthy from The University of Texas covers “Complex event processing fundamentals, stream processing fundamentals …why their integration is needed…”. Couldn’t agree more!
• Lots of pub-sub middleware topics. Some look very interesting.

[*2] Disclaimer: TIBCO was represented on the Program Committee for DEBS.

[*3] Disclaimer: TIBCO is, er, teaching the CEP tutorial as well as presenting at OMG Washington.

Notes:

[1] This should be particularly useful to any vendors whose marketing orcs and trolls managers believe they can adjust industry terminology to suit their (acquired) CEP offerings (not mentioning any names, of course!). Hint: Compare the definition of complex event / complex event processing in both the glossary and James’ blog of a Forrester conference presentation…