Complex Event Processing (CEP) Blog

OMG PRR - the 1st official production rule standard

Paul Vincent — Thu, 02 Jul 2009 18:04:18 +0000

Congratulations to the OMG PRR team - the OMG have voted the Production Rule Representation 1.0 as a standard for UML modellers to represent production rules (including “sequential” if-then statements as well as Rete-type inference rules). Of course this is not the last version of PRR - there may be a few bugs to work out and we have commissioned the Revision Task Force already - with a 1.1 release planned for next year. Maybe other CEP vendors will realize the importance of standards (and rules!) and join the effort?

Close behind is the W3C Rule Interchange Format - Production Rule Dialect - which is “imminent”. There was some overlap between the teams involved with these standards so, usefully, and as PRR Core defines “the” generic production rule metamodel, PRD is “compatible” with PRR - meaning conversion between the 2, from modeling to interchange, should not mean too many hurdles. Indeed, current ideas are for a RIF PRD 2.0 that allows for object models (as opposed to “frames“) and a PRR 2.0 that utilizes the PRR-PRD syntax that uses the PRD expression language… we’ll see.

The Banker 2009 award-winning CEP app: Citibank Hong Hong on real-time marketing

Paul Vincent — Wed, 01 Jul 2009 22:29:39 +0000

The Banker runs annual technology awards, and this year the Citibank CEP application supporting real-time marketing is one of the winners. The article quotes:

“The system deploys complex event processing technology to evaluate static and dynamic events against a customer profile and ‘propensity model’, to determine in real time the next best offer the bank can extend to the customer.”

“Because the system operates on a ‘rules basis’, it can be adapted by business users to design the rules that govern a particular campaign without the involvement of IT staff. “

“The bank is also using the intelligence in the system for fraud prevention and proactive customer service.”

Standards news: PRR and DMN updates…

Paul Vincent — Wed, 24 Jun 2009 17:53:41 +0000

This week OMG’s Technical Meeting has had most buzz around the BPMN 2.0 submission (and congratulations to that team, for which TIBCO is a supporter). Some of the other stuff going on includes:

PRR or Production Rules Representation (which, for complex event processing like in TIBCO BusinessEvents, can include event-condition-action rules using standard forward-chaining semantics) starts the Revision Task Force process for PRR1.1. A few tweaks are due, but more interesting was to see 2 more vendors attend the PRR session this week. Also, note that UML tool NoMagic is presenting on PRR at ORF’09 later in the year…
DMN or Decision Model and Notation started on its first stage of development, which is the identification of use cases and roles for defining UML-based decisions for a future RFP. We had some good discussions (which I’ll report in a future post once I have reported to the DMN community), and it is clear that this could be a very key standard for business modelers, end-users and tool vendors. Of course the relevance of this to event processing is that many CEP/EP systems’ role is to support decisions…

A somewhat hotter debate continues (/is) regarding the proposed Case Management RFP, which was developed from a Dynamic Business Activity Modeling RFI last year (which TIBCO responded to, and was also covered by our session at the recent Semantic BPM day in Berlin). Many BPM applications are also case management applications, but some case management requires more sophisticated event-handling, rule-driven processes, decision management, and case record management and recording (technologies that TIBCO mostly covers under BPM+). One school of thought is that the more sophisticated requirements for case management need to be rolled into the common BPM standards stack (including BPMN); another is that multiple different standards should be used flor flexibility (such as combining BPMN with BMM, PRR and DMN). From an event processing perspective, of course, case management (by one definition at least!) involves applying incoming events to the state of some case in order to determine whether processes need to be started, continued, halted or changed - in other words CEP technology can often be applied for case management areas in government, finance, healthcare, etc.

Some of the other case management discussions can be found from EBizQ, Bruce Silver, and Derek Miers.

Project Managing an Event Processing Application

Paul Vincent — Wed, 24 Jun 2009 16:49:51 +0000

Over the past few weeks I’ve been an “Acting Project Manager” alongside a talented TIBCO team who designed and implemented a telco services gateway using TIBCO event processing software. Of course, the importance of the Project Manager role decreases in direct relationship to the experience of the team - so my role here was somewhat redundant due to the experienced implementation team. The project required Rapid Application Development exploiting a “standard set” of TIBCO technologies:

TIBCO BusinessWorks to host the service entry and exit points
TIBCO BusinessEvents for the main processing rules covering routing, validation, service throttles, KPIs, logging etc
TIBCO EMS as the main external event (/message) transmission mechanism
TIBCO RV for internal communications within the distributed application, e.g. for log handling
TIBCO HAWK for application monitoring
TIBCO Administrator for managing the deployment and updates to the application.

Some observations on the Project Management aspects were:

Good design counts: there were several specification changes (including to the specification of the incoming events) that were to some extent mitigated by the tooling flexibility, and some by the use of good design patterns.
Project Management is sometimes Anti-Agile: some of the (event) specification changes we deferred to later in the application development process in order to provide controlled, manageable, validated releases. The PM’s main role is obviously to protect the development team and delivery schedule from scope creep, although inevitably some scope changes were requirements boundary cases…
Test-driven development: early effort was expended in an appropriate test framework and test cases, used to exercise the event processing application with particular event types and patterns. This paid dividends in allowing defects to be identified, and resolved, early.
Communication and control artifacts: the High Level and Detailed Designs were defined in docs and in prototype BW processes and BE rules; we first used MindManager for internal reporting of issues, and then progressed to an existing Bugzilla instance for co-operation on issue management. Although we had an MS Project plan, we deferred to Excel for internal and customer reporting. Code and docs were controlled using SVN.
Knowledgable customer contacts (and good customer relations) make the PM’s job so much easier…

Some specific project challenges relating to the “event processing” aspects:

- Event definitions are key. If the event model is changing lots as development is occuring then this can be problematic. One traditional and effective solution is to use your own internal abstract event model and map to / extend this as required.
- Testing timed sequences of events could be a challenge. This project was event processing, not complex event processing (there were a few abstract KPI events created, but for the most part the application was stateless). Obviously, live system events are ideal for testing.
- Application performance (throughput, latency) need to be monitored and managed thoughout (although not an issue for this particular application, so far).

So - not much Project Management is specific to the “event processing” aspect. But that is good news and to be expected when insulated by a good development team and good tools!

Other TIBCO blogs…

Paul Vincent — Tue, 16 Jun 2009 17:39:59 +0000

Someone asked what other TIBCO blogs were around, as the “Communities” tab on www.tibco.com currently only mentions this, the CEP blog…

Here is a quick guide to the other TIBCO blogs…

Cloud computing - the TIBCO Silver blog - answering questions about TIBCO’s approach to cloud computing for businesses…
Spotfire community - the Spotfire blog(s) - answering mostly technical stuff about visual BI and analytics with Spotfire, S+, Miner etc
TIBCO developer community - the TIBCommunity blogs - covering a wide range of stuff (divided by product, including for BusinessEvents).

Event-driven Rule Maintenance…

Paul Vincent — Sun, 14 Jun 2009 23:18:34 +0000

One of the interesting attributes of rule-based systems (whether in complex event processing, decision services, or whatever) is the ability for end-users to maintain the rules separately from the IT development cycle. Rule-based development tools (TIBCO BusinessEvents included) typically allow for the hot-deployment of these changes - usually where process instances are updated with the new application logic in between transactions or events. This update process is both inevitable and a requirement when the underlying ontology or Business Object Model changes.

There is however a class of update that is less obtrusive, and can be managed simply by a rule update event. This is where the rule design pattern (or template) does not change, but the change is a create / update / delete operation on such a pattern instance, stored as an object or concept. Alternatively the update could simply be to rule instance metadata (for example, champion-challenger status, or active status, or effective date). In an Event-Driven Decision system, such update events are simply a new type of event to be processed, and would typically be created by some suitable Business User Interface (or automatically generated by some analytics system). Of course, rule maintenance events themselves should be validated by the rule engine prior to deployment, and other controls such as security and authentication may be required.

Of course, with event-driven rule maintenance one has to decide where the *record* of the current rule instances is to be stored - either in running rule system (for example, exploiting the distributed cache), or local to the user interface. Alternatively one can resort to tradition, treat the rule instances as data in a database, and update them via a simple 2-tier UI using database-imported concepts in the rule-based system…

Hi Ho Silver, Away!

Alan Lundberg — Fri, 05 Jun 2009 00:11:04 +0000

And with that, TIBCO’s newest offering, Silver is off and running.

Announced yesterday at the NOWonline show, it seems to be getting a good bit of attention in the press, analyst and blogosphere communities. eBizQ picked up on the announcement and commented on its use of CEP in the automation of cloud-app-balancing. As for me, my head is a bit cloudy at the moment, from all the fuss.

So what is Silver, and what does it have to do with CEP?

Everything.

TIBCO Silver is new software infrastructure for “cloud” computing. A “Silver” lining for the clouds you might say.

And why is this important for CEP?

Because it’s an infrastructure product that embeds a CEP engine in order to solve problems related to governance (managed access, security, privacy and adherence to regulations), and scalability (uses SLAs to automatically scale up / or down as needed). The kicker is that it’s automatic, so both the governance and the scaling is accomplished inherently through embedded monitoring, management and event-decision-action rules rather than manual intervention and programming -which AFAIK, is an achilles heel for current cloud products being introduced.

This should be an interesting announcement for developers of different types of Business2Consumer or Consumer2Consumer apps that are likely to vary widely in resource requirements. The embedded governance allows for various levels of authorization, authentication and encryption policies to be dynamically configured. This is important because some services should be open to everyone and some services, well, just shouldn’t.

As in most cloud architectures, and not counting those who simply put the cloud moniker in front of their latest software product, there is no software to install or hardware to procure or provision, which reduces the barrier to develop and deploy rapid IT solutions (whether that’s infrastructure, platform or applications)

TIBCO Silver is currently in Beta. It will be interesting to see the deployments when they start rolling out.

New! Improved! Blog!

Paul Vincent — Thu, 04 Jun 2009 15:26:00 +0000

So, anyone who frequents this blog may have noticed one or two stylistic changes… Many thanks to Himanshu, Daniel and the TIBCO web team for the update. Regarding the blog voting mechanism: we categorically deny that any blog authors sit under a vote-actuated slime bucket… yet.

Progress on Event-related Standards at OMG…

Interesting trend: CEP at BRForum, ORF, RuleML events

Paul Vincent — Fri, 29 May 2009 08:55:00 +0000

Traditionally (i.e. for the past 2-3 years!) CEP has mainly been covered by the (mostly academic) DEBS event [3rd is this year in Nashville, July 6-9] and the EPTS Symposium [5th is this year in Trento Italy, Sept 21-23]. Interestingly there is increasing spin-off of CEP into other events - such as (interesting for rule-driven CEP tools like TIBCO BusinessEvents) the Business Rules Forum [12th this year in Vegas - presumably as an antidote to this year's economic news - Nov 1-5]. BRForum has acknowledged 2 very-CEP-related agenda entries by 2 very-respected-experts:

Event Processing 2010: Past, Present and Future
David Luckham, Emeritus Professor, Electrical Engineering, Stanford University

This tutorial on Complex Event Processing (CEP) will cover six topics.
1. Developing markets for event processing - a short survey of the growth of CEP in enterprise management applications and Business Activity Monitoring.
2. History. - Event processing 1950 ─ 2000.
3. Adopting event processing — how to analyze your event processing requirements and plan a solution.
4. A survey of basic CEP concepts and their applications.
5. Crossing the Chasms - the four stages in the development of event processing from 2000 to 2050. The need to improve the CEP technology in commercial tools and applications.
6. The age of ubiquitous CEP - event processing goes global and disappears under the hood. Scenarios of current and future applications.
What you will learn:
• What Complex Event Processing is
• How to apply CEP to solve business problems and improve your BI operations
• How CEP enhances Service Oriented Architectures, Business Process Management, and Business Rules systems

A Facilitated Peer-to-Peer Workshop: Semantic Processes, Services and Events
Paul Haley, Founder, Automata, Inc.

Semantic technology provides the most general and flexible form of data modeling along with logical and rule-based capabilities. A new wave of semantic tools and standards, including models of time, events, and processes promise to align enterprise data modeling, application development, service-oriented architecture and business process management more closely with the perspectives of knowledge management and business rules practitioners.
What we will discuss:
• How semantic standards extend model-driven architecture to knowledge management
• How semantic architectures and models unify SOA and BPM, including events
• How semantics increases the impact of business intelligence and activity monitoring
• How BPMN, SBVR, PRR and complex event processing do or don’t intersect

Just the week before BRForum there is the rule-developer-focused October Rule Fest (which I keep wanting to write as Oktober Rule Fest, for some reason) [in Dallas, Oct 26-30] which, apart from a fascinating agenda for rule IT folk, has CEP topics such as:

Early Alert System at Southwest Airlines
Greg Barton: Southwest Airlines, Senior Software Engineer

Southwest Airlines is venturing into the rules development space with the Early Alert System. EAS enables SWA to have a real-time model of it’s entire aircraft fleet, tracking such activities as taxi in, taxi out, and in gate turn. It does this by maintaining a data structure representing physical assets and the activities they perform. Incoming data from those assets update the data structure, and rules react to the changes. We hope to use this paradigm going forward to use rules to monitor other aspects of the enterprise, enabling a more agile and efficient response to the airline’s daily operating challenges. Our main points will be the Overview, Feature Review, Design, Other Uses of Rules at present by SWA and the future of rules at SWA. [Note: this is a TIBCO CEP application in production at SWA]

ET2: Temporal Reasoning: a requirement for CEP
Edson Tirelli: Drools, CEP Designer

As Complex Event Processing grows in popularity and applicability, the convergence between modeling paradigms demand more and more functional requirements from the available tools. One key requirement for CEP use cases (and standard business scenarios) is the ability to write rules and queries that require some degree of temporal awareness, from simple constraints to actual data inference. More than that, temporal reasoning is a feature on top of which the actual platform can leverage internal optimizations, aiming for resource savings and improved scalability. [Note: DROOLS has now become the 2nd CEP inference engine]

A Survey of Complex-Event Processing Models
Charles Young: Solid Soft, Principal Consultant

Prof. David Luckham defined an EPN (Event Processing Network) as a network of ‘lightweight rules engines’ which act as Event Processing Agents (EPAs). He contrasted this with the exploitation of rules-based inference engines as ‘heavyweight EPAs’. Complex-event processing (CEP) is inherently rule-based and centres on pattern matching based, in large part, on temporal constraints. CEP, today, is broadly characterised by the use of diverse processing models embodied within different technologies. What are these models? What are their major differentiators, strengths and weaknesses, and how do they compare with Rete engines and other rules processing approaches? Are some models truly more ‘lightweight’ or ‘heavyweight’? What are the underlying differences and similarities and how might each approach best be exploited in building scalable and agile EPNs?

Still to be announced is the RuleML09 event’s agenda [conveniently co-located with BRForum, Nov 5-7] and targeting the rule representation community… we’ll have to wait and see if CEP gets represented here too!

Disclaimer: TIBCO is presenting at BRF and ORF on rule-CEP topics, too.

NOW looking into the cloud, as well as CEP

Paul Vincent — Thu, 28 May 2009 14:08:22 +0000

Not really too CEP-related, except that events and event processing can take place anywhere: TIBCO’s Cloud Computing initiative is an announced agenda item for next week’s TIBCO NOW global webevent. I haven’t seen the content, but clearly the main area of interest is likely to be Corporate Cloud Computing (i.e. your distributed-managed data center), and there is an obvious connection to virtual service management (i.e. TIBCO’s ActiveMatrix product set).

TIBCO NOW also includes some EP/CEP content under the Enterprise Optimization thread:

- Supporting Dynamic Change Management with CEP
- Philippe Goujard, Senior Manager, Accenture
- For a major micro-electronics manufacturer any change request is a long and complex process: it impacts machinery, production, logistics, training, and documentation. By introducing a complex event processing solution based on TIBCO BusinessEvents the client optimized monitoring of a complete “change request” project, improved time to market, and reduced risks associated with conflicting projects.
- Keeping the Lights On while Managing the Complexities of Compliance
- Lyndon Wilkin, Solution Manager, EnergyAustralia
- EnergyAustralia (EA) has over 100 years of experience providing broad-based energy solutions to over 1.6 million homes and businesses. Hear how EA is using TIBCO software to gain real-time visibility into customer service levels, to detect and respond to events that could affect compliance, market obligations, and providing a foundation technology for enabling the intelligent grid.
- The Future of Enterprise Optimization
- Alan Lundberg, Senior Product Marketing Manager, TIBCO Software
- Visibility into every facet of your business is key to improving quality, customer service, and cost control. TIBCO provides the ability to harness the power of existing data and in-flight information across your many customer touch points in real time. Managing your business at the speed of events is essential to be a leader in today’s market. This session will discuss the event processing technology that SouthWest, QWEST and CitiGroup use to drive competitive advantage and that analyst firm IDC ranked number one in the market.

Registration is, by the way, free.