Ashley described some of the past efforts to model and implement choreographies, using types of process algebra such as  Robert Millner’s Calculus of Communicating Systems (CCS) and its derivative Pi-Calculus. However, Ashley used sequences of events and states (i.e. a state diagram) which he also compared to Michael Jackson’s formalised object lifecycles (e.g. JSD  / Jackson Diagrams). Various W3C efforts have described choreographies too - e.g. WS-CDL. Of course the latest modelling construct for choreography is BPMN2!

As an example of his practice, Ashley described an example - modelling bank account transactions via Protocol Modelling (using simple state diagrams):

• state model 1: defined the close and withdraw events on an active account
• state model 2: defined the freeze and release account events
• state model 3: this had no state transitions, but defined the state by the associated constraints (or business rules)
  • if balance < 0 then account state is overdrawn
  • one cannot close an account if it is overdrawn
• all 3 state models operate in parallel.

To analyse these state models they can be combined into a single state models (with all combinations of states, and all events), and then the unreachable states can be filtered out. The interesting thing here is (1) the analysis of state models for completeness and (2) the use of incomplete state diagrams as a business notation for textual (policy or constraint) business rules.

Other observations:

1. These types of business rule apply to states and data; they can be extracted and modified (by a developer, or state modeller) into event rules or guards in a state transition diagram. Is it interesting to specify these business rules up front before mapping to events and processes? Yes from a business perspective, as new events or states might affect or be affected by existing business rules.
2. Using a state to specify a business rule (in terms of the state and output) is an interesting notation that lends itself well to mapping to appropriate events (or indeed processes). Could it catch on in the business rule community?
3. The use of an explicit choreography language has not had  much success it seems. Google WS-CDL and most entries are dated 2009 or earlier. BPMN2’s choreography may yet prove useful but possibly the concepts are too difficult for business modellers yet imply a co-operative design process for developers that rarely occurs in practice (beyond “this is the interface”!).
4. At the end of the day, the sequence of events in a business system is just a complex event - which maybe can tell you if the choreography is valid or not.

I’ll add a link to the slides to help explain all the above when they become available…

Annex: a Distributed System Choreography Development Process:

This process describes a development process of state diagrams for choreography purposes:

1. Define participants and messages (/events) that interact between them
2. Define states with events as messages from and to, with only 1 sender per state
3. Project the states out to individual participants - i.e the parts of the state model for each participant - allowing ambiguous states but ensuring these have no sends
4. Merge the states for each participant
5. Enact - check each event at a time to prove feasibility of the interacting state models

No related posts.

• Event Processing Platform is a software that enables the creation of event processing network, handle the routing of events among agents, management, and other common infrastructure issues.
• Event Processing Engine is a software that enables the creation of the actual function - in the EPN term implementing agents.

In the CEP Market analysis we don’t try to distinguish between these - probably because it would be contentious. For example, to some folks an “event processing network” is managed as a single process - possibly multi-threaded, but bounded on a single machine instance. To others (like TIBCO) the network is a message or event distribution mechanism for breaking the constraints of a single process or system (e.g. performance, scalability, and fault tolerance constraints). Furthermore “event processing agents” might be viewed as “event processing operations” - like a single pattern detection query, or a pattern matching rule, arranged in some kind of activity or business process diagram - or as more autonomous processing agents that can handle a number of operations and cooperate declaratively towards some solution.

If one views an Event Processing Platform as one that handles routing across multiple processes and distributed systems, then the potential candidates is reduced somewhat [*1]. Of course, any CEP engine can be used acoss multiple systems with a shared middleware infrastructure, but individually they are “blind” to the other agents and the design tools do not handle the cooperative nature of the agents. Of course, one can set up a message type to include management information to allow for some semblance of distributed control, but this is more likely to be a developer task than a platform capability.

Looking at something like TIBCO BusinessEvents, we can see this satisfies the requirements of a (physically distributed) Event Processing Platform:

1. Enables a (computer) network of event processing agents - typically as a minimum of rule agents and cache /datagrid agents, in pretty much any configuration.
2. Enables a (single process) network of event processing operations - typically the network is implemented as  declarative rules, but can be visualised as a network in a report.
3. Enables different types of Event Processing Engines - apart from the rule agents, you can also have (continuous) query agents.  Rule agents can also be customised as “decision agents” (executing decision rules,  or decision tables), “analytics agents” (executing predictive analytics models in Spotfire S+ or R), or “optimization agents” (executing NuOpt optimization routines in  Spotfire Statistical Services) [*2]

Notes:

[*1] Other candidates for an Event Processing Platform across distributed systems include IBM Infosphere Streams (although IBM is very quiet these days about that), and EventZero. If there are any others please comment them, and if enough we’ll update the  Market Analysis with this classification…

[*2] Note that invoking Spotfire services involves invoking the Spotfire platform under the control of a rules agent; from an architecture point of view these are just SOA services, like calling BusinessWorks services during event processing.

No related posts.

TopCom, … is a private communications platform for the two hundred most powerful people in the world.

TopCom is being officially launched in late January at the annual meeting of the World Economic Forum in Davos, Switzerland. It is basically a customized, ridiculously secure version of tibbr, a platform developed by Tibco as a kind of combination Facebook, Twitter, e-mail, texting, and Skype. It is a private social network, essentially - in this case, for world leaders.

… The top two hundred WEF members - basically, the people who run the world - can speak to one another on a given subject, and then they can choose to loop in members from lower tiers (experts, academics, etc.) as needed, widening the pool of knowledge on whatever problem is on the table.

…Tibco consulted with both the Japanese prime minister at the time of last year’s tsunami, Naoto Kan, and his successor, Noda, when it was developing its presentation for the WEF board of directors, to find out what would have been useful to them at the time of the disaster. Schwab, too, collaborated. The result, which will be on display in Davos, is the first time a global organization will introduce its own proprietary communications platform. …

No related posts.

techniques and technologies that make handling data at extreme scale economical.

with “extreme scale” being defined mainly by the metrics of volume and “velocity” - with the latter being the obvious area of interest from an event processing perspective, as stated by Curt:

Low-volume/high-velocity problems are commonly referred to as “event processing” and/or “streaming”.

Ignoring what might constitute high volume / high velocity problems (see later), Curt replaces “velocity” with “structure” to the “big data metrics” chart (with “velocity” being included in his “bigness” metric). But of course the argument over whether “structure” or “velocity” (or neither or both) are relevant metrics for Big Data is entirely perspective-based:

1. both are characteristics of data / events and
2. both affect processing and storage techniques,
3. … along with other metrics like data lifecycles and data value.

From an event perspective, event payloads (real-time data) can be simple values, tuples (such as the equivalent of a database record), or complex explicit data (such as an XML document), for which something like TIBCO BusinessEvents rules, continuous queries or patterns can be applied. For unstructured text then you may want to add TIBCO Patterns, and for non-deterministic data something like TIBCO Spotfire S+ (think neural nets and the like).

From a “big data” perspective, event processing use cases can include customer purchase records, credit card transactions, phone voice packets or text messages, inventory updates, operational sensor reports, etc etc. But from the event processing perspective (i.e. actually exploiting “big data”) there is another dimension to consider: the scale and velocity of the incoming events versus the scale and velocity (and structure) of the existing data it needs to be related to and/or processed against. Some examples might be:

• large volumes of data at high velocities, compared to large volume of data
  = national security applications
• large volumes of data at high velocities, compared to normal volume of data
  = sensor processing like Radar
• normal volumes of data at high velocities, compared to large volume of data
  = web search
• normal volumes of data at high velocities, compared to normal volume of data
  = automated trading in Capital Markets

This might be a useful way of comparing Big Data requirements against the multitude of different IT technologies and solutions out there. Today, CEP is mostly dealing with normal volumes of data at low to high velocities being tested against normal(ish) volumes of data (maybe up to Terabytes but not Petabytes), with the higher end values requiring fast datagrid solutions such as TIBCO ActiveSpaces. But as always, it would be interesting to have some metrics against the Big Data use cases  to see what we are all talking about…

No related posts.

• Real-time everything. … What I think is interesting is the growth in the data replication market specifically to support real-time BI as opposed to failover, disaster recovery, zero-downtime migrations and the like. I would not be at all surprised if we see the introduction of lightweight BI-only data replication products into the marketplace.

Right-on! The world is indeed real-time, and businesses are increasingly realising it (and in some cases, IT departments too). As for BI-only data replication… using real-time data and event technologies (e.g. TIBCO ActiveSpaces) provides the BI data alongside the operational data in real-time - you want to replicate to the database for analytics and archiving only, rather than replicate to the real-time data store!

• Continuous BI. I think we’ll hear a lot more about this as a generic market for complex event processing as opposed to the vertical markets that CEP has previously addressed.

Again, this is already happening, although the traditional BI vendors are resisting the change as much as possible. CEP vendors are using terms like operational visibility, operational intelligence, continuous intelligence, real-time analytics etc - to pretty much all indicate the same continuous BI capability. So computing statistics on-the-fly is an increasing trend, but note is not yet formalised (I haven’t seen any text book on this topic yet).

• CEP adoption by SIEM vendors…. and smaller, more agile companies … all offering CEP in this space …

Security Information and Event Management is in some ways a subset or specialisation of CEP: SIEM is about managing security events, security event processing, and management of detected security events. CEP tools are already doing this at the cyber-security scale, and most SIEM tools I see are providing (usually constrained) CEP capabilities. The problem is that every event can be considered from a security perspective (e.g. TIBCO Hawk detecting resource usage increases and monitoring log files) so there is often an advantage in taking the wider CEP view than some vendors’ limited perspective on what constitutes a security event.

• Warehousing adoption by SIEM vendors. …How can you claim to offer analytics against security and log data if you don’t have an analytic platform to support it? …

Although one should not confuse data warehouses with analytics, the implication is clear: apart from real-time analytics you should also consider long-term trend analysis and other predictive analytics against your events (security and otherwise), using appropriate visual and statistical analytic tools (e.g. TIBCO Spotfire and TIBCO Spotfire Miner respectively).

• Growth in PMML adoption. … the standard for porting data mining models. …

PMML is a great idea and hopefully it will mature more for useage across the SAS and R communities over the coming months. Apart from the analytics standard there is also work on decision modelling (DMN) which could have even wider repercussions (not that I’m biased at all!).

• Lots more big data. … As more products and companies enter this space, or claim to, the more murky the whole big data thing will become.

As more big data is created, there will be more clamour for processing it before it becomes data (i.e. as events).

• The emergence of the Data Scientist. …

Or Company Statistician? Or Business Event Analyst? To go with the Business Process Analyst and Business Decision Analyst presumably…

• The logical data warehouse. …

So is this the start of the demise of the illogical data warehouse? From an event perspective, data warehouses are really for storing old events for long term analytics purposes. That shouldn’t be a big deal…

No related posts.

In the latter category of optimization tools there is an interesting extension to S+ available called S+NuOpt. This provides techniques like LP and MIP, QP, and MOP to allow solving of constraint-rich problems like planning and routing. Traditionally these have been complex, long-running, CPU-intensive algorithms solving particular problems like “what airline routes should we offer this Summer”. But there is increasing interest in applying these resource (asset, personnel etc) optimizations against more and more exception events (such as the closure of some airports due to weather) - termed event-driven optimizations - and where the world of near-real-time event processing and operational decisions push up against the long-term strategic decision processing. Having cloud-based on-demand computing resources for irregular (but still computationally expensive) optimization computations, invoked as required through the assessment of complex events, is likely to be a key capability in future.

Although TIBCO Spotfire is commonly used to report on, and analyse results from, event processing using TIBCO BusinessEvents, I have not heard of any public case studies of NuOpt being invoked from CEP and event processing in an automated fashion. But if any usecases get published, I’ll add them here!

* Notes:

[1] See TIBCO Software 2010-2011 FY reporting on the success of CEP and analytics under “business optimisation”.

No related posts.

• “Our business optimization category, which includes both BusinessEvents and Spotfire, grew over 50% in license revenue for the year and has doubled in the past 2 years.”
• “Business optimization was the clear growth driver this quarter, up 45% over last Q4. Both Spotfire and BusinessEvents, which make up this category, showed tremendous growth, and each had its largest quarter ever. BusinessEvents is also playing a larger and larger role in our platform sales, and therefore making up a larger and larger percentage of revenue on many deals.”
• “…everybody is moving into an eventing engine. And some people — we have clients that are actually replacing app servers with event servers. …”
• “It seems like the world is moving more and more to intelligent systems and driving intelligent outcomes in various verticals. “
• “…so everybody wants to completely go to the eventing platform if you’re a retailer. … Everybody wants to be able to go all the way down to the consumer. They want to be able to make you the offer before you leave the aisle, not 6 months after you leave the store. … so Macy’s was an early TIBCO adopter to go to this eventing platform. And I think their results have shown it. …”
• “One of the use cases we’re very excited about is for cybersecurity where people are now trying to look at events and anticipate that there might be trouble. And the — what we did in the government is now all of a sudden of interest everywhere. And in addition to our technology now being built into the smart grid, they’re using the cybersecurity component because once you have a smart grid, it can be hacked and that’s not so good. So we have some fantastic use cases around that. …”

No related posts.

• TIBCO competed against a C/C++-based CEP product normally seen in Financial Services Front Office applications (i.e. extreme low latency applications). But because TIBCO BusinessEvents could effectively exploit the tested multi-threading / multi-core system, its throughput proved vastly better. Perhaps the Financial Services Front Office folk are missing a trick here?
• One fascinating project involved a Proof Of Concept (POC) that processed 500M events in 4 days, achieving up to 75K events per sec using 11 agents deployed to a 24 core server, with business control via decision tables. What surprised me was that the POC was inserted into a live operational business system - how about that for faith (or rather, reliability)!

No related posts.

I am a veteran at TIBCO software and have worked around the world for the last 12 years as a business developer and technical specialist (Europe, South America, North America, Africa and Middle East). Married to a super-model, and based in Palo Alto now.

I am happy to join the community.

If anyone has a request for specific content, please let us know.

No related posts.

Note: please comment on this post any suggested changes, births, deaths and marriages in the event processing tooling market, and I’ll incorporate the changes in the next version.

No related posts.