Events And Workflows

I was recently challenged with the task of introducing an event-driven architecture for a microservice which manages workflows. Before we get into the details let’s look at what is a workflow and what is an event.

What is a workflow?

A workflow is a representation of a sequence of tasks & can be used to represent any process in the world. Anytime data is passed between humans and/or systems, a workflow is created.

To understand it better let's take an oversimplified example of a workflow that ships an item once the order is received.

Once an order is received, the workflow checks the inventory for the availability of the item & ships the item if it's available. If not, it sends an apology mail to the order creator.

There are multiple components that constitute a workflow —

1. Order Received would be a Start Event, as this is the event that starts the workflow.
2. Check if the item is available is a Task. This represents an action taken in the workflow. In our example, the workflow would call the inventory service and check the items available.
3. Exclusive Gateway determines which step of the workflow should be executed next depending on the outcome of the previous tasks.
4. Once Ship Item or Send email tasks are finished, the process ends at the End Event.

These are just a few components but should be sufficient to get us started with workflows. Now let's try and understand what an event is.

What is an Event?

Just because you put some data into a queue does not mean that it an event.

An event, representing something that has happened in a system to which other systems may react. A popular pattern for using events is via Event Carried State Transfer. This pattern uses the event to capture all the information related to an entity.

In our example, let's consider the Order entity.

{
    "order": {
      "itemDeatails":{
           "productName":"Let us See",
           "category":"Books"
       },
      "shippingInformation": {
            "street":"221 Bakers street", 
            "city":"London",
            "pinCode":5201
       },       
       
       "paymentInformation":{
             "amount":250,
             "paymentType":"cash"
       } 
    }
}

The order entity has all the information on the order. When a user selects a product from the website, the order entity would have the item information. More information gets added to the entity as the user selects a delivery location and makes the payment. This allows multiple applications to take action when an order is placed and the action can be based on the metadata of the order (ex: amount paid).

Note — Covering all the possible eventing patterns may be out of scope for this blog. If you are interested to know more about other eventing patterns, you can refer to this article by Martin Fowler

What do you mean by "Event-Driven"?

Can commands be used instead of events?

In contrast to an event, the command specifies a state change expected in another system. Just like an event, a command can also be used to start or update a workflow. However, the command needs to be specific enough to convey the action it needs to take, to the workflow.

In our example, the command to start a workflow can look like this

{
    "workflow": {
       "name":"shipOrder",
       "orderId":"o10358",
       "action":"startWorkflow"
    }
}

Notice that the event did not have data specific to the workflow & the onus was on the downstream application to derive an action from the event. In the case of commands, the action is explicitly specified. This can be achieved by another microservice which converts the events to commands OR for the order service to emit both an event and a command.

Combining events/commands and workflows

Now that we know exactly what events and workflows are, how do we combine the two? Well, it's really simple. Firstly you would need the capability to listen to an event/command. In the next section, we will be comparing the 2 popular ways of doing it- using a streaming solution or embedding a listener within the workflow microservice.

After listening to the event, the application must act on it. It needs to have the intelligence to start/stop a workflow, move the workflow to the next step, decide which path out of the many possible paths it should follow, or simply do nothing, based on the data it has consumed.

The next question would be — is it better to use events with workflows or commands? Well, it depends on the use-case. In fact, both can be used in conjunction. You can even use both for the same workflow at different steps.

Streaming solutions vs Embedded listener

Now that we understand what to do, let's see how to it. An important decision when it comes to consuming events is whether to have a listener embedded as a part of the application or choose from one of the many streaming solution options (ex: Spark Streaming, Flink, Storm) that are available.

These are some of the factors to consider while making this decision based on the capabilities that you desire :

Conclusion

Hope this blog helped you in understanding how workflows can leverage events/commands and which solution would be more suitable for your use-case. I’ll be continuing this series with another blog (link TBD) which talks about the pitfalls that you need to be careful of when implementing such a solution.