Introduction

All applications are subject to continuing change, meaning they have to "be continually adapted or they become progressively less satisfactory" [Lehman]. Refactoring is a common technique to prepare the program's source code for newly incoming changes.

Refactoring in an event-sourced model is quite different from the more common object-relation model (ORM). In an event sourced model state is derived from events instead of being stored directly. To change the structure of an event means that a mechanism has to be provided to handle all events stored with the old structure. One of those mechanisms is called upcasting. This article will explain how upcasting can be used to facilitate refactoring in an event sourced model. We will use the Axon framework, a framework that helps applying the event-sourcing model, to explain the concepts of upcasting in detail. We'll focus on demultiplexing of events, a new feature that I implemented for the upcoming Axon version (Axon 2).

Upcasting

Originally a concept of object-oriented programming, where: "a subclass gets casted to it's superclass automatically when needed", upcasting can also be applied to event sourcing. Upcasting in an event-sourced model allows for events to be refactored while keeping the complete event history of the model (all previous state) intact. Saving the application's state this way brings several advantages (note that this list is not exhaustive):

• It can be used as an audit log, making problems easier to track down.
• All state changes are made explicit by name (e.g BirthNameDetailsUpdatedEvent), making it easy to understand what the program does.
• It allows you to keep track of historical changes, which might be relevant at a later stage in the development of your application. E.g. the client wants to perform some statistical analysis based on the usage of the application over time.

The main disadvantage of using an event-sourced model is when code needs to get refactored. Performing a big refactoring on your code base is hard because you'll have to let your code work with all events that were fired by the old code base. To make refactoring possible when historical events do not fit your new model anymore is through upcasting.

Upcasters are manually written procedures that have one input event of revision x and output, typically one, new mutated event of revision x+1. Events already stored in the database prior to the moment of refactoring get transformed to the refactored version of that event through an upcaster. Upcasting in Axon is performed in a process before events are sent to the application, meaning that the rest of the application can be completely ignorant about outdated events.

Core to the process of upcasting in Axon is the UpcasterChain: a class containing all upcasters as specified by the caller in the order specified by the caller. The upcaster chain is responsible for upcasting incoming events and delegating each event to the upcasters in the specified order. The interface of the UpcasterChain looks like this:

public class UpcasterChain { 

    private List<Upcaster> chain; 

    public List<Event> upcast(Event event);

    ...
}

The UpcasterChain combines upcasters so that one upcaster only needs to know about one event type of one specific revision. This allows the programmer to write upcasters that are small, isolated, and easy to understand.

Upcasting and demultiplexing

To demultiplex data is to take data from a single input channel and divide that data over multiple output channels. When we apply this concept to upcasting this means that one event can be transformed into multiple events through an upcaster.

Each upcaster indicates whether it can process the type of the event that needs upcasting. If the upcaster's type matches with the type of the incoming event the chain will pass the event to the upcaster. The upcaster will then process the event and return zero, one or multiple 'upcasted' events. Since upcasters are arranged in a chain all events coming from the upcaster are passed into all remaining upcasters that are still left in the chain, as shown in the following pseudo code:

method upcast(chain, event):
    if chain is empty then
        no upcasting needs to be done return a singleton list
        with the given event
    else
        upcast the event with the first upcaster in the list, like this:
        processed events = chain.head.upcast(event) 

        process each upcasted event through all remaining upcasters
        in the chain (chain.tail) by recursively calling the upcast
        method on all these events and return the result as a list.
        this list will represents all fully upcasted events.

To illustrate how demultiplexing works in practice we'll explain a simple case of updating administrative details for a user:

It so happens to be that there is a system running in production containing many AdministrativeDetailsUpdated events. New requirements have let you to introduce two new events: AddressUpdatedEvent and InsuranceDetailsUpdatedEvent which are processed by different parts of the application. Previously though, all information in these two events was contained in the old AdministrativeDetailsUpdatedEvent, which is now deprecated. In Axon 1 it is not possible to handle the situation described in the example properly because upcasters cannot do demultiplexing. In other words, the granularity of old events cannot be changed through upcasting so there is no way of nicely separating the business logic of handling address changes and insurance details changes. Ultimately this will either make programmers reluctant in refactoring their model or it will make them introduce hacks to let it work, degrading the readability of the source code. In Axon 2 an upcaster can be implemented to transform these events into instances of AddressUpdatedEvent and InsuranceDetailsUpdatedEvent and pass them to the application code, solving the previously described problem.

Conclusion

We've introduced the concept of upcasting in an event-sourced application and we've shown how this concept is useful by providing a real world example. We've shown that upcasting in Axon 2 is more flexible from its Axon 1 counterpart due to its ability to demultiplex events.

References

[Lehman] Lehman, M. M. (1980). "On Understanding Laws, Evolution, and Conservation in the Large-Program Life Cycle". Journal of Systems and Software

This blog post focusses on the Google App Engine module that can now be found in the incubator. I'll summarize the required changes to the axon framework so it can be used on google app engine. I did blog about this topic before, so I am not going into details for all the code, but point to the older blogpost where possible and focus on the new stuff.

Overview

In my previous blogpost I discussed the following topics.

• Gradle for building the sample
• XStream (the used serializer) hacks required for google app engine. Things have changed in this area, I will discuss them later on.
• Event store based on the RAW google app engine storage api. Only the package has changed.
• Other technologies for the sample like: Spring web, Spring security and sitemesh

?

So what has changed since that blog post?

New dependencies

We can now find the latest and greatest gae artifacts in the central repository and we have moved to the latest jdk (1.5.1). Next to the google app engine dependencies we now also have a dependency on the incubator module from axon. The following block shows the most important changes in the dependencies for the gradle build file. Have a look at the build.gradle file for the complete config.

    compile "org.axonframework:axon-core:$axonVersion",
            "org.axonframework.incubator:axon-googleappengine:1.1-SNAPSHOT",
            "com.google.appengine:appengine-api-1.0-sdk:$gaeVersion"

I also removed the gae lab dependency. This was required for the task api, but in 1.5.1 the task api has become a standard module for google app engine.

Axon incubator module for google app engine

What does the incubator module for google app engine provide? it contains additions for serialization, the event store and command/event bus implementations. In the future the event and command bus implementations will not be needed. We are going to make changes in the statistics part of axon.

Serialization

This part contains two classes. We have a utility class for Spring enabled applications. This is the XStreamFactory class. This factory initializes GaeExtream class with a PureJavaReflectionProvider. Some of the default providers that XStream usually uses are not supported on Google App Engine. Therefore we have our own GaeXStream class. When writing the previous blog post there was no way to provide your own instance of an XStream class. This has changed since then. Now we can provide our own XStream class. Therefore we do not need the GaeGenericXStreamSerializer and XStreamEventSerializer anymore.

Event store

Not a lot to mention here. I was pretty detailed in the other blog post. We have three classes required for the event store. We have the object to store, the EventEntry. Next to the entry we have The GaeEventStore and the GaeSnapshotter.

Event and Command bus

In the current snapshot version of axon, the SimpleCommandBus and the SimpleEventBus are MBean enabled. This class is not in the supported classes of GoogleAppEngine. Therefore we have to create our own event and command bus. They both are exact copies without the marker interface and the statistics stuff.

The sample project

The CompanyHr project is a google app engine project that makes use of axon. The google app engine hacks are now in the axon incubator module. Therefore we have removed them from the sample project. The sample makes use of basic spring mvc and it integrates the spring security module with google authentication.

All web related technologies have been mentioned in the previous blog post. Therefore I do not explain them here again.

Concluding

Integrating with Google app engine is not hard. We do have to make a few changes that are supported by axon. The only thing that is not very nice is the statistics problems. But we will come with a solution in that area as well.

If you still have questions or remarks, feel free to post a comment or use the axon user group.

References

• http://blog.jteam.nl/2011/01/04/deploying-a-cqrs-application-based-on-the-axon-framework-on-google-app-engine/
• https://github.com/jettro/CompanyHr
• http://code.google.com/p/axonframework/source/browse/trunk#trunk%2Fincubator%2Fgoogle-app-engine

?

Introduction

Let us have a high level look at what I had to do to make the sample work.

Axon framework

If you are not familiar with Axon and want to learn more about axon, this is not the best blogpost to start with. Start reading the excellent reference manual of Axon. If you want to create your own application with Axon on GAE, you have to use the lastest and greatest version of Axon. I am working with the 0.7-SNAPSHOT and I had to patch it a little bit. More on this later on, but if you are interested you can follow this issue.

There are a few things I have created to make Axon run on GAE. For starters, I needed an event store and a query database that make use of the google app engine data store. The other thing I needed was a snapshotter. Like the name says, the snapshotter has the responsibility to initiate creating a snapshot of an aggregate root. The snapshotter makes use of the GAE Task Queues.

By default, Axon makes use of XStream. Although you can configure Axon to use something else, I have chosen to use XStream. XStream and GAE are not best friends, I had to create some tweaks to make it work.

Objectify

I started of using jpa to store everything in the GAE datastore. I had tried this path in the past, but it did not feel right. For the event store I decided to use the low level api. For the query database I wanted to make my life easier. Therefore I decided to look for a framework. I found Objectify to give what I needed.

Objectify-Appengine provides the thinnest convenient layer which addresses these issues, yet preserves the elegance of get, put, delete, and query.

Because I want to store the EventEntry objects in different collections based on their type I did not see advantages in using Objectify for the EventStore as well. Therefore I sticked to the low level api of the data store.

Spring security

For the application I am creating I want to implement security. Since I am used to Spring security I wanted to use spring security. I found an excellent blog post and a sample in the trunk for spring security that helped me a lot.

Other tools

Other tools I used that require less attention are:

• Spring web
• Gradle - for building the sample
• Intellij - for writing code and running the local GAE instance
• Gae SDK - version 1.4 at the time of writing
• Sitemesh - For templating the pages
• jQuery - To make the sample look just a little bit better

?

The sample

I decided to start this sample to automate some of the basic processes around Human Resource Management in a smaller company. I am starting of with a CV creating tool. I am not finished by far. Actually I have just started and I focussed on getting it to work. If you want to have a look at the code, you can find it at Github.

https://github.com/jettro/CompanyHr

I use gradle to build the sample. Tried to find a good GAE plugin, but did not really succeed. Therefore I build with gradle, generate the intellij project files with gradle and run the developer version of GAE from intellij. Uploading the sample is done using the command line utilities provided by the google app engine sdk. If you are interested in gradle or want to build the sample yourself, have a look at the next section. If you are not interested in building the sample, it is safe to skip this section.

Gradle

First you need to be able to find all the dependencies. The section that configures the repositories, shows a few gradle tricks. The first repo to use is you local maven repository. Beware, if you do not keep your maven repository in you home folder under .m2/repository this will not work. The second trick is about finding the google app engine jars. I use the feature to use a flat directory as a repository. You can configure the location of google app engine in the gradle.properties file. Finally we use the central maven repository and some additional repositories for axon snapshots and objectify. The following code block shows the configuration.

repositories {
    mavenRepo urls: ["file://" + System.getProperty('user.home') + "/.m2/repository/"]
    flatDir(dirs: "$gaeHome/lib/")
    mavenCentral()
    mavenRepo urls: [
            "http://oss.sonatype.org/content/groups/public",
            "http://objectify-appengine.googlecode.com/svn/maven"]
}

The other bit of gradle I want to mention is the exploded war file. The google upload utility needs this. We unzip the war to an external folder as a last stage of the war creation. The following code block shows the configuration of the exploded directory.

war.doLast {
    ant.unzip src: war.archivePath, dest: explodedWar
}

Now you can build the project using gradle: gradle clean build. Or you do gradle idea and run everything from intellij.

XStream hacks

As I mentioned before, axon uses XStream to serialize and deserialize objects. A lot of reflection power is used to re-instantiate serialized objects. With GAE there are some limitations in this area. The most important change is in the XStream class, this is the most used class with the XStream library. Within this class Converters are registered. Some of these converters do not work on google app engine. Therefore I have created a subclass of XStream called GaeXStream. I removed the dynamic loading of converters as well as the jdk specific configurations. You can find the classes in the package nl.gridshore.companyhr.axongeahacks.

Axon provides a GenericXStreamSerializer, this class initializes an XStream object. At the moment there is no way to provide your own XStream object. Maybe in the future we can provide a constructor for this class that accepts an XStream object. For now I have created my own implementation of this class,

.

The final class I had to change was the

. This class creates the GenericXStreamSerializer. Again there is no real mechanism to change this construction. I have copied the original class and changed the line where the GenericXStreamSerializer is created into the creation of my GaeGenericXStreamSerializer. My own class is called GaeXStreamEventSerializer.

In the future I'll try to simplify this and maybe change Axon to make this a little bit easier. For now this is working.

Axon Event Store

I already created an EventStore that uses MongoDB. Check my previous blog posts on gridshore.

http://www.gridshore.nl/2010/09/20/learning-mongodb/

http://www.gridshore.nl/2010/09/27/still-learning-mongodb/

Allard improved the code for the MongoDB implementation and now I can reuse the ideas for the Google App Engine Event Store.

EventEntry

The EventEntry is the representation of what we store in the google data store. It has a few fields like aggregateIdentifier and the serializedEvent. The serialized event is our actual content serialized using XStream. The EventEntry also contains some methods to help us with the translation to google data store Entity objects and queries. To give you an idea, the following code block shows you the method to create a query for returning all Entities based on the provided aggregateIdentifier after the provided sequence number. You can find the sourcecode for the complete class here

static Query forAggregate(String type, String aggregateIdentifier, long firstSequenceNumber) {
    return new Query(type)
            .addFilter(AGGREGATE_IDENTIFIER, Query.FilterOperator.EQUAL, aggregateIdentifier)
            .addFilter(SEQUENCE_NUMBER, Query.FilterOperator.GREATER_THAN_OR_EQUAL, firstSequenceNumber)
            .addSort(SEQUENCE_NUMBER, Query.SortDirection.ASCENDING);
}

EventStore

For the GAE event store I implement the org.axonframework.eventstore.SnapshotEventStore interface. The implementation supports storing the events as well as working with snapshot events. GAE provides the DatastoreServiceFactory. From this factory we obtain the ServiceFactory. The event store makes use of the XStream event serializer that was discussed before in this post. All aggregate types are stored in separate collections. We could store them all in the same collection, but according to google tips you should try to limit the amount of items in a collection and especially the amount of queries performed on a collection. We also store the snapshots in a different collection with respect to the normal events. The following image shows you the collections my app currently has.

As you can see, project has three tables. Project, snapshot_Project and ProjectEntry. Project contains the events, snapshot_Project contains the snapshots and ProjectEntry contains the projects from the query database perspective.

Now let us have a look at the implementation. You can find the code here: nl.gridshore.companyhr.app.axon.GaeEventStore. When writing to the collections you have to understand transactions and google app engine. GAE does support transactions, but only within one aggregate root. Therefore we have to create new transactions for all events that we need to store, they are all aggregate roots. We could change this and create one document with all its events as an aggregate root, but I did not choose that direction. Appending an event now consists of a few steps. Loop over all provided DomainEvents, convert them into EventEntry objects, create transaction, store objects within transaction and commit. The following code block shows these steps in code. In the real code base this is refactored, but this is easier to read.

public void appendEvents(String type, DomainEventStream events) {
    while (events.hasNext()) {
        DomainEvent event = events.next();
        EventEntry entry = new EventEntry(type, event, eventSerializer);
        Transaction transaction = datastoreService.beginTransaction();
        try {
            datastoreService.put(transaction, entry.asEntity());
            transaction.commit();
        } finally {
            if (transaction.isActive()) {
                transaction.rollback();
            }
        }
    }
}

Storing snapshots is done in the exact same way. Loading an aggregate root is done using all events or all events starting from a snapshot event. To give you an idea about reading objects using GAE api, I'll show you how to load the last snapshot event for an aggregate root. The query object is created using the special factory method in EventEntry, then we prepare the object and iterate over the results of the query. Only the most recent snapshot is needed. Check the following code.

// from GaeEventStore
private EventEntry loadLastSnapshotEvent(String type, AggregateIdentifier identifier) {
    Query query = EventEntry.forLastSnapshot("snapshot_" + type, identifier.asString());
    PreparedQuery preparedQuery = datastoreService.prepare(query);
    Iterator<Entity> entityIterator = preparedQuery.asIterable().iterator();
    if (entityIterator.hasNext()) {
        Entity lastSnapshot = entityIterator.next();
        return new EventEntry(lastSnapshot);
    }
    return null;
}

// from EventEntry
static Query forLastSnapshot(String type, String aggregateIdentifier) {
    return new Query(type)
            .addFilter(AGGREGATE_IDENTIFIER, Query.FilterOperator.EQUAL, aggregateIdentifier)
            .addSort(SEQUENCE_NUMBER, Query.SortDirection.ASCENDING);
}

The other part I want to have a look at, is the next step in recovering an aggregate root. Reading all events that happened after a snapshot event. The next code block shows the most important part for the implementation.

private List<DomainEvent> readEventSegmentInternal(String type, AggregateIdentifier identifier,
                                                   long firstSequenceNumber) {
    Query query = EventEntry.forAggregate(type, identifier.asString(), firstSequenceNumber);
    PreparedQuery preparedQuery = datastoreService.prepare(query);
    List<Entity> entities = preparedQuery.asList(FetchOptions.Builder.withDefaults());

    List<DomainEvent> events = new ArrayList<DomainEvent>(entities.size());
    for (Entity entity : entities) {
        byte[] bytes = ((Text) entity.getProperty("serializedEvent")).getValue().getBytes(EventEntry.UTF8);
        events.add(eventSerializer.deserialize(bytes));
    }
    return events;
}

Since a serialized event can become big, we store it in a Text field. As you can see we need some steps to extract the actual content. Other than that the code is easy to follow I hope.

I did not implement the org.axonframework.eventstore.EventStoreManagement interface yet. I Will do this in the nearby future.

With the EventStore implementation we can store and read events as well as snapshot events. Axon also has support for deciding when to create a snapshot. Google app engine provides the tasks api. Therefore I wanted to have an implementation of the org.axonframework.eventsourcing.Snapshotter interface that makes use of the task api.

Snapshotter

Creating your own snapshotter is not hard, it is an easy interface with just one method. The next code block shows you the implementation of the interface.

public void scheduleSnapshot(String typeIdentifier, AggregateIdentifier aggregateIdentifier) {
    logger.debug("Schedule a new task to create a snapshot for type {} and aggregate {}",
            typeIdentifier, aggregateIdentifier);

    Queue queue = QueueFactory.getQueue("snapshotter");

    queue.add(TaskOptions.Builder.withUrl("/task/snapshot")
            .param("typeIdentifier", typeIdentifier)
            .param("aggregateIdentifier", aggregateIdentifier.asString())
            .method(TaskOptions.Method.POST)
    );
}

As you can see, the code is very easy. We do a post to a certain url with the parameters typeIdentifier and aggregateIdentifier. Then in a controller we intercept this url and call the following method.

public void createSnapshot(String typeIdentifier, String aggregateIdentifier) {
    DomainEventStream eventStream =
            eventStore.readEvents(typeIdentifier, new StringAggregateIdentifier(aggregateIdentifier));
    DomainEvent snapshotEvent = createSnapshot(typeIdentifier, eventStream);
    if (snapshotEvent != null) {
        eventStore.appendSnapshotEvent(typeIdentifier, snapshotEvent);
    }
}

private DomainEvent createSnapshot(String typeIdentifier, DomainEventStream eventStream) {
    AggregateFactory<?> aggregateFactory = aggregateFactories.get(typeIdentifier);

    DomainEvent firstEvent = eventStream.peek();
    AggregateIdentifier aggregateIdentifier = firstEvent.getAggregateIdentifier();

    EventSourcedAggregateRoot aggregate = aggregateFactory.createAggregate(aggregateIdentifier, firstEvent);
    aggregate.initializeState(eventStream);

    return new AggregateSnapshot<EventSourcedAggregateRoot>(aggregate);
}

That is all not to hard, now we have an a-synchronous snapshotter. An alternative would be to use the standard axon implementations with the org.axonframework.util.DirectExecutor and the org.axonframework.eventsourcing.SpringAggregateSnapshotter. I leave this as an exercise to yourself

Concluding

That is it, this is what is required to use axon on google app engine. Now you can configure the CommandBus, the EventBus, repositories, snapshotters, etc. In future posts I'll have a more detailed look at some of the other aspects of the sample like Objectify and the spring security implementation. If you want more information on certain topics, feel free to ask for it in the comment section. I'll answer your questions and could even write a new blog post if the answer becomes to big