The quest for Business Agility: Emergent Architectures
|22 February 2011|
"Business Agility", every enterprise wants to have this ability, don't they? Business Agility is the ability of an enterprise to adapt rapidly and cost efficiently in response to changes in the business environment. James McGovern stated it nicely: "Increasingly, the stability of an enterprise is rooted in its ability to be dynamic, to move fast and change quickly".
For enterprises to become agile a lot of things have to become more flexible. This doesn't only ask for changes in the structure and behaviour of an enterprise itself. As the level of automation in most enterprises is only increasing, the role of IT is an important factor in enabling business agility. However, enterprise IT isn't known for its ability to be agile...
Hence, today's question: how to build an agile enterprise?
Article highlights (for the tl;dr crowd):
- We need to simplify an enterprise to understand it, to reason about it, and to adapt it, but we shouldn't neglect its complex behaviour by linearizing it.
- We need emergent enterprise architectures focusing on the essence of the construction of an enterprise: social interaction among actors defined by transactions and rules.
- If we go down the road of emergent enterprise architectures, we will need emergent IT architectures too. Based on the definition of the parts (components with their services and events), the whole emerges.
- As changes on the service level will lead to new services or changes in the implementation of existing services, we also need agility in implementing new components and changing existing ones. To provide this agility, agile component life-cycle management is needed, rooted in model-driven techniques.
- To conclude: we need a vision on the social enterprise, focus on emergent architectures, and agile, model-driven component life-cycle management to build an agile enterprise.
Systems and ComplexityIf you ever walked around in a rather big enterprise you probably noticed that people in such enterprises are busy with executing all kinds of activities, thereby using a variety of tools. Employees are communicating, exchanging information in all forms. As a naive observer you probably asked yourself what everybody is doing and why. You probably noticed the complexity, but didn't really understand the structure and logic of it. However, the end result apparently lead (in most cases) to the selling of goods or services to the market.
As it is difficult to understand the structure and logic of an enterprise, it is even more difficult to change it. Everybody who executed a change program in an enterprise will admit the difficulty of such operations. Although some change programs succeed, it is questionable whether this is due to understanding, designing, and deliberately changing things. Nevertheless, in order to be agile, an enterprise needs to be able to change quickly.
An enterprise can be seen as a system. When we look at systems we can distinguish between their function and their construction. The function of a system can be described with a black-box model and describes the external behaviour of a system. If we want to use or control a system we only need to know about the function of a system. In order to change a system we need to have knowledge concerning the construction and operation of that system. The construction of a system is modeled with a white-box model.
When I say that enterprises are complex systems I mean that the construction of an enterprise is complicated and complex. It's important to understand the difference between complex and complicated. A system can be categorized along two dimensions:
- The structure of a system, which can be simple (easily understandable) or complicated (very hard to understand).
- The behaviour of a system, which can be ordered (fully predictable), complex (somewhat predictable but with many surprises), and chaotic (very unpredictable).
Simplification is the act of making the structure of a system better understandable. Linearization is the act of making the behaviour of a system better predictable (see this excellent article on simplicity by Jurgen Appelo). The use of models can help us to make systems better understandable (simplification) but we need to be aware not to change the behaviour / meaning of the system (linearization). Changing the behaviour of a system means changing the kind of system, which will not help in understanding the actual system.
When we think about enterprises we need to be aware of the fact that they are complex systems. We need to simplify an enterprise to understand it, to reason about it, and to adapt it, but we shouldn't neglect its complex behaviour by linearizing it (i.e. by making it more predictable).
Understanding and modeling complex enterprises
If we look at ways to model an enterprise nowadays, we often see process-oriented approaches. The question is if a business process is simplifying an enterprise. Does a business process model create a better understanding of an enterprise? Or is a business process a linearization of an enterprise? Let's look at an enterprise in more detail to search for the right concepts to model an enterprise.
As I said before, an enterprise is a system. A system has a function and a construction. A system is in most cases used by other systems. Think about a driver and a car. The driver (a biological system) uses the car (a mechanical system). In such cases we talk about a using system (the driver) and an object system (the car). The construction of the using system, uses the function of the object system. Reasoning about developing or changing the object system can be done using the Generic System Development Process .
In case of an enterprise we can say that the market is the using system, the enterprise is the object system. Actors in the market request goods or services, while actors within the enterprise deliver these goods or services. In this case, actors are social individuals who perform two kinds of acts: production-acts and coordination-acts (according to the PSI theory ).
By performing production-acts the actors contribute to the delivery of goods or services. By performing coordination-acts actors enter into and comply with commitments towards each other regarding the performance of production-acts. Coordination-acts consist of an intention (e.g. request, promise, decline) and result in a commitment of the performer and the addressee regarding the bringing about of the corresponding production-act. Coordination-acts and production-acts occur as steps in a generic pattern, called transaction.
This may all sound a bit abstract (read more about Enterprise Ontology), but let's look at an example, visualized in Figure 1. If you go to a restaurant, you will probably start a transaction with the waiter. You request some food, he promises to produce it for you. The waiter will start a transaction with the kitchen in the same pattern (you can figure the details yourself). Once the food is produced the waiter states the result, i.e. he serves you the food. You accept it by thanking him and starting to eat.
Figure 1 - The transaction pattern
(copyright - and used with permission of - ICRIS B.V.)
Transactions always have a consumer and a producer. These two actors perform the different steps in a transactions based on a set of rules. Rules describe on what conditions an actor produces coordination-acts and production-acts. Hence, we can say an enterprise consists of autonomous actors reacting to events (incoming transaction state changes), acting on these events, and producing events (outgoing transaction state changes), all according to their own set of rules.
If we want to understand an enterprise we need to look trough the distracting and confusing actual appearance of an enterprise. By looking at the transactions among actors we focus on the essence of an enterprise, without linearizing! The essence of an enterprise is the social interaction and collaboration among actors, which is perfectly covered by the transaction pattern described before.
This view on enterprise has impact on how we look at and manage people in enterprises. Humans are autonomous and inherently agile. That's why new leadership models are taking the stage. Teams can become more powerful if you empower your team members in a growing level of authority. However, that's not the topic of this article, so, back to modeling.
I did start this section with questioning the role of business process models. Do they simplify reality (i.e. making it easier to understand the construction of an enterprise) or do they linearize it (i.e. making the behaviour more predictable but thereby neglecting the real-world complexity)? It depends. In my opinion business process models can give a good overview of some of the structures within an organization. However, we shouldn't see them as a model of the construction of an enterprise.
The construction of an enterprise can best be modeled by showing the actors and their transactions. Processes emerge from these individual transactions. A business process can be defined as :
A business process is a collection of causally related transaction types, such that the starting step is either a request performed by an actor role in the environment (external activation) or a request by an internal actor role to itself (self-activation).
A holistic view on the enterprise is needed to understand its essence. A model of the construction of an enterprise provides this view by showing the structure of the enterprise. The behaviour can be modeled with a set of rules for each autonomous actor. Modeling an enterprise in this way emphasizes that processes in an enterprise emerge from social collaboration among actors.
This "emergence" also shows how business agility can be reached: not by imposing hierarchical structures and processes, but by empowering people. The best products and services are not defined, but evolve organically, adapting to their environment as they grow. This also holds for the structure and behaviour of an enterprise itself: it needs to evolve and adapt. That's why we need emergent enterprise architectures focusing on the essence of the construction of an enterprise: social interaction among actors defined by transactions and rules.
Understanding and modeling complex IT systems
Let's move to the realm of software. As I stated in the introduction, the level of automation in enterprises is only increasing. This leads to an ever growing complexity of the IT systems in enterprises. How can we create models of these IT systems to understand them without linearizing them?
Like enterprises, software can be seen as a system too. In the context of an enterprise we can say that the enterprise is the using system, while the IT system is the object system. The construction of the enterprise makes use of the function of the IT system. In other words: the function of the IT system supports the construction of the enterprise. As we have seen in the previous section the construction of an enterprise consists of actors and transactions. Hence, an IT system needs to support actors (in some cases they can even implement them) and transactions.
If we want to model such an IT system it's important to distinguish between the function and the construction of the system as visualized in Figure 2. The function of a system is always described in terms of the using system and is modeled with a so-called black-box model. This function model of an IT system supporting an enterprise can be seen as a service specification model. Each described service supports an actor in executing a transaction . The construction of a system is modeled with a white-box model describing the different components and their interaction relationships. In case of an IT system the construction model describes the components implementing the services specified in the function model.
Figure 2 - The function and construction of an emergent IT system
Mirroring the complexity of an enterprise with an IT system asks for a service specification model which is inherently flexible. Following the construction model of an enterprise this model should describe services (supporting an actor in executing a transaction) with their
- pre-conditions (incoming events and the rules which describe how to handle them),
- post-conditions (outgoing events and created facts), and
- the actual task (supporting or implementing a production-act) which is initialized when the pre-conditions are met. This can be a human or a system task.
From these kind of service descriptions the orchestrations or flows automatically emerge in the same way as how processes in an enterprise emerge from the social collaboration among actors. Due to the central notion of events, such an approach to modeling and implementing IT systems if often referred to as information-centric SOA or Event-Driven Architecture (recently I also saw terms like social BPM). Figure 2 shows a basic overview of such an IT system.
While the service specification model describes the function of the IT system of an enterprise, the construction of the IT system should actually implement the tasks described for each service. I believe it is important to distinct between the function of an IT system mirroring the real-world (problem domain) as close as possible, and the construction of an IT system based on a sound technical design describing the task implementations, components, and interactions from a technical perspective (solution domain). The decision what tasks to implement in which component, with what technology, and how to deploy these components often needs to be taken by a different role than the role deciding on the needed services and their interactions.
If we look at the actual implementation of components (containing the implementation of at least one task) we can describe it in a lot of detail (like I did almost 3 years ago with my article about architecture requirements for Service-Oriented Business Applications), but I think it's more important to just describe the essential elements of such implementations as the detailed technology choices can differ a lot based on the context. In essence, components need to provide implementations for:
- Human tasks: a human being needs to interact with the system, using a user interface, and retrieve and store data to complete the task.
- System tasks: some automated logic needs to be executed.
- Data services: both human and system tasks need all kinds of data. Some of this data is part of the incoming service calls, other data needs to be retrieved during the execution of the task. Hence, components need to provide data services for these task implementations. This can be done using a separate database for each component or using a central data storage solution.
So, how can we model an IT system in such a way that it's easy to understand without linearizing the system? The essence of an IT system (or maybe we should call it an application landscape or architecture) is that it consists of a set of components (or applications if you like) providing services to other systems or humans in order to support an organization. If we model the function (the services and their interactions) with fixed service orchestrations, we are (in most cases) linearizing the complexity of the IT system. As the function directly supports the construction of the enterprise, it should mirror the event-driven nature of the enterprise (as explained in the previous section: the essence of an enterprise is the social interaction and collaboration among actors). Modeling the essence of the construction of an IT system boils down to defining what services are implemented in which components and where the data is stored and managed.
If we go down the road of emergent enterprise architectures, we will need emergent IT architectures too. Based on the definition of the parts (components with their services and events), the whole emerges. True business agility can only be reached by having alignment between evolutionary architectures on the organization and technology level.
Designing and growing IT systems to enable business agility
Are you still with me? Then you're now probably thinking along the lines of "nice theoretical story, but it won't work in practice" or "ivory tower architecture thoughts, miles from reality". You're right. Only in theory you can just change the interaction of services to adapt to changes at the business level. Only in theory such an architecture is agile enough to facilitate business agility by changing pre- and/or post-conditions of services. In practice, you almost always need to add new services or you'll have to change the implementation of existing services (i.e. the component implementations). How can we make sure this isn't slowing us down?
Don't get me wrong, I think the previous sections about emergent enterprise architectures and emergent IT architectures are important! However, we cannot stop there. We also need to make sure the component implementations are easy and fast to adapt too, and don't forget that we'll probably need to build new components to facilitate the new or changed services. As you'd expect by now, "agile" and "emergent" will again be keywords in finding a solution.
In 2001 seventeen people met in Utah to find some common ground to uncover better ways of developing software. They came up with the, since then historic, Manifesto for Agile Software Development, which emphasizes individuals and interactions, working software, customer collaboration, and responding to change. Agile approaches, often referred to as "lightweight" approaches, focus on delivering software in relatively short iterations while requirements evolve through collaboration. From a technical perspective things like continuous integration, test-driven development, refactoring, etc. are needed to keep the implementation agile.
The interesting thing (in the context of this article) about the agile practice is that it advocates emergent architectures. Instead of creating a big design up front, the design and architecture of a system evolves along with the requirements. During the iterations the architecture emerges as requirements-driven changes are applied. This doesn't mean you can just hack away without any design or architecture. See  for a nice series of articles about evolutionary and emergent design to handle complexity in a proper way.
Is agile software development the solution to our challenge of developing and changing components fast and easy? It depends to what development process you're applying the agile principles. If you still develop your software using, for example, Java, you'll still need quite some time to apply changes. In my opinion we need Model-Driven Development (MDD) to close this gap. Model-Driven Development can provide the needed agility on the implementation level by providing short iterations and improved collaboration between business and IT by defining the implementation with a ubiquitous language understandable by both business and IT people.
It gets a bit boring to say, but we're still not there. As explained in my previous article MDD is necessary, but not sufficient. The full application life-cycle needs to be automated as much as possible, to provide us with the agility we need. We need to support the requirements gathering process. We need to translate requirements into models and deploy these models in an easy way. Once an application is deployed we need to gather feedback from users and translate this feedback into new requirements, create a new model version, and deploy a new version of the application. If this application (or component) life-cycle can be automated in a proper way, we elevate the main bottleneck in our quest for business agility.
So, in order to support business agility we need emergent enterprise and IT architectures, which, in their turn, ask for agility in implementing new components and changing existing ones. To provide this agility, agile application life-cycle management is needed, rooted in model-driven techniques.
It's about time to define in more detail what "business agility" means in practice. I started this article by saying that business agility is the ability of an enterprise to adapt rapidly and cost efficiently in response to changes in the business environment. Using the terminology introduced in this article, this means:
- The function of an enterprise needs to change to properly support the changed market (business environment).
- The construction of an enterprise needs to change in order to provide the new function. This means changes in transactions and actors.
- The function of the IT system needs to change to properly support the changed enterprise. Service definitions will need to change as well as their interactions. In a lot of cases we probably need new services.
- The construction the IT system needs to change in order to provide the new function. This means changes in the implementation of existing components as well as implementing new components.
All these steps need to be executed in an unprecedented speed to stay ahead of the competition. To handle the complexity resulting from this quest for agility on all layers in the enterprise, we need triple emergence:
- emergent enterprise architecture: focusing on the essence of the construction of an enterprise - social interaction among actors defined by transactions and rules.
- emergent IT architecture: based on the definition of the parts (components with their services and events), the whole emerges.
- emergent design and implementation: agility in implementing new components and changing existing ones provided by agile application life-cycle management, rooted in model-driven techniques.
The challenge for emergent architecture is the validation and verification. One of the downsides of emergence is that it isn't easy to track the behaviour of the system to individual parts and rules. This can be unpredictable and difficult to steer. Furthermore, as you need to start small (e.g. by implementing a single component) the system needs to grow continuously in a controlled way. This all asks for tools to manage the services and their interactions as well as tools to simulate and debug a whole landscape with thousands of events. On top of that you'll need the tools to implement and change the needed components. Tools that support agile component / application life-cycle management, rooted in model-driven techniques.
Let's finish up. Today's questions was: how to build an agile enterprise? The answer: a vision on the social enterprise, focus on emergent architectures, and agile, model-driven component life-cycle management.
 J. L. G. Dietz. Architecture - Building strategy into design. Academic service, The Hague, The Netherlands, 2008.
 J. L. G. Dietz. Enterprise ontology - understanding the essence of organizational operation. Enterprise Information Systems VII, pages 19-30, 2006.
 J. den Haan. An Enterprise Ontology based approach to Model-Driven Engineering. Technical report, Delft University of Technology, 2009.
 Neal Ford. Evolutionary architecture and emergent design. IBM developerWorks, 2011. http://www.ibm.com/developerworks/java/library/j-eaed19/index.html
If you're not tired of reading, you can check these articles which provide more background / context:
- Architecture requirements for Service-Oriented Business Applications
- SOA is dead; long live Model-Driven SOA
- A Framework for Model-Driven SOA
- Modeling an organization using Enterprise Ontology
- An Enterprise Ontology based approach to Model-Driven Engineering
- The Process Centric vs. Information Centric approach to SOA
- Pragmatic new models for enterprise architecture take shape by Dion Hinchcliffe
Photo by qthomasbower
Johan, you said:
The full application life-cycle needs to be automated as much as possible, to provide us with the agility we need. We need to support the requirements gathering process. We need to translate requirements into models and deploy these models in an easy way. Once an application is deployed we need to gather feedback from users and translate this feedback into new requirements, create a new model version, and deploy a new version of the application. If this application (or component) life-cycle can be automated in a proper way, we elevate the main bottleneck in our quest for business agility.
Is it possible to automate the full application life-cycle and how can we do that? Isn’t it that every time you try adapt to your business requirements they’re already started changing. So is there an endpoint in this process?
Samet Kaya (URL) - 25 02 11 - 00:46
> Is it possible to automate the full application life-cycle and how can we do that?
It is possible to automate most of the steps or at least support them with tooling to make your life a lot easier. Think about tool support to write and plan user stories, tool support to connect user stories to your application model, tool support to easily deploy that model in a cloud environment, tool support to gather feedback and turn that feedback into new user stories, etc. You’ll of course need manual input, but the cycles in the life of an application can become quite fast.
> Isn’t it that every time you try adapt to your business requirements they’re already started changing. So is there an endpoint in this process?
Good question. I think you’re right. That’s why you need to be as flexible as possible. However, don’t forget that people need to change too when business requirements / processes / transactions change. In a lot of cases that can be more difficult than changing the software.
And yes… I think it’s a never-ending story. As long as an enterprise exists, it will have to change.
Johan den Haan () (URL) - 25 02 11 - 17:32
We’re already building our apps with Mendix. We want to improve the business agility by using Scrum. I think Scrum and Mendix is the perfectly combined mix to automate your processes. At least we will get more feedback from our customers and they will under their own requirements better. What do you think about Scrum in combination with Mendix? Do you believe in practices like Scrum?
Samet Kaya () (URL) - 25 02 11 - 22:23
As stated in the article, I think MDD is a necessary element if you want to reach the full potential of agile software development. On the other hand, in the context of business agility MDD isn’t sufficient, you’ll need to combine it with an agile approach to managing your projects (like Scrum), but you’ll also have to take a holistic approach to all aspects of enterprise architecture, from organization design and people to technology and application lifecycle management.
So, to answer your question: I think Mendix and Scrum can be a happy marriage.
Johan den Haan () (URL) - 26 02 11 - 11:31
Thanks Johan for your explanation.
Samet Kaya () (URL) - 26 02 11 - 12:14
From Johan:[It is interesting to see how you focus on a holistic view on the enterprise instead of an app focus. However, how realistic is it to have such a view in place? Isn’t that too complex? Doesn’t it change too much?
Don’t get me wrong: I believe in a holistic approach, but only if we are able to focus on the essence of an enterprise, see: http://www.theenterprisearchitect.eu/arc.... Can you maybe explain in the comments on that article how this compares to your views?]
I appreciate your comments, and yes it is realistic; I have delivered 3 whole enterprise architectures with integrated applications and shared data structures for the whole application portfolio. The rewards are high with significantly higher capability and much lower cost and complexity. I am dreaming of a 4th. One of the jobs of Enterprise Architect is to find what can be stable, what can be reused, what can be made consistent, what parts need to be de-coupled, how to engineer adaptive assets. My mindset is engineering machinery. In manufacturing they don’t build new machines for each variation of the product, they use process control systems and machine settings to change. The complexity reduction that can be had is enormous. However, standing too close to anything large and complex causes one to not be able to do this; and not changing perspectives (too much hierarchy thinking for example) also causes loss of vision. One has to Architect in the Meta World and very few people go there for architecture.
The usual position is it CANT be done. My mother was Dutch and for her the word did not exist. My other problem is that I happened to believe that data quality on an enterprise scale was paramount if we were really to harness it as an asset. People say data is an asset, but they create it to be an enterprise liability.
The key to “can” is through the data and how it is defined. Master Data Management (MDM) is beginning to become noticed, and I was fortunate to find a mentor who had developed enterprise masters in consistent structures, and a SOA platform with independent rules 20+ years ago. Once you can change the data mess, define stable structures, the software follows. Operational systems are smarter and aligned. Enterprise BI becomes simple and real. The IT world is smaller, simpler and the quality and increase in capability to the business is a major benefit.
If we were to make MDM an infrastructure reality, this would be a game changer. How to do this? How can Data Governance reduce complexity and contribute to Enterprise Agility? See the following link and download the Roadmap – 4 pages
From Data to System:
The transaction pattern above is right on. I use the “Demand – Response” pattern, same idea. I take this to the Enterprise level, and instead of thinking of the enterprise as a “system”, I see it as a collection of Demand – Response systems. However, the Enterprise is a single entity not a single system. Just like a person is a single entity with multiple interoperable systems.
Does this add complexity? The reality is “no” because the systems architecture is persistent, stable, and patterned. We see a holistic view of the enterprise, we see how the systems are related, what is different and the same. A pattern of how they are related and a pattern of purpose emerges. This enables us to see alignment or to architect alignment of the systems. Alignment contributes to both efficiency and agility.
The big contribution to agility is that once you have this complete pattern working, each new system, each new requirement is easily and reliably implemented, and with minimal time and cost. Approx 1-5% is my experience. Equally (if not more) important; the change result is optimal for the enterprise.
I have published a paper with the Enterprise Functional Framework based on this pattern in a paper at the following link. The pattern of work system relationships is very powerful for “holistic enterprise” people (all five of us).
Click on “Intelligent Systems Engineering”.
In this paper, there is also a “holistic” (of course) model of a work system. It is much more complex than a process model, but it is less complex for Enterprise Systems because it covers everything that in reality has to be there. Completeness is a foundational strength that enables agility because gaps cause problems. See the Meta-Model at the end of the document.
Johan, your article is fantastic. I am delighted to post. Main point: The key to Enterprise agility is not in software, it is in fixing the data mess, harnessing standard patterns of work systems, and then adding software. Begin with focus on Enterprise Meta Architecture. Join me in the Meta World :)Comments welcome.
James Pennington () (URL) - 02 07 11 - 02:20
Thanks for sharing your practical experiences! I will take a look at your papers, thanks for sharing them.
Johan den Haan () (URL) - 04 07 11 - 21:17
The first and foremost ‘thing’ we need for contemporary agile operation is… infrastructurally organised information. From this truly firm base the simplest applications can be quickly built, used and thrown away.
Why do we always skip this essential first step? Why do we rush into the IT part and complex and bla bla bla systems?
Infrastructurally organised information…
Jan () (URL) - 09 08 11 - 22:17
Thanks for your help in discussing about this. Great review!
Software Development () (URL) - 10 08 11 - 10:04
Be nice. Keep it clean. Stay on topic. No spam.