Explaining the framework: why these rows and columns?

A while ago I wrote a lengthy article that introduced the above framework and explained it step by step. Since the introduction I did some small updates. This page contains the most recent version.

Why do we have 7 rows/layers? There basically is a continuum from hardware to SaaS. Based on my knowledge of the market and technology I created these 7 layers, that I think help to categorize vendors and services in a meaningful way.

Why do we have 3 main columns: compute, communicate, and store? It is a well-known distinction used on the infrastructure layer. The nice thing is that it also works well on the software layers.

If we look at software, and especially the object-oriented paradigm, we can make a distinction between the 3 main aspects of software: state, behavior, and messages (or data, methods, and messages, if you will). This beautifully aligns with the previous distinction between storage (vs. state), compute (vs. behavior), and networking (vs. messages).

The same distinction can be made if you look at the different types of middleware. There is middleware that focuses on facilitating compute, like application servers, process and rule engines, etc. There is also middleware that is mainly focused on facilitating communication, like enterprise service buses (ESBs), messages queues, etc. And finally we have all kinds of middleware that facilitate the storage of data, like database management systems in all their flavors (ranging from relational to document- and key/value-oriented storage).

Even for us humans we can distinguish these 3 aspects of operating with data: we store data (e.g. memorizing it, writing it down), we process data (e.g. combine, compute), and we communicate data (e.g. speak, write). That’s why I use three columns in my layered framework: compute, communicate, and store. In my opinion they help a lot in clarifying the technology stack on each layer.

Back to framework

Explaining the framework: a detailed explanation of each cell

The remainder of this page contains a detailed explanation of each cell of the framework. You can easily navigate to the explanation by clicking a cell in the framework above.

Hardware: Servers – Switches, Routers – Storage

The physical hardware layer. The servers provide compute capacity. The switches and routers provide communication among servers and other devices. The storage is often provided by dedicated storage solutions.

Back to framework

Software-Defined Datacenter: Virtual Machines

The main hardware elements of a datacenter can be virtualized: servers, networks, and storage. In a Software Defined Datacenter these elements are all provisioned, configured, and managed by a central software solution.

Server virtualization is the most mature part of a Software Defined Datacenter. Virtualization is not just abstracting the logical view of the servers from the physical hardware that implements them. It is also possible to pool resources of many different servers, thereby creating a virtual layer that offers automatic load balancing and higher levels of availability than the underlying hardware can offer. In principle, commodity hardware can be used as the more advanced management and reliability is taken care of by the virtualization layer. When people talk about “cloud” or “IaaS” they often refer to server virtualization. The best-known examples in the market today probably are the Amazon Elastic Compute Cloud (EC2) and OpenStack Nova. But basically any big IT player has an offering, like Microsoft Windows Azure, Google Compute Engine, and IBM SoftLayer, to name a few.

Back to framework

Software-Defined Datacenter: Software-Defined Networking (SDN)

Network virtualization decouples and isolates virtual networks from the underlying network hardware, in the same way as server virtualization abstracts the underlying physical hardware. The network hardware is only used for the physical link; virtual networks can be created programmatically. These virtual networks offer the same features and guarantees of a physical network, but with the benefits of virtualization like higher utilization of the hardware, more flexibility, and easier management. Network virtualization is often referred to as Software Defined Networking (SDN). The best-known example of SDN probably is Nicira, not in the least because of their $1 billion acquisition by VMWare. OpenStack Neutron (formerly known as Quantum) is also in this space.

There is also a lot of talking about Network Function Virtualization (NFV), which is in this framework cell as well. NFV and SDN are not the same thing, but it is a nuanced distinction.

Back to framework

Software-Defined Datacenter: Software-Defined Storage (SDS), Block Storage

It shouldn’t be a surprise that storage can be virtualized too, and that it is referred to as Software Defined Storage (SDS). Storage virtualization means decoupling storage volumes from the underlying physical hardware. This means that more advanced features like caching, snapshotting, high availability, etc. are managed in the software layer (and thus independent from the specific hardware brand). It also means that a lot of optimizations can be done to organize IO in such ways that performance degradation due to server virtualization can be minimized. An example of a storage hypervisor is Virsto (again, acquired by VMWare). Another (different) example is the OpenStack Cinder project that provides block storage and integrates with a range of storage vendors.

Back to framework

Foundational PaaS: Application Containers

We move one layer up to the “Foundational PaaS” layer. I tend to call it this way to distinguish it from the higher-level PaaS approaches in the layer above this one. The Foundational PaaS layer provides an additional layer on top of (virtual) infrastructure that provides a higher level of abstraction and automation. The focus switches from infrastructure-centric to application-centric. Let’s see what that means for each of the three columns: compute, communicate, and store. In short: a place to run applications, communication to and from these places, and a way to store objects, all with additional features like high-availability and horizontal scalability.

For compute this means that we do not talk about virtual machines anymore, but application containers. An application container is an isolated, self-described, infrastructure-agnostic container for running apps. Usually running an app means compiling the code and bundling it with the necessary runtime, like Java code and a Java Virtual Machine (optionally with additional middleware like Tomcat and/or Spring). The resulting package can be deployed in the app container. Docker is the defacto standard in this area nowadays. Other examples are what Heroku calls a Dyno and Cloud Foundry Warden.

Back to framework

Foundational PaaS: Routing, Messaging, Orchestration

The communicate part of the Foundational PaaS layer focuses on routing messages among app containers, storage systems, and external systems. This includes routing, queuing, and scheduling system instructions for e.g. app lifecycle management (e.g. the messaging / NATS component of Cloud Foundry). Especially the container orchestration part is a fast changing landscape at the moment with Docker Machine, Swarm, and Compose, Mesos Marathon, Spotify Helios, and Kubernetes by Google.

This framework cell also includes the smart routing and load-balancing of incoming requests to application instances (e.g. the Heroku routing layer or the Cloud Foundry router).

Back to framework

Foundational PaaS: Object Storage

Lastly, the store part of the Foundational PaaS Layer isn’t about block storage anymore but provides object storage. This means you don’t have to bother about mounting, partitioning, and formatting virtual disks, you can just store and retrieve objects via an HTTP API. Examples of object storage are Amazon S3, OpenStack Swift, and Ceph Storage.

Back to framework

PaaS: Application Platform-as-a-Service (aPaaS)

While the Foundational PaaS layer (layer 2) is focused on application infrastructure, the PaaS layer (layer 3) shifts the focus to code. In other words: layer 2 has binaries as input, layer 3 has code as input. With “code” I mean a bunch of text that describes an application in a certain programming language, integration configurations, or database configurations. Let’s have a more in-depth look at these elements while we discuss the three columns of the framework again.

For compute we do have language runtimes on top of the application container. This means you can deploy e.g. Java code to such an environment, the Java Virtual Machine (JVM) and other runtime packages are already available as a service. You could see this as an application server as a service. This is mostly referred to as aPaaS (application Platform-as-a-Service). Examples are Google App Engine (which allows you to deploy Java, Python, Go, and PHP) and Heroku Buildpacks. The buildpack mechanism basically works on top of a layer 2 application container but automatically finds the right runtime if you upload code and compiles it into a binary that can be deployed into an application container. Cloud Foundry borrows the same mechanism from Heroku.

Back to framework

PaaS: Integration Platform-as-a-Service (iPaaS)

In the communicate column we see what is called iPaaS (integration Platform-as-a-Service). An iPaaS provides the communication among applications whether it is in the cloud or on-premise. This communication is defined using integration flows. An iPaaS can be seen as an ESB (Enterprise Service Bus) in the cloud and contains features like routing, transformations, queuing, and in most cases a range of pre-defined adapters. Examples of iPaaS are MuleSoft CloudHub, TIBCO Cloud Bus, WS02 StratosLive and Windows Azure BizTalk Services.

However, there is also whole range of lighter-weight services that live in this category. Examples being Amazon SQS, CloudAMQP (RabbitMQ as a Service), and IronMQ.

Back to framework

PaaS: Database Platform-as-a-Service (dbPaaS)

We also see a higher level of abstraction in the store column. It probably isn’t a surprise that we talk about dbPaaS (database Platform-as-a-Service) here. dbPaaS basically is a database delivered as a service (in the cloud), including things like availability and scalability, in most cases in a multi-tenant setup. These services are not limited to relational ones, but include key-value and column-oriented datastores. Amazon provides a range of database services: SimpleDB, DynamoDB, Relational Database Service, and RedShift. Other examples are Heroku Postgres, Windows Azure SQL Database, and Salesforce Database.com.

Back to framework

Model-Driven PaaS: Model-Driven aPaaS, bpmPaaS

The Model-Driven PaaS layer (layer 4) is focused on the same kind of artefacts as layer 3, but the specifications and configurations are at a higher abstraction level. Layer 4 targets “Rapid Developers”. I use this name to distinguish them from “Coders”, they are more business-oriented and the higer-level languages they use allow them to be more productive in specific domains. It might be easier to define these people with some examples: the users of business process tools, the users that configure business rules engines, people that configure reporting and analytic tools, people that define applications using high-level models, etc. Technologies and solutions in layer 4 are able to empower these Rapid Developers by focusing on two things: abstracting away from technical details and a focus on easy to learn languages.

The definition of this layer can look a bit artificial at first sight because the users of layer 3 and layer 4 deliver the same type of things: applications, integrations, and data definitions. However, you will see that it makes perfect sense to have a different layer because of the clear difference between solutions/technologies on this layer versus the solutions/technologies in layer 3. Additionally, you will see that this layer creates a nice transition between layer 3 and layer 5. Let’s make things concrete by looking at the columns of our framework again.

In the compute column we see language runtimes as a service, just as on layer 3. The difference is in the type of languages that are supported. These languages are of a higher abstraction level and are often domain-specific (we call these languages DSLs – Domain-Specific Languages). I would still categorize the solutions and technologies in this column as Application Platform-as-a-Service (aPaaS). It basically is a subset of aPaaS, the subset that is focused on Model-Driven Development (MDD). Examples are workflow or business process management (BPM) tools that are delivered as a service (some people call this bpmPaaS). In this category it will be interesting to keep an eye on Effektif, which focuses on letting “end-users” automate workflows. There are also platforms that cover all aspects of application development (data, UI, logic, integration, etc.) like the Mendix App Platform that enables rapid delivery of enterprise mobile and web applications (full disclosure: I work for Mendix).

Back to framework

Model-Driven PaaS: Model-Driven iPaaS

In the communicate column we see the same difference as in the compute column. We still talk about iPaaS, but the languages used are aimed at different people. They are DSLs for the integration domain. Examples going into this direction are the Mulesoft Anypoint Studio and the SnapLogic platform with its Designer. IFTTT is a great example of how a focus on simplicity can lead to an easy to use service while still being very powerful. I’m waiting for the first iPaaS to focus on creating an IFTTT like experience, combined with enterprise grade service bus features.

Back to framework

Model-Driven PaaS: Business Analytics Platform-as-a-Service (baPaaS), Data Analytics Platform-as-a-Service

The store column of layer 4, the Model-Driven PaaS layer, contains tools that focus on making data storage, data retrieval, and data analytics accessible for rapid developers (usually more business-oriented people that that use a DSL to be productive in a certain domain). Data has never been available is such huge amounts (that’s why people refer to data as big data in most cases nowadays) and it never has been more important to quickly act upon this data. These trends have pushed vendors to have a strong focus on enabling business people to work with their tools, in the cloud. Examples in this category are Tableau Online, GoodData, Domo, Birst, and Platfora.

Back to framework

App Services: App Services

In layer 5 we enter the area between developers and end-users. If we approach layer 5 from a developer perspective we can define it as a layer of out-of-the-box services that can be re-used and orchestrated to compose (part of) a new application. If we approach layer 5 from the end-user perspective we can define it as SaaS extensions or SaaS customizations. We then talk about end-user development or so-called citizen developers. So, to summarize it: layer 5 is about pre-built services that can be configured and composed to extend existing applications or build completely new ones.

Let’s look at some examples. The best-known example for building SaaS extensions or deep customizations that involve integrations is force.com (for customizing and extending Salesforce). The Mendix App Services approach is an example of building and re-using out-of-the-box services (via an App Store) that can be composed into (or used as a basis for) new enterprise applications. The last example I want to mention is MuleSoft, which basically turns any API out there into an app service that can easily be re-used. They have a big number of so-called Anypoint connectors available that give access to third-party services.

The examples mentioned above all provide tools and solutions that cover all three columns of the framework. They are platforms that can be used to develop or connect to any type of customization or service. Examples of app services that are specifically in this category are the Google Prediction API), IBM Watson Services, and Microsoft Azure Machine Learning Services.

Back to framework

App Services: Communication and Social Services

In this category we talk about communication and/or social services. Sometimes this category is referred to as Communications-as-a-Service (CaaS). Examples include Amazon Simple Email Service, the Twilio Platform, Plivo, and Ameche from Voxeolabs.

Back to framework

App Services: Data-as-a-Service (DaaS)

The last App Services category is “Data-as-a-Service”. Basically any API that provides access to data is in this category. Examples are the Dun & Bradstreet data services, the Google Books API, Crunchbase, and NASA’s Data Services.

Back to framework

SaaS: Applications

We reached layer 6, which finally is about the applications themselves. We do not talk about the columns of our framework anymore on this layer. These applications cover all three columns, as they all need to compute, communicate, and store data.

We call the applications on this layer Software-as-a-Service (SaaS). SaaS is as-a-Service from the user perspective. From the developer/provider perspective SaaS can be built on top of the previously mentioned layers, but if an application for example isn’t built using a PaaS and runs on dedicated hardware, it can still be SaaS. This basically holds for each layer in the framework: it can be, but is not necessarily build on top of the layers below. It is all a matter of choice for the provider what part of their stack they get from another provider as a service and what part of the stack they build themselves.

Back to framework

Platforms plotted on the Cloud Framework