Indigo is the next generation application connectivity and services from Microsoft, superseding the variety of .NET connectivity solutions available today: ASMX Web services, Remoting, and Enterprise Services.

Since .NET debuted some five years ago, all three technologies have been inundated in either hype or misconceptions. With Indigo around the corner, it is time to take a long hard look at these three technologies, and separate fact from myth so that you will be best prepared for Indigo. This article starts by examining the existing technologies, describing their merits and shortcomings, putting them in the correct perspective of a modern distributed application, and suggests where to best apply them. Then the article briefly describes the Indigo programming model, and assesses how to best mitigate the cost of the migration.

Crossing Boundaries with Web Services

Web services are a generic name used to describe method invocation using a text-based protocol over HTTP. Web services are not synonymous with SOAP - most non-binary, text-based protocols such as HTTP GET or HTTP POST could qualify as Web services. Web services are not specific to the Web and are often used by intranet-based applications. In fact, today, most deployed Web services actually are used in the context of intranets. The architecture of Web services is simplicity itself - a Web server such as IIS receives a request over HTTP. It is then up to the Web server to interpret that request. In .NET, a request for an ASMX page causes IIS to load a corresponding Web application in a worker process and let a set of .NET DLLs use a class associated with that ASMX page to handle the request. This architecture is shown in Figure 1. .NET does the conversions from the request payload (often in SOAP XML) to Web method calls. .NET is also responsible for converting returned values from managed types to standard text-based representations. Under .NET 2.0, the Web server need not even be IIS - any running process can easily become a HTTP requests handler.

Converting types to and from text representation has a performance penalty, but more important is the limitation on the expressiveness of the types you can expose on the Web service methods. In addition, using HTTP as a connectionless protocol imposes certain restrictions on the object model, as it is best-suited for single call objects. Maintaining state is usually confined to a session.

You can use Web services to cross all types of boundaries

Because Web services themselves are based on simple text-based protocols, they lack the ability to support high level business semantics. With respect to security, raw Web services have no support for propagating security call context, single sign-on authentication, credential management, impersonation, and so on. Web services do not flow transactions, and there is no way using raw Web services to comprise a few Web service calls into a single logical transaction. Web services lack concurrency management so they leave developers susceptible to distributed deadlock and hard to resolve reentrancy issues. To address these issues, you need to implement complex emerging standards such as WS-Security or WS-AT (Atomic Transactions). Since most of these emerging standards are still, well, emerging, anything concrete you do now is likely to be proprietary and couple the service provider to the service consumers. In addition, extending Web services is a notorious task, and while .NET makes it easier than other platforms, it is still not a job for the faint of heart, and well beyond the reach of most developers.

The reason organizations and developers are willing to put up with these shortcoming is interoperability. Since Web services cater to the lowest common denominator (text over HTTP), virtually every operating system, Web server, and development technology can expose or consume Web services. Imagine a .NET client that wants to interact with a J2EE component. Neither one of these technologies support natively calling the other, yet both support Web services, so you can use Web services to interoperate between the two. C++ components on Windows can use Web services to interact with C++ components on Linux. In general, any kind of technology boundary, from operating system to development language can be bridged using Web services.

The presence of a firewall poses another kind of a boundary; a trust boundary in this case. Fortunately, since all that is exchanged between the Web service provider and the Web service consumer is text, most firewalls will allow Web services calls by default. In general, you can use Web services to cross all types of boundaries: technological, trust and security, deployment topology, organizational, geographical, and so on.

Extensibility with .NET Remoting

.NET remoting is a .NET-specific technology used to cross app domain boundaries. .NET remoting requires both the client and the object to be running in .NET, and the server object must derive from a class called MarshalByRefObject. The .NET remoting architecture is a modular and extensible architecture. The basic building blocks on the client side are proxies, formatters, and transport channels, and on the host side, the building blocks are transport channels, formatters, and call dispatchers. These building blocks are shown in Figure 2. The client interacts with a proxy which .NET creates on the fly based on the object's metadata. The proxy converts the method call to a message and passes it through a set of message sinks. The default configuration uses two sinks - one to serialize the message (the Formatter), and the other to transport it (the Channel). On the server side, there are two message sinks that accept the message, deserialize it, build the stack on the object, and make the call. When the call returns the process is reversed. Out of the box, .NET 2.0 supports two message formats - binary and SOAP, and three channels?TCP, HTTP, and IPC. For best performance across machine boundaries you should use binary format over TCP. However, if a firewall is present, you can use SOAP over HTTP to go across the firewall. In the scope of a single machine you can use IPC to make inter-process communication calls. IPC is based on named pipes and will refuse any calls from outside the machine. .NET remoting usually requires the developers to provide their own hosting process for the remote object. Remoting supports several object activation models that cater to most (but not all) needs of scalability and throughput. In addition, there are complex issues involving managing the lifetime of the remote object using leasing and sponsorship, and most types of callbacks and events are disabled by default, though you can unlock them. Out of the box, the only high level services offered by remoting in .NET 2.0 is call security - you can propagate the security call context, authenticate and authorize callers, and even sign and encrypt the messages. Similar to Web services, remoting by default does not offer any other services such as transaction propagation, host life time management, concurrency management, discontented calls, event broadcasting, etc. Remoting's saving grace is its architecture. Almost every point in the architecture in extensible, and you can inject your own message processing sinks both on the client and on the server side (Figure 3). These message processing sinks can perform pre- and post-call services, such as custom security, logging and tracing, concurrency management, caching, transaction management, and many more. A specific type of remote objects called context-bound objects is specifically designed for extensibility this way, and it offers a concurrency management service out of the box.

Productivity and Power with .NET Enterprise Services

.NET Enterprise Services is a set of powerful services designed to considerably ease the task of developing robust, scalable, high-quality applications, in a fraction of the time it takes to develop such applications without these services. Enterprise Services requires both the client and the server to be .NET components. You configure your methods using attributes that specify the functionality required by your component. To use almost all of the services offered by Enterprise Services, the server must derive from the ServicedComponent class. The Enterprise Services architecture is interception-based, meaning, Enterprise Services intercept the calls to the object and perform pre- and post-calls processing (Figure 4). The client of an Enterprise Services component interacts with a proxy. When the client makes a call on the proxy, the proxy makes a highly optimized call to unmanaged code to a set of chained interceptors asking them to do pre-call processing. The chain is comprised of interceptors according to the configured services. The first interceptor could verify the caller is authorized to call the object, the second interceptor could acquire a synchronization lock, the third could begin a transaction, the fourth could retrieve an object from a pool, and so on. When the pre-call processing is done, control returns to the proxy that makes the call on the object. The method call itself never leaves managed code, so there is no interop penalty. Contrary to common knowledge, Enterprise Services usually does not use interop unless it will yield significant performance gain over managed code. In addition, for remote calls, Enterprise Services uses DCOM under the hood, but again, the method call parameters are never marshaled to and from unmanaged code so there is no marshaling penalty. Consequently, Enterprise Services is actually the most performant option at the disposal of developers today, out performing both remoting and Web services by at least one order of magnitude if not more.

What is truly impressive about Enterprise Services is the set of services it offers: host startup and idle time management, instance management and scalability, transaction management, concurrency management, granular security and full security call context propagation, loosely coupled events, asynchronous disconnected calls, administrative tools for services management and application activation, and more. Enterprise Services can take care of all these aspects of the application, and let developers focus on implementing business logic.

.NET remoting architecture is a modular and extensible architecture.

The downside of Enterprise Services is that it cannot cross boundaries, and it is a closed architecture - you cannot add your own custom services to the interception chain. In addition, Enterprise Services requires a fair amount of time and effort to master the details of how the services are set up and combined, making them an expert tool, rather than something every developer can easily use.

Contrasting Options

Without considering Indigo, if you have to choose a technology today, the best way to do that is by elimination, as shown in Figure 5. If you need to interoperate across boundaries, then Web services are pretty much your only option. If you do not need to cross technology boundaries, and you can assume that both the client and the server are developed in .NET, you need to ask yourself whether you need extensibility. If the answer is yes, then .NET remoting is for you. You still need to decide which format and channel to use. If a firewall is present, then SOAP format over HTTP is what you use. For any other cross-machine calls choose binary over TCP for best performance, and if all the calls are made across processes but on the same machine, then use IPC. If you do not need extensibility, then bite the bullet and use Enterprise Services. You gain not only productivity and faster time to market, but also quality because Microsoft has done an excellent job in implementing these services, both in robustness and in performance. You can also adopt a hybrid approach. When implementing a service that needs to be accessed across boundaries, allow clients to access the service over Web services; while inside the service, use Enterprise Services.

Introducing Indigo

Indigo is the code name for the next generation application "plumbing" technology from Microsoft. Indigo will be released in the form of an SDK separately from the major .NET release cycles. Indigo requires .NET 2.0 and an underlying Windows XP family machine. No doubt you have heard the hottest buzzword in software engineering today - SOA or Service Oriented Architecture. Indigo is intended to ease the task of developing SOA applications. In a nutshell, SOA is merely a natural evolution of existing analysis and design methodologies, in particular of component-oriented programming. With SOA, you strive to minimize the coupling between the client and the server by using interface-based programming (a core principle of component-oriented programming), and avoid sharing any type metadata between the client and the server (like Web services). In SOA you allow the service provider and service consumer to interact across boundaries. SOA requires a high degree of abstraction of the underlying technology, and uniform use of interaction standards. It is impractical to try and develop the required SOA connectivity on your own. This is exactly what Indigo provides - ready-made, out-of-the-box connectivity and plumbing that you can use to build SOA applications.

Allow clients to access the service over Web services, while inside the service use Enterprise Services.

It is very important to realize that while you can use Indigo to build applications that address business models that are very difficult and costly to implement using other methodologies, you can and should use Indigo to address existing business needs. What this means is that you can use Indigo instead of Web services, remoting, or enterprise services, even if your application is not fully SOA. Doing so will ease the transition into SOA later on, when the business requirements will mandate such a degree of interaction, agility, and decoupling.

The Indigo architects were well-aware of the devil's alternatives that today's technologies pose to developers, forcing them to choose between interoperability, extensibility, and productivity. Indigo actually unifies the advantages of the three distinct technologies that preceded it: Indigo offers the same interoperability and boundaries-crossing ability of Web services. Indigo is based on a set of industry standards for Web services that are completely agnostic of development language, implementation technology, platforms, and vendor.

Indigo offers interoperability, extensibility, and productivity - the best of all worlds.

Indigo utilizes an extensible architecture, which like .NET remoting, uses messages to transport the client request to the service. The message processing on the client and service side is extensible, and you can add your own hooks and custom behavior when required. In fact, many parts of Indigo itself are implemented using Indigo's own extensibility model. Indigo offers a powerful set of services that significantly ease the task of developing modern applications. Like Enterprise Services, Indigo supports propagating distributed transactions, granular security and flow of security call context, concurrency management, asynchronous disconnected calls, reliable messaging, a publish-subscribe event broadcasting model, smart instance management techniques, and even service hosting and administration tools. In short, Indigo offers interoperability, extensibility, and productivity - the best of all worlds.

Indigo Migration

With such benefits and advantages, you should strive to move to Indigo sooner rather than later, or at the very least, prepare your application for Indigo as best you can today to ease the transition and mitigate the cost. Whenever you adopt a new technology, there are two elements to the cost - syntactical changes and semantic changes. Syntactical changes are usually easy, trivial, and low in cost and effort. For example, imagine converting a C# code base to VB .NET or a C code base to Pascal. Syntactical changes are simply the act of translation or the mapping of similar concepts and constructs from one technology to another.

What is truly impressing about Enterprise Services is the set of services it offers.

Semantic changes, on the other hand, are usually very difficult and can be horrendously expensive. For example, consider the cost involved in changing an event-driven application to an application that is not event-driven. Even if you maintain the implementation language such as C#, the cost of such a change is far from trivial. For another example, consider changing a single-threaded application to a multi-threaded application, or a transacted application to non-transacted. The reason semantic changes are so expensive is because they are actually changes to the application programming model. Changes to the programming model amount to design changes after the application is already developed. Any veteran developer can attest that such changes are probably as expensive as changes can get, often costing hundreds more in time and effort than what it would have taken to implement the design differently in the first place.

Migrating to Indigo is no different, and you will have to pay for both syntactical and semantic changes. To minimize the cost, you should choose a programming model that is most compatible with the Indigo programming model, because the semantic changes are where the cost will be.

As it turns out, the Indigo programming model resembles the Enterprise Services model. Both technologies use attributes to declaratively apply connectivity services and the component or service code itself can be free of plumbing, and dedicated to the business logic at hand.

For example, consider a transactional component. Without Enterprise Services, you are responsible for enlisting the resources, propagating the transaction to other components and resources, deciding on the outcome of the transaction, and instructing the resources to commit or abort the changes performed. This pollutes the code with the transaction management and coordination logic. When using Enterprise Services to manage the transaction, here is the required code:

using System.EnterpriseServices

public interface IMyService
{
   void MyMethod();
}

[Transaction]
public MyService : ServicedComponent,IMyService
{
   [AutoComplete]
   void MyMethod()
   {...}
}

The code inside the MyMethod() is dedicated to the business logic implementation, without any shred of transaction management.

Here is the equivalent code using Indigo (pre-Beta 1):

[ServiceContract]
interface IMyService
{
 [ServiceOperation(TransactionFlowAllowed = true)]
 [ServiceOperationSettings(
                       TransactionRequired = true,
                       AutoComplete = true)]
   void MyMethod();
}
class MyService : IMyService
{
   void MyMethod()
   {..}
}

Due to NDA restrictions, the Indigo code snippet is only pseudo code, yet the actual namespaces, attributes, properties names, and values are irrelevant - the important thing is the programming model - the code inside the method is free of any transaction-management plumbing. Now consider the two pre-Indigo implementations and what it would take to port them to Indigo. If you were using Enterprise Services, the changes would be syntactical-change the Enterprise Services namespace and attributes to the Indigo ones, removed the derivation from ServicedComponent, and you are done. If you were managing the transaction yourself, you are now facing the delicate task of surgically removing that code from the business logic code, then applying the Indigo services on top. That is, of course, an expensive semantic change.

The Indigo programming model resembles that of Enterprise Services the most

Almost all of the Indigo services can be mapped in a similar manner to Enterprise Services: from security, to disconnected asynchronous calls, to loosely-coupled events, object activation, hosting, and so on. Table 1 summarizes the various plumbing aspects you will encounter when migrating an existing application to Indigo and the expected changes, with and without Enterprise Services.

The conclusion is clear - when anticipating Indigo, if you want to minimize changes and the cost of changes to your code today, you should use .NET Enterprise Services inside your service, and wrap the service with a Web service if you require boundary crossing.