Yesterday Microsoft released BizTalk Services CTP to the public. This one little CTP changes everything. Honest.
In the Beginning
It used to be very hard to build a distributed application. The pioneers in this field were Napster, Seti@Home and ICQ. To make applications like this work, you needed to have clients identify themselves and a message relay on the back end. Think of a switchboard for your applications telling user 1 how to communicate to user 2. If you wanted to build these connected systems, it would require a lot of infrastructure-a tremendous amount if you your application became popular. This has always been the barrier to entry.
The Enterprise Service Bus
As time passed and web services came on to the scene, things got easier. A lot of the hard part of the pluming was taken care of; transport was easy via HTTP and speaking the same language was easy with SOAP. As time moved on and things got easier, people started pushing Web Services to the limit and the vendors started to really support building connected systems. After nearly a decade of XML and SOAP, most developers take it for granted. Enterprises now rely on this technology.
Enter the Enterprise Service Bus (ESB). ESB is an acronym (we always need a TLA, don’t we?) that is hot right now. The notion is that you have a set of shared services in an enterprise that act as a foundation for discovering, connecting and federating services. This is the natural evolution of the technology, as enterprises rely on the technology and themselves grow more federated, enterprises will standardize discovering, connecting and federating services.
Internet Service Bus
As Clemens argued yesterday, the release of the BizTalk Services CTP creates the first Internet Service Bus. Clemens says:
Two aspects that make the idea of a "service bus" generally very attractive are that the service bus enables identity federation and connectivity federation. This idea gets far more interesting and more broadly applicable when we remove the "Enterprise" constraint from ESB it and put "Internet" into its place, thus elevating it to an "Internet Services Bus", or ISB. If we look at the most popular Internet-dependent applications outside of the browser these days, like the many Instant Messaging apps, BitTorrent, Limewire, VoIP, Orb/Slingbox, Skype, Halo, Project Gotham Racing, and others, many of them depend on one or two key services must be provided for each of them: Identity Federation (or, in absence of that, a central identity service) and some sort of message relay in order to connect up two or more application instances that each sit behind firewalls - and at the very least some stable, shared rendezvous point or directory to seed P2P connections. The question "how does [MSN] Messenger work?" has, from an high-level architecture perspective a simple answer: The Messenger "switchboard" acts as a message relay.
Changing Business Models
In order to build distributed applications (and make money!) you have to scale to support the load your users and customers will add. This forced businesses to spend a disproportionate amount of money on hardware and too much time building the plumbing software. Let’s take my business Corzen for example. Corzen collects specific data from the internet via spidering (like Google.) Corzen then crunches the data: we de-dupe it, aggregate it, match it with Dun & Bradstreet and US Govt data, and then apply some statistical models. We then deliver this “crunched” data to customers on a weekly basis.
The value add is the “crunching” of the data. Where do you think 75% of my technology budget goes to? You guessed it, the spidering infrastructure. We spider 24X7 and collect between 4-10 million records each week from spidering. As you can imagine our customers want more and more sites to be spidered and more frequently. We spent a year building an amazing XML based, queue and batch based, distributed application with WCF. We can load URLs into a queue and then go to a web page where we can send that job to several spider servers via WCF. Our spider servers are low cost Virtual Private Servers on ISPs around the world running a simple spidering engine that uses reflection to dynamically compile C# code instructions and apply RegEx patterns. When Corzen was young, we manually started these engines via RDP. As we scaled, we had to build this system.
How does an Internet Service Bus change the Corzen business model and cost structure? Every few weeks we get new requests by customers to add more sites to our list of sites and we have to add spidering capacity. Since spidering is so basic, I have always wanted to have our customers spider for us (or fan out the spidering to 3rd parties for a small fee). I can then offer our customers a discount on their monthly fees based on the amount of spiders they will run for us.
This is win win win, they get more data faster, and I lower my overhead and pass that savings on to the customer. This drastically changes my focus. I spend 50% of my time managing the spidering, worrying about capacity, and expanding the spidering infrastructure. Corzen’s cost structure changes as does the relationship with our customer-we become partners in the data acquisition “plumming” and Corzen can focus on the analytics-what the customers really want us to do. In other words we get out of the raw material business (spidering) and focus on the manufacturing of the product (the analytics.) Corzen was recently acquired by a company that does spidering as well, but not the analytics. This is an exciting move since we can join forces with the spidering capacity and Corzen can focus more on the analytics.
Today, the problem with having our customers do spidering (or paying you on your idle time to do some spidering for us) is simple. With over 100 customers, you try getting our WCF application to work on all of their servers through our firewall and their firewall. BizTalk services solves this problem. BizTalk services will provide a globally addressable name for Corzen’s service and securely expose that service to the Internet from behind a firewall or NAT as shown here.
As Dennis argues:
Use the Relay at http://connect.biztalk.net. We’ve shipped an SDK with a few samples showing you how you use the relay and identity services together. If you’re familiar with Windows Communication Foundation, you’ll find this trivial to use (by design!). Basically, your service opens at a URI on the connect.biztalk.net machines. Then a client connects to that URI and can start sending messages. We don’t want to be in the way of your app, so our relay will immediately try to establish a direct connection between clients. More details on this how this all works in a later post. Here’s a quick diagram that describes it at a high level.
Microsoft vs. Google
Everyone always likes to compare the “war” between Microsoft and Google. Maybe it is Google’s stock price or the popularity of its search engine or Gmail system. I look at the companies as light years apart. At its core, Google is an advertising company (highlighted by its recent acquisition of DoubleClick.) Sure Google will expose some APIs for developers, however, everything it does from Gmail to search is to gain more eyeballs for its ads.
Microsoft is a company about selling Windows. What has made Windows so popular is that Microsoft gives developers amazing tools to build applications around Windows, Office and the Internet. The strength of Microsoft is the developer community surrounding its products. You always hear about the next great thing that is going to “take Microsoft down.” The only thing that will take Microsoft down is a company with a compelling platform that also provides tools for developers to create applications on that platform.
The latest thing to come to take Microsoft down is Software as a Service. Think Google Spreadsheet. Businesses will all use the Spreadsheet in the sky and store its data on Google’s servers. Ditto with Gmail, why bother with Exchange?
Microsoft does have a cool differentiating factor: its hybrid approach. Microsoft is offering Software + Services. As I said in eWeek, with such a huge commitment to the OS and other installed software already, Microsoft is actually in a position to deliver software and a service on top of it.
The marketplace wants a hybrid solution and Microsoft is the only one who can deliver it in the short to medium term, giving Microsoft a competitive advantage. Everyone thought Google Docs would kill Office but in reality, Google docs are cool but Enterprises have issues today with using it offline and inside of a browser (copy and paste is strange, so is right clicking in a cell). Personally I use Google Spreadsheet to keep track of simple things but Microsoft Excel to do the more processor intensive operations. In addition, I work offline a tremendous amount as well don’t trust Larry and Sergey to store my very sensitive documents.
Think of Excel as software + services. Excel can be sold and run on your computer. You can store your documents locally or up in the cloud. Processor intensive operations can utilize your local super-fast Pentium 1million Processor and your 2 GB of RAM. Collaborative efforts can be handled by the cloud. Additional services like statistical number crunching or anything that needs to be distributed can be handled by the cloud. Multiple editors and viewers, in the cloud. You get the point.
This is where BizTalk Services come in. It is an early way for developers to deliver Software plus Services.
The Future of Business
As I said, this changes everything. We can all agree that distributed applications are the future. In order to make money you have to scale to support the load your users and customers will add. This forced businesses to spend a disproportionate amount of money and focus on technology (and pay CTOs way too much!)
We are so focused on technology that CEOs and Venture Capitalists are desperately trying to learn about the basics of technology-time they should be spending working through business models and looking for competitive advantages.
BizTalk services and ultimately all of the Software plus Services (including other vendors, not just Microsoft) will change the way we do business in 5-10 years. Imagine if we had to run a switchboard to run our phones in the office? So an oil company or a bank would have to develop the technical expertise to run the phones. This infrastructure is solved by the phone company (and now VoIP!) In the future, businesses will only have to focus on their core businesses and most software will run locally with services up in the cloud, drastically reducing the investment in core IT infrastructure internally. It’s a brave new world out there.