collective unconscious of the smarter planet

Google has opened the door to one of its data centers to Wired. Pretty amazing.

If you’re looking for the beating heart of the digital age — a physical location where the scope, grandeur, and geekiness of the kingdom of bits become manifest—you could do a lot worse than Lenoir, North Carolina. This rural city of 18,000 was once rife with furniture factories. Now it’s the home of a Google data center.

Engineering prowess famously catapulted the 14-year-old search giant into its place as one of the world’s most successful, influential, and frighteningly powerful companies. Its constantly refined search algorithm changed the way we all access and even think about information. Its equally complex ad-auction platform is a perpetual money-minting machine. But other, less well-known engineering and strategic breakthroughs are arguably just as crucial to Google’s success: its ability to build, organize, and operate a huge network of servers and fiber-optic cables with an efficiency and speed that rocks physics on its heels. Google has spread its infrastructure across a global archipelago of massive buildings—a dozen or so information palaces in locales as diverse as Council Bluffs, Iowa; St. Ghislain, Belgium; and soon Hong Kong and Singapore—where an unspecified but huge number of machines process and deliver the continuing chronicle of human experience.

This is what makes Google Google: its physical network, its thousands of fiber miles, and those many thousands of servers that, in aggregate, add up to the mother of all clouds. This multibillion-dollar infrastructure allows the company to index 20 billion web pages a day. To handle more than 3 billion daily search queries. To conduct millions of ad auctions in real time. To offer free email storage to 425 million Gmail users. To zip millions of YouTube videos to users every day. To deliver search results before the user has finished typing the query. In the near future, when Google releases the wearable computing platform called Glass, this infrastructure will power its visual search results.

The problem for would-be bards attempting to sing of these data centers has been that, because Google sees its network as the ultimate competitive advantage, only critical employees have been permitted even a peek inside, a prohibition that has most certainly included bards. Until now.

A breakdown of the latest IDC and Gartner server numbers for the quarter. Two things of note - IBM continues its dominance of the UNIX space, growing its worldwide marketshare by 5% in the quarter; and stripped down “cookie-sheet” servers, modeled on the bare-bones servers that Google uses in their datacenters, are becoming a sizable part of the server business.

The cookie-sheet servers created by Google for its own use – recently commercialized by all the top-tier vendors in one form or another as hybrid rack-blade boxes – have become a sizeable and important part of the server business.

In the third quarter ended in September, the box counters at IDC reckon that end users snapped up 2.07 million machines from server makers and their channel partners, an increase of 8.7 per cent compared to the year-ago period. Revenues increased a more modest 4.2 per cent, to $12.74bn. The market is cooling a bit thanks to tough compares, and as fellow box counter Gartner pointed out earlier this week, shipment and revenue levels on a global basis have more or less recovered to the levels prevailing ahead of the server crash in the wake of the Great Recession three years ago.

Blade servers, which have been around for a little more than a decade, are in essence racks in miniature with integrated backplanes for linking servers to integrated switching and system management processors in the chassis, seemed poised to become a dominant server architecture based on the hockey-stick uptake of rack servers during the dot-com boom, but blades are a high-end product that is a tougher sell than many server-makers had expected. Still, blades are the best option for many customers, and at $2bn in sales for the quarter (16 per cent of revenues) and just under 280,000 units (13.5 per cent of machines sold) they are an important system option even if they have not knocked out rack servers, as many expected.

Blade servers are full of system management and resiliency features that most supercomputer and hyperscale cloud operators simple won’t pay for. That’s why over the past several years the cookie sheet server – which jams multiple server nodes into a rack chassis on metal trays – has become popular. These nodes are all about low-cost and ripping out any extraneous stuff in a blade box – such as service processors, integrated node management, and switching. The assumption with hyperscale servers is that compute and local storage are all that matter and the application environment itself will provide the resilience. And hence these hyperscale servers, as IDC calls them, have come into their own.

In some cases, such as those of Google and Amazon, the company is building all or some of their own hyperscale machines, and even while Facebook has designed its own servers, it still farms out the manufacturing and thus those servers get counted as commercial boxes in IDC’s numbers.

In the third quarter, the hyperscale server segment accounted for 118,888 shipments and $428.5m in revenues, which is an increase of 8.7 per cent in terms of sales and 4.3 per cent in terms of shipments. So hyperscale machines now account for 3.4 per cent of revenue and 5.7 per cent of shipments and have much lower average selling prices per node.

How much lower? If you do the math across the whole server market in the third quarter, the average server cost $6,149. There were a few thousand mainframes and high-end RISC/Itanium boxes in there to raise the class average, but x86 machines account for the lion’s share of shipments in any quarter these days. (All but about 70,000 machines in this case.) The average rack or tower server cost $6,163, and the average blade server cost $7,151. You can see now why blades have had limited appeal: they offer operational cost advantages, but you pay a premium for the hardware. The average hyperscale server cost a mere $3.604 according to IDC’s data, which shows you why you might want to build resiliency in your software stack instead of on any particular server node.

Dicing and slicing server sales

In addition to casing server sales by form factor, IDC takes a stab at estimating the shipments of servers based on the primary operating system that gets installed on the boxes as well as by price band. The System zEnterprise 196 mainframes announced a little more than a year ago gave Big Blue a big bump in sales, but that refresh cycle is starting to slow; the company only booked $970m in sales of mainframes in the third quarter, according to IDC. Windows server sales also cooled a bit, with revenues up 5.3 per cent to $6.3bn against shipment increases of only 2 per cent. That said, Windows-based machines are by far the dominant server platform in the world in terms of shipments.

Unix machines showed some life, with sales up 1.6 per cent to $2.6bn, thanks mainly to IBM pushing its AIX boxes like crazy. “IBM is really starting to dominate this market,” Jed Scaramella, research manager of enterprise servers at IDC, tells El Reg, adding that IBM accounted for 46 per cent of all Unix revenues in the third quarter and gaining five points of revenue market share.

Hewlett-Packard lost 5 points of share and Oracle lost 1.5 points, and they were in a statistical tie for second place in Unix system sales in the quarter. Linux systems saw a very nice 12.3 per cent revenue jump in Q3, to $2.3bn and now account for 18.6 per cent of total server revenues. If you want to be generous, you could say that the combined Unix and Linux markets – call it Unilinux – experienced a 6.4 per cent revenue bump to $4.9bn.

Various proprietary platforms from Unisys, Fujitsu, Bull, NEC, IBM, and HP (not including IBM System z mainframes) accounted for a mere $570m in sales, down 8.8 per cent over last year’s third quarter.

Server sales were not uniform around the world, as you might expect given the differences in regional economies and the state of IT infrastructure in different regions.

“After nearly two years of steady revenue growth, the server market began to decelerate in Q3 2011 as demand stabilized for many system categories,” explained Matt Eastwood, general manager of enterprise platforms at IDC, in a statement accompanying the server stats. “Asia/Pacific and Japan exhibited strong revenue growth while server demand in EMEA, North America, and Latin America was flat to slightly down year over year. IDC continues to believe that weakening macroeconomic conditions around the world will serve to further moderate demand for new servers in 2012.”

By segment, IDC calculates that volume servers (machines that cost under $25,000) had a 5 per cent increase in revenues as a group in the quarter. The high-end segment, which covers machines that cost more than $250,000, had a 1.1 per cent revenue bump (despite the System z decline and because of improving Unix system sales), and the midrange machines between these two bookends had a 4.7 per cent revenue increase as a group.

If you look at the server business by vendor, IBM and Hewlett-Packard were in a dead heat in Q3 as far as IDC can tell, with IBM and HP both getting $3.79bn in sales. (Technically, IDC believes Big Blue had $3m more in sales, but declares a tie when the difference between the vendors is less than one per cent.) IBM grew 3.5 per cent and HP dropped 3.8 per cent.

Dell was the number three server seller, with $1.93bn in sales, and grew at a pace that was nearly three times as fast as the market at large. Another way of saying that is this: If you take Dell out of the numbers, the other vendors only grew their sales by 2.7 per cent, so Dell accounted for two-thirds of the growth of the overall market. Oracle’s server sales declined by 3.2 per cent to $764m, and Fujitsu shrank four-tenths of a per cent to $605m. Other vendors – helped by supercomputer-makers Silicon Graphics and Cray as well as upstarts Lenovo and Cisco Systems – grew as a group by 22 per cent to $1.86bn.

This “rant” from a Google employee has been getting a lot of attention on the interwebs over the past couple of days. And for good reason. It’s a very good, IT manager-/developer-eye view of how to architect services in huge, web-centric organizations like Google and Amazon. Very interesting.

So one day Jeff Bezos issued a mandate. He’s doing that all the time, of course, and people scramble like ants being pounded with a rubber mallet whenever it happens. But on one occasion — back around 2002 I think, plus or minus a year — he issued a mandate that was so out there, so huge and eye-bulgingly ponderous, that it made all of his other mandates look like unsolicited peer bonuses.

His Big Mandate went something along these lines:

1) All teams will henceforth expose their data and functionality through service interfaces.

2) Teams must communicate with each other through these interfaces.

3) There will be no other form of interprocess communication allowed: no direct linking, no direct reads of another team’s data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network.

4) It doesn’t matter what technology they use. HTTP, Corba, Pubsub, custom protocols — doesn’t matter. Bezos doesn’t care.

5) All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions.

6) Anyone who doesn’t do this will be fired.

7) Thank you; have a nice day!

Ha, ha! You 150-odd ex-Amazon folks here will of course realize immediately that #7 was a little joke I threw in, because Bezos most definitely does not give a shit about your day.

#6, however, was quite real, so people went to work. Bezos assigned a couple of Chief Bulldogs to oversee the effort and ensure forward progress, headed up by Uber-Chief Bear Bulldog Rick Dalzell. Rick is an ex-Armgy Ranger, West Point Academy graduate, ex-boxer, ex-Chief Torturer slash CIO at Wal*Mart, and is a big genial scary man who used the word “hardened interface” a lot. Rick was a walking, talking hardened interface himself, so needless to say, everyone made LOTS of forward progress and made sure Rick knew about it.

Over the next couple of years, Amazon transformed internally into a service-oriented architecture. They learned a tremendous amount while effecting this transformation. There was lots of existing documentation and lore about SOAs, but at Amazon’s vast scale it was about as useful as telling Indiana Jones to look both ways before crossing the street. Amazon’s dev staff made a lot of discoveries along the way. A teeny tiny sampling of these discoveries included:

- pager escalation gets way harder, because a ticket might bounce through 20 service calls before the real owner is identified. If each bounce goes through a team with a 15-minute response time, it can be hours before the right team finally finds out, unless you build a lot of scaffolding and metrics and reporting.

- every single one of your peer teams suddenly becomes a potential DOS attacker. Nobody can make any real forward progress until very serious quotas and throttling are put in place in every single service.

- monitoring and QA are the same thing. You’d never think so until you try doing a big SOA. But when your service says “oh yes, I’m fine”, it may well be the case that the only thing still functioning in the server is the little component that knows how to say “I’m fine, roger roger, over and out” in a cheery droid voice. In order to tell whether the service is actually responding, you have to make individual calls. The problem continues recursively until your monitoring is doing comprehensive semantics checking of your entire range of services and data, at which point it’s indistinguishable from automated QA. So they’re a continuum.

- if you have hundreds of services, and your code MUST communicate with other groups’ code via these services, then you won’t be able to find any of them without a service-discovery mechanism. And you can’t have that without a service registration mechanism, which itself is another service. So Amazon has a universal service registry where you can find out reflectively (programmatically) about every service, what its APIs are, and also whether it is currently up, and where.

- debugging problems with someone else’s code gets a LOT harder, and is basically impossible unless there is a universal standard way to run every service in a debuggable sandbox.

That’s just a very small sample. There are dozens, maybe hundreds of individual learnings like these that Amazon had to discover organically. There were a lot of wacky ones around externalizing services, but not as many as you might think. Organizing into services taught teams not to trust each other in most of the same ways they’re not supposed to trust external developers.

This effort was still underway when I left to join Google in mid-2005, but it was pretty far advanced. From the time Bezos issued his edict through the time I left, Amazon had transformed culturally into a company that thinks about everything in a services-first fashion. It is now fundamental to how they approach all designs, including internal designs for stuff that might never see the light of day externally.

At this point they don’t even do it out of fear of being fired. I mean, they’re still afraid of that; it’s pretty much part of daily life there, working for the Dread Pirate Bezos and all. But they do services because they’ve come to understand that it’s the Right Thing. There are without question pros and cons to the SOA approach, and some of the cons are pretty long. But overall it’s the right thing because SOA-driven design enables Platforms.

That’s what Bezos was up to with his edict, of course. He didn’t (and doesn’t) care even a tiny bit about the well-being of the teams, nor about what technologies they use, nor in fact any detail whatsoever about how they go about their business unless they happen to be screwing up. But Bezos realized long before the vast majority of Amazonians that Amazon needs to be a platform.

You wouldn’t really think that an online bookstore needs to be an extensible, programmable platform. Would you?

Well, the first big thing Bezos realized is that the infrastructure they’d built for selling and shipping books and sundry could be transformed an excellent repurposable computing platform. So now they have the Amazon Elastic Compute Cloud, and the Amazon Elastic MapReduce, and the Amazon Relational Database Service, and a whole passel’ o’ other services browsable at aws.amazon.com. These services host the backends for some pretty successful companies, reddit being my personal favorite of the bunch.

The other big realization he had was that he can’t always build the right thing. I think Larry Tesler might have struck some kind of chord in Bezos when he said his mom couldn’t use the goddamn website. It’s not even super clear whose mom he was talking about, and doesn’t really matter, because nobody’s mom can use the goddamn website. In fact I myself find the website disturbingly daunting, and I worked there for over half a decade. I’ve just learned to kinda defocus my eyes and concentrate on the million or so pixels near the center of the page above the fold.

I’m not really sure how Bezos came to this realization — the insight that he can’t build one product and have it be right for everyone. But it doesn’t matter, because he gets it. There’s actually a formal name for this phenomenon. It’s called Accessibility, and it’s the most important thing in the computing world.

The. Most. Important. Thing.

If you’re sorta thinking, “huh? You mean like, blind and deaf people Accessibility?” then you’re not alone, because I’ve come to understand that there are lots and LOTS of people just like you: people for whom this idea does not have the right Accessibility, so it hasn’t been able to get through to you yet. It’s not your fault for not understanding, any more than it would be your fault for being blind or deaf or motion-restricted or living with any other disability. When software — or idea-ware for that matter — fails to be accessible to anyone for any reason, it is the fault of the software or of the messaging of the idea. It is an Accessibility failure.

Like anything else big and important in life, Accessibility has an evil twin who, jilted by the unbalanced affection displayed by their parents in their youth, has grown into an equally powerful Arch-Nemesis (yes, there’s more than one nemesis to accessibility) named Security. And boy howdy are the two ever at odds.

But I’ll argue that Accessibility is actually more important than Security because dialing Accessibility to zero means you have no product at all, whereas dialing Security to zero can still get you a reasonably successful product such as the Playstation Network.

So yeah. In case you hadn’t noticed, I could actually write a book on this topic. A fat one, filled with amusing anecdotes about ants and rubber mallets at companies I’ve worked at. But I will never get this little rant published, and you’ll never get it read, unless I start to wrap up.

That one last thing that Google doesn’t do well is Platforms. We don’t understand platforms. We don’t “get” platforms. Some of you do, but you are the minority. This has become painfully clear to me over the past six years. I was kind of hoping that competitive pressure from Microsoft and Amazon and more recently Facebook would make us wake up collectively and start doing universal services. Not in some sort of ad-hoc, half-assed way, but in more or less the same way Amazon did it: all at once, for real, no cheating, and treating it as our top priority from now on.

But no. No, it’s like our tenth or eleventh priority. Or fifteenth, I don’t know. It’s pretty low. There are a few teams who treat the idea very seriously, but most teams either don’t think about it all, ever, or only a small percentage of them think about it in a very small way.

It’s a big stretch even to get most teams to offer a stubby service to get programmatic access to their data and computations. Most of them think they’re building products. And a stubby service is a pretty pathetic service. Go back and look at that partial list of learnings from Amazon, and tell me which ones Stubby gives you out of the box. As far as I’m concerned, it’s none of them. Stubby’s great, but it’s like parts when you need a car.

A product is useless without a platform, or more precisely and accurately, a platform-less product will always be replaced by an equivalent platform-ized product.

Google+ is a prime example of our complete failure to understand platforms from the very highest levels of executive leadership (hi Larry, Sergey, Eric, Vic, howdy howdy) down to the very lowest leaf workers (hey yo). We all don’t get it. The Golden Rule of platforms is that you Eat Your Own Dogfood. The Google+ platform is a pathetic afterthought. We had no API at all at launch, and last I checked, we had one measly API call. One of the team members marched in and told me about it when they launched, and I asked: “So is it the Stalker API?” She got all glum and said “Yeah.” I mean, I was joking, but no… the only API call we offer is to get someone’s stream. So I guess the joke was on me.

Microsoft has known about the Dogfood rule for at least twenty years. It’s been part of their culture for a whole generation now. You don’t eat People Food and give your developers Dog Food. Doing that is simply robbing your long-term platform value for short-term successes. Platforms are all about long-term thinking.

Google+ is a knee-jerk reaction, a study in short-term thinking, predicated on the incorrect notion that Facebook is successful because they built a great product. But that’s not why they are successful. Facebook is successful because they built an entire constellation of products by allowing other people to do the work. So Facebook is different for everyone. Some people spend all their time on Mafia Wars. Some spend all their time on Farmville. There are hundreds or maybe thousands of different high-quality time sinks available, so there’s something there for everyone.

Our Google+ team took a look at the aftermarket and said: “Gosh, it looks like we need some games. Let’s go contract someone to, um, write some games for us.” Do you begin to see how incredibly wrong that thinking is now? The problem is that we are trying to predict what people want and deliver it for them.

You can’t do that. Not really. Not reliably. There have been precious few people in the world, over the entire history of computing, who have been able to do it reliably. Steve Jobs was one of them. We don’t have a Steve Jobs here. I’m sorry, but we don’t.

Larry Tesler may have convinced Bezos that he was no Steve Jobs, but Bezos realized that he didn’t need to be a Steve Jobs in order to provide everyone with the right products: interfaces and workflows that they liked and felt at ease with. He just needed to enable third-party developers to do it, and it would happen automatically.

I apologize to those (many) of you for whom all this stuff I’m saying is incredibly obvious, because yeah. It’s incredibly frigging obvious. Except we’re not doing it. We don’t get Platforms, and we don’t get Accessibility. The two are basically the same thing, because platforms solve accessibility. A platform is accessibility.

So yeah, Microsoft gets it. And you know as well as I do how surprising that is, because they don’t “get” much of anything, really. But they understand platforms as a purely accidental outgrowth of having started life in the business of providing platforms. So they have thirty-plus years of learning in this space. And if you go to msdn.com, and spend some time browsing, and you’ve never seen it before, prepare to be amazed. Because it’s staggeringly huge. They have thousands, and thousands, and THOUSANDS of API calls. They have a HUGE platform. Too big in fact, because they can’t design for squat, but at least they’re doing it.

Amazon gets it. Amazon’s AWS (aws.amazon.com) is incredible. Just go look at it. Click around. It’s embarrassing. We don’t have any of that stuff.

Apple gets it, obviously. They’ve made some fundamentally non-open choices, particularly around their mobile platform. But they understand accessibility and they understand the power of third-party development and they eat their dogfood. And you know what? They make pretty good dogfood. Their APIs are a hell of a lot cleaner than Microsoft’s, and have been since time immemorial.

Facebook gets it. That’s what really worries me. That’s what got me off my lazy butt to write this thing. I hate blogging. I hate… plussing, or whatever it’s called when you do a massive rant in Google+ even though it’s a terrible venue for it but you do it anyway because in the end you really do want Google to be successful. And I do! I mean, Facebook wants me there, and it’d be pretty easy to just go. But Google is home, so I’m insisting that we have this little family intervention, uncomfortable as it might be.

After you’ve marveled at the platform offerings of Microsoft and Amazon, and Facebook I guess (I didn’t look because I didn’t want to get too depressed), head over to developers.google.com and browse a little. Pretty big difference, eh? It’s like what your fifth-grade nephew might mock up if he were doing an assignment to demonstrate what a big powerful platform company might be building if all they had, resource-wise, was one fifth grader.

Please don’t get me wrong here — I know for a fact that the dev-rel team has had to FIGHT to get even this much available externally. They’re kicking ass as far as I’m concerned, because they DO get platforms, and they are struggling heroically to try to create one in an environment that is at best platform-apathetic, and at worst often openly hostile to the idea.

I’m just frankly describing what developers.google.com looks like to an outsider. It looks childish. Where’s the Maps APIs in there for Christ’s sake? Some of the things in there are labs projects. And the APIs for everything I clicked were… they were paltry. They were obviously dog food. Not even good organic stuff. Compared to our internal APIs it’s all snouts and horse hooves.

And also don’t get me wrong about Google+. They’re far from the only offenders. This is a cultural thing. What we have going on internally is basically a war, with the underdog minority Platformers fighting a more or less losing battle against the Mighty Funded Confident Producters.

Any teams that have successfully internalized the notion that they should be externally programmable platforms from the ground up are underdogs — Maps and Docs come to mind, and I know GMail is making overtures in that direction. But it’s hard for them to get funding for it because it’s not part of our culture. Maestro’s funding is a feeble thing compared to the gargantuan Microsoft Office programming platform: it’s a fluffy rabbit versus a T-Rex. The Docs team knows they’ll never be competitive with Office until they can match its scripting facilities, but they’re not getting any resource love. I mean, I assume they’re not, given that Apps Script only works in Spreadsheet right now, and it doesn’t even have keyboard shortcuts as part of its API. That team looks pretty unloved to me.

Ironically enough, Wave was a great platform, may they rest in peace. But making something a platform is not going to make you an instant success. A platform needs a killer app. Facebook — that is, the stock service they offer with walls and friends and such — is the killer app for the Facebook Platform. And it is a very serious mistake to conclude that the Facebook App could have been anywhere near as successful without the Facebook Platform.

You know how people are always saying Google is arrogant? I’m a Googler, so I get as irritated as you do when people say that. We’re not arrogant, by and large. We’re, like, 99% Arrogance-Free. I did start this post — if you’ll reach back into distant memory — by describing Google as “doing everything right”. We do mean well, and for the most part when people say we’re arrogant it’s because we didn’t hire them, or they’re unhappy with our policies, or something along those lines. They’re inferring arrogance because it makes them feel better.

But when we take the stance that we know how to design the perfect product for everyone, and believe you me, I hear that a lot, then we’re being fools. You can attribute it to arrogance, or naivete, or whatever — it doesn’t matter in the end, because it’s foolishness. There IS no perfect product for everyone.

And so we wind up with a browser that doesn’t let you set the default font size. Talk about an affront to Accessibility. I mean, as I get older I’m actually going blind. For real. I’ve been nearsighted all my life, and once you hit 40 years old you stop being able to see things up close. So font selection becomes this life-or-death thing: it can lock you out of the product completely. But the Chrome team is flat-out arrogant here: they want to build a zero-configuration product, and they’re quite brazen about it, and Fuck You if you’re blind or deaf or whatever. Hit Ctrl-+ on every single page visit for the rest of your life.

It’s not just them. It’s everyone. The problem is that we’re a Product Company through and through. We built a successful product with broad appeal — our search, that is — and that wild success has biased us.

Amazon was a product company too, so it took an out-of-band force to make Bezos understand the need for a platform. That force was their evaporating margins; he was cornered and had to think of a way out. But all he had was a bunch of engineers and all these computers… if only they could be monetized somehow… you can see how he arrived at AWS, in hindsight.

Microsoft started out as a platform, so they’ve just had lots of practice at it.

Facebook, though: they worry me. I’m no expert, but I’m pretty sure they started off as a Product and they rode that success pretty far. So I’m not sure exactly how they made the transition to a platform. It was a relatively long time ago, since they had to be a platform before (now very old) things like Mafia Wars could come along.

Maybe they just looked at us and asked: “How can we beat Google? What are they missing?”

The problem we face is pretty huge, because it will take a dramatic cultural change in order for us to start catching up. We don’t do internal service-oriented platforms, and we just as equally don’t do external ones. This means that the “not getting it” is endemic across the company: the PMs don’t get it, the engineers don’t get it, the product teams don’t get it, nobody gets it. Even if individuals do, even if YOU do, it doesn’t matter one bit unless we’re treating it as an all-hands-on-deck emergency. We can’t keep launching products and pretending we’ll turn them into magical beautiful extensible platforms later. We’ve tried that and it’s not working.

The Golden Rule of Platforms, “Eat Your Own Dogfood”, can be rephrased as “Start with a Platform, and Then Use it for Everything.” You can’t just bolt it on later. Certainly not easily at any rate — ask anyone who worked on platformizing MS Office. Or anyone who worked on platformizing Amazon. If you delay it, it’ll be ten times as much work as just doing it correctly up front. You can’t cheat. You can’t have secret back doors for internal apps to get special priority access, not for ANY reason. You need to solve the hard problems up front.

I’m not saying it’s too late for us, but the longer we wait, the closer we get to being Too Late.

I honestly don’t know how to wrap this up. I’ve said pretty much everything I came here to say today. This post has been six years in the making. I’m sorry if I wasn’t gentle enough, or if I misrepresented some product or team or person, or if we’re actually doing LOTS of platform stuff and it just so happens that I and everyone I ever talk to has just never heard about it. I’m sorry.

But we’ve gotta start doing this right.

ReadWrite Enterprise provides a good analysis of HP’s acquisition of Autonomy and what it means for their future prospects. They argue that ultimately the Autonomy acquisition is an attempt to re-imagine the traditional underpinnings of content on the Web and in businesses, centered on structured databases, and to disrupt Google.

The rate at which data, or content, is being produced for the Web and being generated for businesses has outpaced the rate at which conventional databases are evolving to better manage it all. It’s a fact of life that we perceive on a gradual basis every day, but that we haven’t yet acknowledged to be as significant or dangerous a trend as it is: Data is getting slower. Networks are getting bigger as the cloud is getting broader, and data that was already difficult to manage is becoming impossible. Content management systems today continue to be based on the types of structured database systems about one or two steps more evolved than dBASE. We’ve known they would be insufficient for the task, but we’ve put off the problem of composing a new architecture.

It’s already too late for major IT companies to start that new architecture from square one; if a company has any hope of addressing this colossal, underappreciated problem, it will need to acquire the architectural project in progress. This is what Hewlett-Packard announced yesterday that it intends to do: acquire a software firm whose core product aims to supplant everything we know about databases, both the SQL kind and the Google kind. In its place would come a clustered approach whose goal is no less than to be the central repository for meaning in the world.

And in exchange for this, HP is willing to let go of the promise of Palm.

“Autonomy represents an opportunity for HP, for us to accelerate our vision to decisively and profitably lead a large and growing space, which is the enterprise information management space,” said Léo Apotheker, the company’s CEO, during yesterday’s quarterly conference call (our thanks to Seeking Alpha for the transcript). “It also brings HP higher value business solutions that will help customers manage the explosion of information. But as we execute this deal, this will position HP as a leader in the large and growing space. It will complement our existing technology portfolio and enterprise strategy. It will provide differentiated IP for services and extensive vertical capability in key industries.”

Apotheker’s bet is that managing the explosion of information will bring higher revenue for his company than addressing a market interest that largely centered around consumers. HP knew that it was acquiring a consumer technology company when it announced its intent to purchase Palm just 16 short months ago. At the time, its executive vice president for the Personal Systems Group, Todd Bradley, said in order to make the Palm purchase profitable, it would have to exploit the potential revenue channels for webOS for business customers…

That never happened. As recently as last Tuesday, consumer oriented publications were speculating on the outlook for webOS in embedded scenarios, such as automobiles, HDTVs and, yes, refrigerators, based on guidance provided by HP’s public relations handlers. Some were wondering why the question of these opportunities was only being brought up now.

The final analysis of HP’s progress with webOS, such as it was, will show that in this particular instance, scale - the thing both HP and Palm had said the acquisition was all about - worked against them. No one seemed to believe you could graft the same OS that powers a Pixi phone and a refrigerator to a heart monitor.

Which brings us back to Autonomy. As CEO Apotheker told analysts yesterday, HP intends to exploit the prospects for using Autonomy’s technology as a foundation for a content management system. For now, that CMS would be a project for what, on the surface, seems an unlikely department: the Imaging and Printing Group (IPG). Autonomy describes this technology - which it calls Intelligent Data Operating Layer (IDOL) - as nothing less than a replacement for, a complete substitute for, a revolutionary disruption of, Google.

An Autonomy white paper (PDF available here) explains its iManage Universal Search product, which is based on its IDOL platform, this way:

“Traditional search engines cannot understand the meaning of information. Unfortunately, this inability to understand information means that other documents that discuss the same idea (i.e., are relevant) but use different words, are often overlooked. Equally, documents with a meaning entirely different to that which the user searches for are frequently returned.

“Users expect search to be simple, fast, comprehensive and secure. Universal Search is built on the IDOL platform, the leader in Meaning Based Computing, which forms a conceptual and contextual understanding of any piece of data, including text, voice and video, regardless of data type or storage location, performing advanced operations on that information in real-time.”

Elsewhere in Autonomy’s literature is a monkey wrench it hurls directly at Google, with hopes of messing up its gears. Here, the company attacks the value of Google’s page ranking technology in the enterprise: “in many cases, the most popular information is also the most relevant. The importance or popularity of a Web page is approximated by counting the number of other pages that are linked to it, and by how frequently those pages are viewed by other users. This works quite well on the Internet but in the enterprise it is doomed to failure. Firstly, there are no native links between information in the enterprise. Secondly, if a user happens to be an expert, perhaps in the field of gallium arsenide laser diodes, there may be no one else interested in the subject, but it is still imperative that they find relevant information.”

This is what HP is buying: an opportunity to disrupt Google. If IDOL is every bit the next stage of database evolution that Autonomy makes it out to be, then HP (at least in its executives’ own minds) is not surrendering to Google at all, as some consumer publications this morning are suggesting. As HP perceives it, rather than cutting off Google’s left arm, it’s targeting the gut. Apotheker pointed out yesterday that Autonomy is in the midst of a transition of its product portfolio (which to date has targeted high-volume information generating verticals such as legal) to SaaS-based deployment. It’s about one-third of the way, he said, and HP is very much interested in seeing Autonomy go the other two-thirds.

In other words, Apotheker is trying to make the scale argument work the other way: He envisions a scheme where a high-volume data engine that addresses high-end clientele scales down to a more general level, by way of the cloud. Along the way, he would be happy to provide collateral damage to Oracle, his arch-nemesis and the standard bearer for all things SQL.

“We will be looking at identifying as many synergies as we can, as quickly as possible,” said the CEO. “The first obvious one is to give Autonomy access to our channels across the world, which will help them in a significant manner. And at the same time, we will also be working across our software teams and their software teams to actually create additional capabilities there.”

The end result of which could be a restoration of the IPG division, whose cash cow at one time was quite literally black ink, to a growth position that’s just as much engaged in services as every other division of the company. Even if HP stops making PCs (which apparently wouldn’t hurt Apotheker’s feelings all that much), it could be the standard bearer for a class of cloud-based technology that could break businesses’ dependence upon both structured databases like Oracle, MySQL, and SQL Server, and contextual databases like Google.

But for this new and improved scale argument to work, customers will have to believe that you can graft the same vision of an all-seeing repository of all written meaning, to the same Android phone that gives you the traffic report and lets you pummel birds against the sides of pigs. Scale has failed HP once already.

Amazon is making some headway with purpose-built clouds for government.

Amazon is upping the pressure on both Microsoft and Google in the battle to scoop up cash-strapped government customers into the cloud.

The online bookseller has moved to win over more government customers with the launch of AWS GovCloud, its cloud for US gov departments that crunch super-secret defence data.

AWS GovCloud is a region, Amazon’s sixth, which is designed to meet a tough set of US government rules known as the International Traffic in Arms Regulations (ITAR).

ITAR says controlled data must be stored in an environment where logical and physical access is limited to US persons or permanent residents. That data covers the import and export of a long list of goods and services, ranging from warheads and missile-control systems to aircraft carriers and tanks.

Without ITAR compliance, US departments could not legally have uploaded this data to Amazon’s existing service, given that compute and storage happens in three non-US regions.

AWS GovCloud is based on the US West Coast, Amazon said, while the service already meets the same US regulatory controls as the rest of its regions – including Federal Information Security Management Act (FISMA), PCI DSS Level 1, ISO 27001, and SAS 70.

Now the etailer says it is ready to talk to other governments about adopting GovCloud, potentially adopting similar regulatory restrictions.

Werner Vogels, Amazon chief technology officer, blogged: “We do not envision that over time GovCloud will address only the needs of the US government and contractors. We are certainly interested in understanding whether there are opportunities in other governments with respect to their specific regulatory requirements that could be solved by a specialized region.”

Government is a hot area for cloud providers. Amazon claims more than 100 federal, state and local government agencies already onboard its existing services.

Microsoft and Google, meanwhile, are running PR campaigns against each other, claiming government customers’ scalps to prove their cloud is winning against the others.

National and local government has proved a fertile hunting ground as the US’s economy has stalled and austerity becomes the watchword for the public sector.

Microsoft and Google are pushing hard to sign up agencies to their hosted email and docs. For the public-sector IT people involved, this has meant they get the prospect of brand-new systems without either the up-front purchase cost or the long-term maintenance costs.

Much of the Microsoft and Google fight has been over email and docs; at times, though, it has been point-scoring over precisely the kind of regulatory approval Amazon has on GovCloud. The idea has been to sow uncertainty about their rivals in customers’ minds or to reverse customers’ decisions.

In April, Microsoft accused Google of misleading the market about the FISMA certification of Google Apps for Government. Google shot back saying Microsoft’s cloud services for government wasn’t FISMA-certified. It is now suing the US Department of the Interior for its selection of Microsoft’s Business Productivity Online Suite-Federal – Microsoft’s rival to Google Apps for Government – saying the department had violated government procurement policies.

While Google and Microsoft tussle over details, both are behind on what they’d really like to be: the leading platform for hosting devs apps and data. That crown belongs to Amazon, which is now beginning to serve as a platform for other cloud businesses such as Ruby host Heroku.

Microsoft’s alternative to Amazon is Azure: the company claims a list of customers using the Azure compute fabric and storage layer to deliver apps, but it has yet to find Amazon-levels of traction.

Microsoft this week re-jigged the pricing and packaging on its smallest level of compute instance for the third time since last autumn’s launch.

Forrester looks into what Google’s acquisition of Motorolla means to IT professionals.

Google sent shock waves through the mobile world this morning as it announced a planned acquisition of Motorola Mobility for $12.5 billion in cash. The initial commentary has largely focused around Motorola’s patent portfolio, how this will affect the other Android manufacturers, and what Google will do with the rest of Moto’s hardware business which my colleague John McCarthy summed up nicely in his blog post.

So what kind of an impact does this have on infrastructure and operations (I&O) professionals? For the most part, not much of one.  I&O professionals are working to make their organizations platform-agnostic by deploying mobile device management (MDM) solutions. For them, Android is only one in an increasingly crowded space of platforms including iOS, Blackberry, and Windows 7 Mobile. 

Still, there is one interesting implication in this deal that I&O pros should take note of — Google gets 3LM. Back in February Motorola Mobility acquired 3LM, a startup including former Google employees who worked on Android, which specializes in enterprise security and management software. Rumors had already been flying that some of the 3LM functionality like storage encryption and anti-malware would be included in the next version of Android (Ice Cream Sandwich). With 3LM now a part of Google, firms might finally management and security capabilities I&O and security pros have been asking for in Android.

So while this deal might not directly affect I&O professionals in their day to day operations it does show an additional $12.5 billion commitment from Google to the Android platform and its future. Given the increasing amount of Android devices connecting to corporate networks every day, we’ve written reports stating that firms should include Android (among others operating systems) in their next generation mobile support strategies and work to adopt MDM solutions which take advantage of all of the available management and security APIs. Android still has plenty of room to grow within the enterprise and this acquisition could bring some of the critical pieces to help it along.

The big news today is Google’s purchase of Motorolla’s mobility unit.

Google has made its largest-ever acquisition, and biggest corporate gamble, by splashing out $12.5bn for Motorola’s phone division, Motorola Mobility. The deal puts Google into the hardware business in a serious way – and into direct competition with licensees of its Android operating system, who woke up this morning thinking they were Google’s business partners.

Google will pay $40 per share in cash – a premium of 63 per cent over the trading price – to grab itself a hardware manufacturer. The deal is subject to regulatory approval, which may not all be plain sailing, but Google hopes to wrap up the deal by next spring. It dwarfs Google’s previous largest acquisition, DoubleClick.

Google recently lost out in a patent auction for networking IP, one it didn’t appear to take seriously. Perhaps this explains why it was behaving so childishly. Or perhaps Google realised what a serious issue it is. After losing the Nortel auction, it went out and acquired $1bn worth of IBM patents. We shall find out which it is, eventually.

The deal represents a Houdini escape for Motorola, which has been bleeding red ink for much of the past four years. Motorola split the phone business off into an independently traded stock last year, and in its most recent quarter [transcript] reported earnings of $3.3bn, up 28 per cent year on year.

But what happens to Android? Buying Motorola might buy Google some IP, but it also brings with it a whole heap of new problems.

“Our vision for Android is unchanged and Google remains firmly committed to Android as an open platform and a vibrant open source community,” writes Android chief Andy Rubin in the canned statement. “We will continue to work with all of our valued Android partners to develop and distribute innovative Android-powered devices.”

But once Google has a preferred hardware partner that it owns outright, it is hard to see why its former partners – now rivals – would wish to continue with Android.

Expect to hear a splashing sound as dozens of OEMs dump their green plastic robots overboard. But where will they go?

The entire article is worth reading for it’s discussion of Google’s and Facebook’s differing philosophies on datacenter construction and policy, but the part that stood out for me was the anti-workload optimized approach to servers that both competitors follow.

Giant data centers—even energy-efficient ones—are, of course, nothing without the proper servers. Facebook will be populating its Oregon and North Carolina locations with custom-designed servers, just as Google has long done.

Facebook’s Amir Michael, manager of hardware design, explains that when the company decided to build its own facilities, “we had a clean slate,” which allowed him and his colleagues to optimize the designs of their centers and servers in tandem for maximum energy efficiency. The result was a server that “looks very bare bones. I call it a ‘vanity-free’ design just because I don’t like people to call it ugly,” says Michael. “It has no front bezels. It has no paint. It has no logos or stickers on it. It really has only what is required.”

Google also keeps server frills to a minimum. Like Facebook, it buys commodity-level computing hardware and just fixes the many pieces that break, instead of purchasing high-end systems that are less prone to failure but also much more expensive. Economics, if nothing else, drove engineers at both companies to similar conclusions here. Fit and finish might count if you’re buying one server or even a hundred, but not when you’re shopping for tens of thousands at a time. And striving for high reliability is a little pointless at this scale, where failure is not only an option, it’s a daily fact of life.

Facebook’s Michael explains that he helped design three basic types of servers for running the Facebook application. The top layer of hardware, connected most directly with Facebook’s many users, consists of outward-facing Web servers. They don’t require much disk space—just enough for the operating system (Linux), the basic Web-server software (which until recently was Apache), the code needed to assemble Facebook pages (written in PHP, a scripting language), some log files, and a few other bits and pieces. Those machines are connected to a deeper layer of servers stuffed with hard disks and flash-based solid-state drives, which provide persistent storage for the giant MySQL databases that hold Facebook users’ photos, videos, comments, and friend lists, among other things. In between are RAM-heavy servers that run a memcached system to provide fast access to the most frequently used content.

Alpha geeks will recognize that these pieces of software—Linux, Apache, PHP, MySQL, memcached—all hail from the open-source community. Facebook’s programmers have modified these and other open-source packages to suit their needs, but at the most basic level, they are doing exactly what countless Web developers have done: building their site on an open-source foundation.

Not so at Google. Programmers there have written most of their company’s impressive software from scratch—with the exception of the Linux running on its servers. Most prominent are the Google File System (or GFS, a large-scale distributed file system), Bigtable (a low-overhead database), and MapReduce (which provides a mechanism for carrying out various kinds of computations in a massively parallel fashion). What’s more, Google’s programmers have rewritten the company’s main search code more than once.

Speaking two years ago at the Second ACM International Conference on Web Search and Data Mining, Jeff Dean, a Google Fellow working in the company’s system infrastructure group, said that over the years his company has made seven significant revisions to the way it implements Web search. However, outsiders don’t realize that, because, as Dean explained, “you can replace the entire back end without anyone really noticing.”

How are we to interpret the difference between Google’s and Facebook’s engineering cultures with respect to the use of open-source code? Part of the answer may just be that Google, having started earlier, had no choice but to develop its own software, because open-source alternatives weren’t yet available. But Steve Lacy, who worked as a software engineer for Google from 2005 to 2010, thinks otherwise. In a recent blog post, he argues that Google just suffers from a bad case of not-invented-here syndrome. Many open-source packages “put Google infrastructure to shame when it comes to ease of use and product focus,” writes Lacy. “[Nevertheless, Google] engineers are discouraged from using these systems, to the point where they’re chastised for even thinking of using anything other than Bigtable/Spanner and GFS/Colossus for their products.”

IEEE has just published a special report on social networking that seems to be as much about the nascent conflict between Google and Facebook as it as about the broader social trend.

In the beginning was the personal computer.

Not long after, people started connecting them together on networks, culminating in the World Wide Web and the Web browser, which launched the first great era of the Web. Then came the search engine, which launched the second great era of the Web, the era of Google.

Now comes the third: the era of social networks. Facebook has jumped out to a commanding lead, but Google hasn’t really started fighting yet. So the stage is set for a battle of, well, biblical proportions. The wizards at the Googleplex in Mountain View, Calif., are working furiously on a social network to rival Facebook’s. Just a few miles away, in Palo Alto, Facebook is preparing for an initial public offering to give it the money it needs to take on Google’s Goliath. And last month, the clash got a bit ugly when it was revealed that Facebook had hired a public relations agency to slime Google’s social networking tactics. 

We are about to witness the next great conflict of the information age, a rich and complicated match on the scale of mainframes vs. micros, RISC vs. CISC, Windows vs. Unix. Like those battles, Google-Facebook will shape the industry’s landscape for years to come.

The Web has had a social dimension almost from the start. It just took a while for the right software to come along and make it compelling. “We’re now seeing a Web built around people, where their profiles and content are moving with them as they visit different websites,” notes Paul Adams, who made his mark as a user-experience researcher at Google before jumping recently to Facebook. Socializing is something that people used to do on the Web; gradually it is becoming the Web…

That’s important, because ads are what makes this cockeyed caravan go. Google and Facebook have the same basic model: Offer the services free and charge for advertising. And, as any adman will tell you, the more popular your service, the more money you can get for ad space. That’s why Google and Facebook are vying to be the de facto home for Web users.

Nearly all of Google’s and Facebook’s revenues come from advertising. Google posted US $29.3 billion in revenue in 2010. A recent report said that privately held Facebook generated about $2 billion in revenue last year, which places its size on a par with that of Google when Google was also just six years old.

What Google and Facebook have that old media don’t is information about you—data that they collect and process with a barrage of advanced technologies, software, and math to wring money out of you with far greater efficiency. They do that by using the information to target you with ads that can be so specific and relentless that they seem a little creepy at times. Use Google’s Chrome browser to search for a fruit-flavored green tea and you will probably find yourself hounded for days or weeks by ads from tea sellers that pop up to the side of other pages that Google points you to. Writing the code that does that is how some of the greatest mathematical minds of the current generation make their living these days.

That’s Google’s edge: It is in the enviable position of benefiting from having users online in almost every way (but it greatly prefers to keep them at sites available to its scrutiny through the Chrome browser and Android apps). Facebook, on the other hand, can learn about people and profit from them only when they’re on the site (a fact that helps explain Mark Zuckerberg’s fervent desire that we all just get over our archaic notions about privacy). So now Facebook’s triumph is emboldening the network to take on more and more services in the interest of keeping users within its walls.

Here, Facebook has two potent weapons: crowdsourcing and games. Google’s success at collecting information and driving commerce has created incentives for sites to manipulate the system with search-engine optimization tricks that artificially elevate their ranks in search results. Distorted search results are in turn prompting frustrated Web surfers to crowdsource their questions to trusted branches of their social networks. The idea is that your friends and relations can steer you toward a good answer much more reliably than Google’s immensely powerful but compromised data centers.

Quora, a question-answering social site, has sprung into being for precisely that purpose. Here, too, Google isn’t sitting still, but you already knew that. Improved search is part of the rationale for the +1 feature, which allows users to elevate their favorite search results in queries from their friends.

Facebook has also made extremely successful use of online games to keep people within its domain. Some 250 million people play social-network games every month, according to the analysis firm Parks Associates. The ascendance of these games has been so swift that media analysts are blaming it for the death of the TV soap opera in the United States, after a reign that lasted for decades. There are scores of Facebook games, but just two—Zynga’s CityVille and FarmVille—account for 140 million of the 250 million people who play social games every month. Google’s plans for online games aren’t known, but nobody doubts that some of the fiercest battles for online revenues will occur in the arena of gaming. And as our contributor David Kushner notes, these new social diversions may be as different from action-heavy console games as bucolic FarmVille is from the bloody Call of Duty first-person shooters [see “Betting the Farm on Games”].

As improbable as it might seem now, Google and Facebook could yet lose their grip on the new social Web. They will thrive only as long as online ad revenue flows, and that flow can be maddeningly fickle and elusive [see “The Revolution Will Not Be Monetized”]. Their snooping may even backfire. Some users have already decided that they would rather not blindly trust their social networking and Web-search history to anyone. More customized service, too, is always a lure. So four young techies in San Francisco have found a niche and are trying to fill it with a different kind of social network, called Diaspora. We’ve got the inside story on the ups and downs of life at the tech start-up of the moment [see “The Anti-Facebook”].

Google and Facebook, meanwhile, are grappling with a rather different sort of engineering problem: how to build data centers that push the boundaries of what’s possible now, to keep up with epic demands for processing power, data storage, and more [see “Under the Hood at Google and Facebook”]. Even here, the two giants embrace fascinating philosophical differences, and this is just one tier of a fast-evolving technology ecosystem to which the two must be constantly adapting. Smartphones, tablets, and netbooks are now so popular that accessing the Web to settle a bet, tweet a short movie review, check in with work, or post a Facebook status update has become an almost inescapable adjunct to any other activity. No one with a smartphone is ever out of touch with his or her online social circle, for better or worse. Even shooting, editing, and posting video—something that professionals could do only in the studio a few years ago—can now be done on the fly.

Technology will also give us a whole new concept of mobility. Just as GPS units in phones make it possible for people to spontaneously advertise their coordinates at all times, new kinds of sensors linked to the Web and embedded in clothes, buildings, vehicles, and other common objects will be able to convey ongoing updates about your every action. People will need to do less and less in the future to loose a torrent of data about themselves and their ongoing activities onto the Web. Whether anyone will really want to be the recipient of these data deluges is a moot point. Much of this information might seem meaningless, but the availability of robust computing capabilities for processing constellations of such data drawn from many parts of our lives—and the lives of other demographically relevant people—could certainly make it vivid and might very well make it compelling.

There’s a downside, and it’s a doozy. A system for tracking everyone’s actions more closely than 1984’s telescreens sounds uncomfortably like the greatest threat to personal privacy that the world has ever seen. But in capitalist democracies, at least, the more immediate worries are that corporate marketing could gain major advantages from knowing so much about us, and that minor lapses or, as they say, youthful indiscretions could wind up wrecking some people’s lives and careers [see “Welcome to the Surveillance Society” and “Me, Myself, or I”].

Regaining traditional privacy may be impossible, but that doesn’t mean we should be entirely at the mercy of corporate whims in their stewardship of our data. And so in this issue Web pundit Jeff Jarvis proposes a framework for a “bill of rights” that might help to curb abuses by Google, Facebook, and the other giants weaving this new social web and struggling for advantage in it [see “Privacy, Publicness, and the Web: A Manifesto”].

Google and Facebook are both counting on human creativity to drive their future success. So they are fostering lavish workplace cultures—with beautiful campuses, gourmet food, and at Google, conveniently located dog-poop disposal stations. (Really.) You may be surprised (we sure were) by what it takes to lure, pamper, and retain a top techie these days [see “Campus Life” and “Food Fight”].

The social networks that will come out of these brainy hothouses will undoubtedly have surprising cultural consequences. Life support excepted, the most fundamental human need is companionship. And so humankind has turned its newest technologies—computers, networks, mobile gizmos, and software—to one of its oldest and most basic requirements. A new and interesting chapter has begun. You’ll like it, although there are bound to be a few scary parts. We can’t tell you how the chapter will turn out. But when you’ve read our report, at least you’ll know what to fear and what to hope for.