collective unconscious of the smarter planet

More details on Calxeda’s latest ARM server processor - a tweaked version of the Cortex-A9 that is specifically designed for enterprise computing.

Calxeda, formerly known as Smooth-Stone in reference to the river rock that the mythical David used in his sling to slay Goliath, doesn’t think the server racket can wait for the 64-bit ARMv8 architecture (announced late last week) to be designed and tested in the next few years.

And that is why Calxeda has spent the past several years tweaking the 32-bit ARMv7 core to come up with its own system-on-chip (SoC) and related interconnect fabric suitable for hyperscale parallel and distributed computing where nodes have only modest memory needs.

Today, Calxeda takes the wraps off its much-anticipated ARM server processor, which has been given the name EnergyCore in reference to the fact that like other ARM chips used in smartphones and tablets, it is focused on doing computing work for the least amount of energy possible. The idea is that by switching to ARM cores, Calxeda can do a unit of computing work burning less juice than an x86 chip from Intel or Advanced Micro Devices, the Power chip from IBM, the Sparc T from Oracle, or the Itanium from Intel.

The EnergyCore ECX-1000 Series chips, as the first EnergyCores will be called, are based on the Cortex-A9 designs from ARM Holdings. The ECX-1000 chips are in fact based on a quad-core implementation of the Cortex-A9 chip, but like other server implementations of the ARM chips, such as the X-Gene announced last week by Applied Micro Circuits, there is a lot more to these chips than the core.

There is a slew of other stuff, including a fabric interconnect and a management controller that would otherwise be an add-on for the system board, on the chip. One big difference between the EnergyCore and X-Gene is that the latter is based on the 64-bit ARMv8 and won’t ship until the second half of next year if all goes well at Applied Micro. And that will be early silicon. It remains to be seen when server makers will pick up the X-Gene chip and actually get it into servers, but that might take until 2013.

Calxeda thinks there’s money to be made now, and for some workloads, the EnergyCore chips are going to fit the power bill. “ARM does for the processor world what Linux did for the operating system world,” Karl Freund, vice president of marketing at Calxeda, tells El Reg. “It opens up the chip market to a whole lot of innovation.”

The ECX-1000 chips are implemented in a 40 nanometer process and are manufactured by Taiwan Semiconductor Manufacturing Corp, which seems to be the foundry of choice for server chip makers that don’t have their own wafer baking facilities. Each Cortex-A9 core runs at 1.1GHz or 1.4GHz and includes a scalar floating point unit that can do single-precision or double-precision operations as well as a NEON SIMD media processing unit that has 64-bit and 128-bit registers and that can also do floating point ops…

Calxeda is not trying to do cache coherency over one to four ECX-1000 sockets on a system board or across the 1,024 possible system boards that the integrated fabric switch scales to. And it is not particularly worried about latencies as parallel workloads pass data around this switch fabric.

“If you look at the workloads we are aiming at, they are not latency sensitive,” says Freund. This includes offline analytics like MapReduce big data chewing, Web applications, middleware and Memcached, and storage and file serving. It would be interesting to see how a network of these puppies runs a shared-nothing database cluster.

The topology of the EnergyCore Fabric Switch can be changed on the fly from one style to another and the settings are stored on flash memory on the chip package. Bandwidth can be dynamically allocated in 1Gb/sec, 2.5Gb/sec, 5Gb/sec and 10Gb/sec virtual pipe sizes by the fabric switch and presents two Ethernet ports to the operating system.

The idea is to eliminate the top-of-rack Layer 2 switch that is typically used in a cluster these days with the on-chip switch. While it is possible to build a cluster with 4,096 ECX-1000 chips by using 10Gb/sec XAUI cables and the four ports coming off the Calxeda board to cross link them all, Freund says that most companies will put two real 10GE switches in a rack and use these like end-of-row switches and only lash together 72 four-socket nodes (about a half rack of servers) with the integrated fabric.

The other important thing about that EnergyCore Fabric Switch is that is has dynamic routing, which means you can get around congestion in a network of nodes and also, in conjunction with that management controller, optimize operations for latency or reduced power consumption – or boosted power consumption if you have some work that needs to run faster…

It’s hard to imagine server makers won’t be lining up to get their hands on these EnergyCards. And if they want to start selling them right away, Calxeda is good with that, too. The chips will be able to run Canonical’s Ubuntu and Red Hat’s Fedora Linuxes to start; Windows Server 8 could eventually get there if Microsoft gets interested. (So far, it has made no commitments, even with Windows 8 for clients and mobile phones getting an ARM port.)

The ECX-1000 chips will sample in late 2011, right on time, and volume shipments of the chips will start in the middle of 2012. That is about when Applied Micro will begin sampling its 64-bit X-Gene chip, which has its own crossbar switch but one that implements up to 128-way symmetric multiprocessing across 64 of its two-core chips.

It will be interesting to see how these two ARM server chips compete against each other as well as against other RISC chips and Intel Xeon and AMD Opteron processors. And remember, graphics chip maker Nvidia, which sells ARM-based SoCs for smartphones and tablets, has also promised ARM chips of its own designs for PCs and servers. Thank heavens for a little competition to keep Intel and AMD honest. Or whatever you might call it.

HP has just announced the creation of a new hyperscale computing business unit. They are partnering with Calxeda to build an ecosystem around low-energy computing platforms composed of hundreds of ARM-based processing cores. These platforms will support emerging high-volume web and cloud workloads.

Emerging ARM server Calxeda has been hinting for some time that they had a significant partnership announcement in the works, and while we didn’t necessarily not believe them, we hear a lot of claims from startups telling us to “stay tuned” for something big. Sometimes they pan out, sometimes they simply go away. But this morning Calxeda surpassed our expectations by unveiling just one major systems partner – but it just happens to be Hewlett Packard, which dominates the WW market for x86 servers.

At its core (unintended but not bad pun), the HP Hyperscale business unit Project Moonshot and Calxeda’s server technology are about improving the efficiency of web and cloud workloads, and promises improvements in excess of 90% in power efficiency and similar improvements in physical density compared with current x86 solutions. As I noted in my first post on ARM servers and other documents, even if these estimates turn out to be exaggerated, there is still a generous window within which to do much, much, better than current technologies. And workloads (such as memcache, Hadoop, static web servers) will be selected for their fit to this new platform, so the workloads that run on these new platforms will potentially come close to the cases quoted by HP and Calxeda.

The Program And New HP Business Unit

Officially, the announcement was HP announcing their new hyperscale business unit, based on the premise that very high-volume data centers will continue to proliferate, driven by massive continued increases in demand for web and cloud-based applications handling massive amounts of data, and that the trajectory of current systems technology with respect to power, cooling and density may be inadequate for emerging requirements.

HP’s hyperscale initiative consists of three major components:

• Discovery centers – Facilities where potential partners and customers can experiment with HP’s new hyperscale products. These centers are a vital component of this initiative because for the most part the workloads are new to this new platform, and both potential users, HP, Calxeda and other partners have a lot of learning to do to about which applications really fit and how to tune them. In effect a learning lab for both customers and suppliers. In many ways the knowledge gained in these centers is more valuable in the early phases than any product revenues that flow from them.
• Partner ecosystem – The usual suspects, software and hardware partners needed to facilitate the success of the new business unit. While HP was clear in their statements that they will be looking at multiple technologies, the entire announcement is centered around Calxeda, who is simultaneously announcing their EnergyCore server architecture and their accompanying EnergyCard reference architecture. In addition to a number of cloud-centric partners, the initial partnership roster includes Cannonical (Ubuntu Linux), putting a fully functional Linux distribution in the plus column for the nascent ARM ecosystem.
• A product – The HP “Redstone” development system, based on the existing SL6500 system enclosure and Calxeda’s EnergyCore servers. The SL6500 is HP’s current dense rack offering for HPC and hyperscale web computing. Redstone swaps out the current x86 servers and substitutes modules with 18 Calxeda EnergyCard servers, cute little 10” x 3” cards that contain four complete SOC quad-core server nodes with integrated memory, management processor, scalable fabric and integrated switch and all network and SATA interfaces, with a  5W per server/20W per card maximum power draw. In total, each server tray packs 72 quad-core ARM servers in 1 RU equivalent of space. If you read my last post on Calxeda’s reference architecture, you can guess that the basic Calxeda architecture is indeed the core of the HP offering, but in keeping with Calxeda’s OEM business model, HP has added value around packaging, extending the SOC fabric topology & I/O, management and power/cooling technology, and will add further value as the line matures.

What Does It Mean

Reduced to its essence, this means that ARM servers are on the industry road map. Among the major effected constituents:

• System vendors - HP is clearly placing its bets on an emerging segment of the server market that cannot be met with current CPU x86 CPU technology and current server designs. As the dominant x86 server vendor, HP is making an intelligent bet, and is now well positioned, with a solid first-mover advantage over its competitors, to capture new opportunities among its existing customer base as well as to capture others who might have gone away and patronized a new ARM server startup in search of ultimate energy efficiency. We might suspect that ARM has had discussions with Dell and IBM. More news to come?
• Customers – Now have (or will have in 2012 when HP officially ships the Redstone) a viable alternative CPU architecture to deploy for appropriate workloads, and I expect a lot of demand for evaluation units and access to the discovery centers. The potential to improve throughput per watt by such huge factors is incentive enough to seriously evaluate the new alternative, and my recommendation is to take a look and see how it works with your own applications.
• Intel and AMD – How about a giant wakeup call? I seriously doubt that this has caught them totally by surprise – the studied silence and nonchalance over the past year with which they responded to any inquiries about the impact of ARM competition had me convinced that they were actually quite worried. But being concerned in the abstract and having your No. 1  customer endorse not only your competition but an entirely new architecture are two different things entirely. Will this destroy Intel and AMD as server vendors? The thought is absolute nonsense. Aside from the large number of workloads that will not particularly benefit from the ARM model, both will respond with further focused R&D to continue to improve their power efficiency, leveraging their strengths in software compatibility and in Intel’s case, their market dominance.

My Takeaway

Not that it was exactly boring before, but the server world just got a whole lot more interesting, and customers just got a major early Christmas present – a whole new technology platform for emerging high-volume web and cloud workloads. All in all a very positive event for the industry and for us, the eventual beneficiaries of this technology.

Calxeda is beginning to reach out to big server and storage manufacturers to help push their ARM-based system-on-chip designs. HP is their first partnership.

HP is partnering ARM-licensee Calxeda to build energy-efficient micro-servers for large data centres, the WSJ reports.

Calxeda is producing 4-core, 32-bit, ARM-based system-on-chip (SOC) designs, developed from ARM’s Cortex A9. It says it can deliver a server node with a thermal envelope of less than 5 watts. In the summer it was designing an interconnect to link thousands of these things together. A 2U rack enclosure could hold 120 server nodes: that’s 480 cores.

The company is supporting an ecosystem of hardware and software partners with a focus on Linux.

From the WSJ report, it appears that HP has joined that ecosystem. It will have made the comparison between the Calxeda server nodes’ power consumption and that of Intel’s 20 watt and 15 watt Sandy Bridge Xeon processors, and a planned Xeon drawing less than 10 watts due next year. Calxeda’s thermal envelope is far superior for data centre operators needing to optimise for power efficiency and willing to forego any advantages of Xeon’s 64-bit memory.

Intel’s Atom products also use more power than ARM chips and Intel intends to develop them to draw less than 10 watts as well. These are due next year. It looks as if HP and Calxeda servers will have a thermal envelope advantage – at least until 2013, when Intel may have Atom designs closer to Calxeda’s server node power characteristics, not that Intel is saying anything about that.

HP has 30 per cent or so server market share and is closely aligned with Intel, using its Itanium design for high-end servers. Intel believes that performance is more important to the broad mass of its customers than power efficiency but, even so, is developing more power-efficient chips. However the X86 architecture is a power hog.

Calxeda is said to be talking to other server manufacturers and storage vendors and we might expect more ARM-powered server and storage controller news to be revealed in the coming months.

It’s not super surprising, but Canonical is developing a Ubuntu variant for ARM-based microservers. SeaMicro and Calxeda are still waiting to take off, but the industry is beginning to place bets that microservers are going to take off.

Canonical is suiting up for the coming microserver wars, confirming that Ubuntu Server 11.10 will run on ARM chips.

Just under three years ago when ARM-based netbooks were taking the PC market by storm and iPad tablets were just a gleam in Steve Job’s eye, Canonical, the commercial sponsor of the Ubuntu Linux distro, made ARM processors full peers with x64 processors running its Ubuntu Desktop variant. And now, perhaps at the dawn of an ARM-based server era that will see the x64 architecture get some tough competition for the first time in a decade, Canonical is getting out on the bleeding edge by supporting Ubuntu Server on ARM-based servers.

Chris Kenyon, vice president of OEM services at Canonical, has confirmed in a blog post that the upcoming “Oneiric Ocelot” Ubuntu 11.10 due in October would include a server variant that would boot on ARM-based machines. ARM support will not be an afterthought, but come out simultaneously for machines as well as x86 and x64 servers and support server-class chipsets and peripherals…

The initial focus for Ubuntu Server will be on microservers, a broad category of machines that offer better performance per watt than standard or even low-volt Xeon and Opteron parts can deliver running distributed server workloads.

“We are very excited about this area,” says Kenyon in a video snippet on the blog. “We are convinced that we will see ARM and frankly x86 processors in here. There’s some really interesting collaboration between SeaMicro and Intel going on here and between Calxeda and ARM. And we are already seeing some big Wen 2.0 properties experiment with this type of computing. It’s a very interesting area and one that we see Ubuntu Server being a part of.”

Hopefully, for the sake of competition, the ARM-based server racket will do better than netbooks over the long haul. The tablet and cheap PCs came along just in time to make them irrelevant.

Microservers will probably be a niche market, accounting for around 10 per cent of shipments, according to Intel. But given that hyperscale data centers will be buying scads of these small, power-efficient, multicore and often multi-node machines, Intel is not about to cede this market to ARM and if fighting back with low-powered Xeons aimed at single-socket nodes and low-powered Atoms with server features like ECC memory and virtualization support.

Intel’s market to lose

Calxeda, which has been cooking up its own quad-core ARM processors and a fabric interconnect for server nodes, says it can cram 120 server nodes into a 2U rack-mounted chassis. The Calxeda machines will be based on the 32-bit Cotrex-A9 processor design from ARM Holdings, and it will have a DDR3 memory controller with ECC scrubbing added to them.

The company says it can deliver a server node consisting of a quad-core chip with the memory controller and the interconnect electronics plus one 4GB memory stick for under 5 watts of juice consumed.

Calxeda is not planning on making servers, but selling the chips to server makers. And it looks like Ubuntu Server will be one of the first – if not the first – operating systems running on the box when it ships next year. Calxeda is planning to ship chip samples and reference servers to partners before the end of the year.

Whitebox server maker ZT Systems, like Calxeda, couldn’t wait for the Cortex-A15 design to settle down and last November launched an ARM server packing eight STMicroelectronics’ Spear1310 processor nodes into a 1U rack-mounted chassis.

SeaMicro has a microserver called the SM1000-64HD that currently jam 384 of Intel’s Atom N570 processors and a proprietary load balancer and torus interconnect into 10U chassis. The company is currently enthusiastic about the Atom roadmap, but has made it clear since it came out of stealth mode last June that it was not married to any particular processor. Expect an ARM-based version at some point if the performance per watt of the Atom chip doesn’t keep pace.

While Microsoft has said that it is porting Windows 8 to the ARM architecture, thus far it has not agreed to port Windows Server and its related systems programs – SQL Server, Exchange Server, and so on – to ARM chips. And unless Microsoft thinks it will lose money by not doing such a port, it is unlikely that Microsoft will put a lot of work in.

At this point, the move toward microservers from plain-vanilla two-socket rack servers could end up being a Linux-only phenomenon, given the hyperscale data center operators that are most interested in these tiny machines.

Calxeda is beginning the arduous task of building up a software/developer ecosystem for their ARM-based server solution, and by the looks of it, they’re opening line-up is pretty strong.

With Intel’s top brass bad-mouthing ARM-based servers, upstart server chip maker Calxeda can’t let Intel do all the talking. It has to put together an ecosystem of hardware and software partners who believe there’s a place for a low-power, 32-bit ARM-based server platform in the data center.

To that end, Calxeda, formerly known as Smooth-Stone, is launching the “trailblazer initiative” - a team of 10 software companies that will support upcoming servers based on Calxeda’s impending ARM-based system-on-chip (SoC) designs

The Calxeda ARM super friends include Autonomic Resources, Canonical, Caringo, Couchbase, Datastax, Eucalyptus Systems, Gluster, Momentum SI, Opscode, and Pervasive.

Canonical is of course, the commercial sponsor of the Ubuntu distribution of Linux, which is now first in line as the server operating system of choice for Calxeda ARM-based servers.

Eucalyptus Systems provides a cloud framework that clones the API stack and operations of Amazon’s EC2 cloud. Canonical was a big champion of Eucalyptus but has now thrown its weight behind the OpenStack cloud framework for compute and storage clouds created initially by NASA and Rackspace Hosting and now the darling of the open source community.

If Windows happens, great

Microsoft is not one of the trailblazing software partners for Calxeda, but it would be interesting if it were. Karl Freund, the vice president of marketing at Calxeda, tells El Reg that the chip maker is building its business plan around Linux. “Microsoft has not made any announcements about Windows Server on ARM,” Freund warns. “If Windows happens, great. That’s all upside for us and we’ll love it if it happens.”

Opscode is used to automate cloud operations, and Autonomic Resources is a cloud computing provider to the US government, and Caringo and Gluster do cloudy and clustered storage. Couchbase provides a NoSQL database called Membase Server, and Datastax rides the Hadoop big data chewer atop the Cassandra distributed database with the Hive query language.

Pervasive has commercialized the PostgreSQL database in the past and now sells a database based on Btrieve called PSQL and a line of data integration and analytics products called DataRush. Momentum SI sells a set of tools to do load balancing and auto-scaling on private clouds based on VMware’s vCloud Director or the Eucalyptus framework.

“The biggest opportunity for us in is cloud and big data,” says Freund. “And now, when someone says, ‘Forget ARM servers’, we can say take a look at these companies. They think there is a latent market for ARM servers, and based on the ARM skillsets out there, that this is going to be easy.”

‘Multiple thousands of cores’

We do know that Calxeda is designing the interconnect to link “multiple thousands of cores” together, albeit not in a cache-coherent manner but more link what you do with a rack of servers or multiple racks of servers in a cluster or cloud. The initial reference architecture machines pack 120 server nodes (that’s 480 cores) in a 2U chassis.

The Calxeda super friends will get early access to hardware, which should be in their hands by the end of the year. End-user trailblazers will also get machines at this time.

Calxeda has not said exactly when its chips will be ready for distribution for commercial use, or who will make systems based on its ARM SoCs. But Freund says that each partner that commits to building systems will have a handful of real customers putting the machines through the paces on a beta test by the end of this year.