collective unconscious of the smarter planet

A really good piece on how server arrays work - and the battle that’s taking place to reshape how they work in fundamental ways.

There is a battle going on behind the scenes over the location of storage’s soul: the controller hardware and software. Oracle, Dell, EMC and VMware want it to be in the server, while NetApp and HDS want it to be in the array, an array operating with servers but distinct from them.

The picture is not as clear-cut as this on the surface – NetApp is working with Oracle for example – but this is my take on what is happening down in the development depths, among the strategists and engineers with multi-year product horizons.

The modern storage industry, the one shipping networked external storage arrays, has been built on two foundations. One is EMC’s establishment of a market for third-party external, block-addressed storage arrays distinct from the server suppliers of the time: HP, IBM, Digital Equipment, etc.

The other was the invention and establishment of file-addressed network-attached storage (NAS or filers). NetApp is the single most effective proponent of that, although EMC grew to ship more filers than NetApp. EMC and NetApp represent the twin peaks of the external storage array.

A storage array comes in two flavours. It is either monolithic, with multiple controllers or engines and some fancy interconnect hardware to link these to the storage shelves – think Symmetrix, latterly VMAX – or modular. Modular arrays have two controllers linked – by simpler Fibre Channel or latterly SAS – to the storage shelves. NetApp’s FAS arrays and EMC’s CLARiiON are classic embodiments of this idea.

Applications in servers sent SCSI block requests or file access requests to these arrays, which presented themselves, logically, as a single pool of storage, separated into dedicated logical disks (LUNs) for the server apps, or sharable filestores.

This long-lived storage concept is now being discarded, and the first nail in its coffin came from Sun and the inventive Mr Andy Bechtolsheim.

Honeycomb upsets the storage hive

Bechtolsheim’s idea was that co-locating servers and storage in the same overall enclosure would speed server apps dependent on lots of stored data. Thumper, a server-rich NAS device delivered as the X4500, was one result of this and Honeycomb another.

Neither set the world on fire but they did show the way to getting more data into servers faster. Then Oracle bought Sun in 2009 and suddenly Bechtolsheim’s idea got a rocket boost from the Exadata product, a set of server resources running Oracle software with their own storage resources. This is setting the Oracle World on fire, with much encouragement from Oracle marketing because its own bunch of modular arrays was pretty second-rate.

What Sun invented and Oracle extended is the NoSAN server. EMC has seen this idea and responded by devising an opposite of this, the No-Server SAN, a kind of reverse engineering in its way.

EMC brings the servers to the array

EMC is trying to have it both ways. VMAX, VNX and Isilon arrays are going to be able to run application software in server engines in the array controller complex. There is a natural fit with VMware’s ESX running the whole shebang and VMs being loaded to run storage controller software and applications that benefit from low-latency access to buckets of data. These array-located app servers use the array’s own internal network or fabric, VMAX’s Virtual Matrix for example, instead of the normal Ethernet or Fibre Channel fabric. This isn’t SAN access as we know it.

EMC also has its Project Lightning to have its arrays manage the loading and running of flash caches in servers, PCIe-connected flash. That’s a step on the road along which Dell appears to be further along. The Round Rock company is also going to build servers with flash, but this is a storage tier and not cache. This tier zero storage is logically part of the entire array controller-managed storage pool with automatic data movement.

Now EMC may well have this in mind as well, with FAST VP shipping data to and from the server flash which is then not really a cache but tier zero too. However Dell’s vision, as I understand it, is to move the storage into the servers and hook it up to the same PCIe gen 3 bus that the flash and server’s DRAM hook into. Once again this means that the servers will not use traditional external storage links to access data. This again is not a SAN. What do these NoSAN ideas mean for external array vendors?

Storage array vendor conundrum

The mainstream external storage array vendors are facing two, not one, strategic challenges to their long-term hegemony. The nearer term one is the introduction of networked flash-only arrays from companies like Huawei, Nimbus, Pure Storage and others, which threaten to cream off their profitable fast array business based on 15,000rpm Fibre Channel disk drives. All primary data storage could migrate to flash-only arrays, leaving the EMC, Dell, HDS, HP, IBM and NetApp-style vendors to store the bulk data crumbs, the low-access rate, lower profit, online data repositories.

The second threat though is the more important one because, after all, a networked external flash storage array is still a networked external storage array. Co-locating servers and storage with no traditional storage networking protocol connecting them is different. If servers absorb external storage, as is the Oracle Exadata plan, or external storage absorbs servers, as is the import of Dell and EMC’s plans, then networked external storage is no longer needed: we have the NoSAN server.

There’s a distinction to be made between converged systems with co-located servers and storage using storage networking protocols and converged systems that do not, the NoSAN servers.

A VCE Vblock, an HP VS3 and a NetApp Flexpod set-up all use networked external storage. An app-running VMAX has no storage networking protocol needed to link servers (app-executing VMAX engines) and storage (storage controller app-executing VMAX engines) as they all hook into the VMAX internal fabric. This is fundamentally an entirely different beast. For a start the server-storage link is proprietary and not open. You wouldn’t be able to substitute a NetApp, HP or IBM array for the VMAX storage part of VMAX app-executing system. It would be like trying to put a BMW engine in a Mercedes car. Good luck to you and don’t expect any sympathy for the grief heading your way.

Criticise Fibre Channel, FCoE and iSCSI all you like but they are open systems allowing you to switch suppliers of servers and storage either side of the storage networking link.

FCoE becomes irrelevant

Discussions about whether to move to Fibre Channel over Ethernet (FCoE) from Fibre Channel will be rendered irrelevant by co-located servers and storage communicating across some kind of I/O backplane or mesh network, or servers and storage talking across PCIe. FCoE is a transition from one instantiation of a legacy storage-server connection protocol to another.

The future may be not to have a storage networking protocol at all. Data will ship between server memory and flash to co-located array storage across PCIe buss links or an array’s internal fabric.

What this means for stand-alone storage array products is that a razor thin end of a wedge is appearing in the door through which they sell their arrays. This wedge can be expected to fatten. If it does, then the external array business will be impacted. It may be that data growth will be such that array sales increase while NoSAN server storage sales grow as well. But there is a long-term threat here, looking at a five-year or 10-year timescale, in which the external array business declines.

Its only hope of rescue will be through the provision of external array links that match the latency and bandwidth of third-gen PCIe, or internal array fabrics such as is found inside VMAX.

A strategic battle has been joined and the existence of the external storage array business is being questioned. Yes, it’s a razor thin wedge that has been slipped into the external array door, but wedges do what wedges do: they become thicker. If Dell and EMC pull their NoSAN server ideas off, then a world of hurt awaits external array vendors that have no effective response.

Oracle has just released a new database appliance that has Mark Hurd’s handwriting all over it…

Oracle CEO Larry Ellison has been in love with the idea of “all-in-one” and “out-of-the-box” since before the turn of the century. In Matthew Symonds’ portrait of Ellison entitled Softwar, he noted how four years prior to the book’s publication, Ellison had had an epiphany: the software and hardware industries should never have separated. “If Detroit ran like Silicon Valley,” the CEO told the author, “no one would sell cars, just parts.”

But Ellison was lacking two key ingredients, which he doesn’t lack now. One was Sun Microsystems, whose high-end UltraSPARC processor-based systems are credited with contributing to the very positive growth for Oracle reported just yesterday. The other key ingredient was former HP CEO, now Oracle President, Mark Hurd. Today, they’ve all come together to produce what Ellison had always envisioned: the Oracle database box.

From a software standpoint, the new Database Appliance is Oracle 11g Enterprise Edition, which uses the company’s clustering technology called RAC to run single database applications across multiple servers. Think of it as “vertical virtualization.”

From a hardware perspective, the Appliance is a Sun Fire 4U rack-mount chassis, that starts out with two nodes, each with 96 GB of memory and a pair of six-core Intel Xeon X5675 processors. Each of those has dual threading, 12 MB of cache and is clocked at 3.06 GHz.

In an effort to appeal to customers intrigued by cloud-based licensing as an alternative to hosting their databases locally, Oracle is introducing a “pay-as-you-grow” licensing model that charges customers by the scale of processors and database capacity used. “Customers can deploy the Oracle Database Appliance with as few as 2 processors cores to run their database servers, and incrementally scale up to the maximum of 24 processor cores,” reads an Oracle white paper released this afternoon (PDF available here). “This enables customers to deliver the performance and high availability that business users demand, and align software spending with business growth.”

It may have come from Larry Ellison’s inspiration, but it’s got Mark Hurd’s signature all over it.

Chuck Hollis looks at how virtualization is changing the way that organizations design and provision databases. His focus is obviously VMware, but it’s an interesting discussion nonetheless…

Inexorably, virtualization is changing how we think about every aspect of IT.

It’s already vastly changed how we think about physical IT infrastructure: servers, storage and network. 

From static allocations to dynamic pools of resources – without VMware’s popularity, we really wouldn’t be talking much these days about cloud and transforming to IT to a service.

But what about databases and data management?  

Clearly, many of these technologies haven’t made the transition to the new model.  At best, we’ve only been able to encapsulate and containerize legacy databases using virtual infrastructure vs. revisiting how databases might intelligently work in this new model.

How do we get databases to intelligently use dynamic resources?  How do we deliver database as a service?  And how do we make databases as easy to consume as other forms of infrastructure?

…

Today’s Databases Were Designed For A Physical World

The more you look, the more you realize that the vast majority of databases in use today were designed to operate in the physical world, and not the virtual one.  And that’s far from ideal.

One immediate example is the lack of dynamic resource utilization.  All databases use precious resources: memory, CPU and storage.  Even though most database workloads are extremely variable, the vast majority of databases expect a big, fat over-allocation of these resources – they’re not smart enough to request more when they need it, and give it back when they don’t.

Any “virtualized” database should be smart about the environment it’s running in – smart enough to request and release resources as circumstances change.

Another example is the absence of integrated provisioning using standard templates and workflows.  

Today, provisioning infrastructure can be dead simple on something like a Vblock: the administrator defines standard infrastructure services and templates, and creates or changes resource instances using a very high level of automation.

This isn’t true for provisioning new database instances – it’s still mostly a manual process that requires hands-on work by an important and scarce resource – the database administrator.   Any “virtualized” database should be as easy to provision as a virtual machine.

A final example might be the need for self-service portals.  In the infrastructure world, using products like vCloud Director, it’s easy for administrators to expose resources to anyone at all: other IT groups, even end users.  A simple portal explains your choices, collects your details, and gives you what you want: typically with little or no human intervention.  

More importantly, the system administrator is additionally armed with powerful tools that help manage the pool of resources: allocation, service levels, and so on.  Again, if we’re talking fully virtualized databases, the same generic model should apply.  

Consuming a new instance of a database should be as conceptually simple as consuming a virtual machine.  And managing pools of databases ought to be as straightforward as managing pools of virtual machines.

Ideally, virtualized databases would support dynamic resource usage, integrated provisioning and self-service pooled consumption models.  But, outside of a few exceptions, that’s not the case today.

Dynamic Resource Usage

One of the first things that leaps out of the announcement is the virtual enhancements the VMware team has made to the popular PostgreSQL database.  At the outset, a “balloon driver” is able to request and release memory based on changing circumstances.  The same sort of capability seems to be there for GemFire.  The announcement is pretty clear: more options coming over time.

Extending this idea a bit, it would be logical to assume that – eventually – this concept could include to storage performance (perhaps using a variety of mechanisms that are extensions of VAAI: linked clones, Storage vMotion and/or storage service pools) —  creating and releasing additional database storage instances (or perhaps relocating them to different storage tiers) thereby increasing or decreasing performance.  The same expand/contract potential exists for dynamically using virtual or physical cores.

I have observed that massive over-provisioning seems to be the accepted norm in the database world: overprovisioning on memory, overprovisioning on storage performance, and overprovisioning on CPU.  

Wouldn’t it be wonderful if databases were smart enough to take what they needed to meet changing service level requirements, and no more?  If they had the same elastic properties as other portions of the infrastructure?

That’s the goal here.

Integrated Provisioning

Everyone who has had the pleasure of doing physical server provisioning knows all of the sequential, labor-intensive and occasionally error-prone steps involved.  

Indeed, anyone who’s working in a fully integrated virtualized environment (such as a Vblock using UIM) probably doesn’t want to go back to the old way of doing things anytime soon.

Indeed, in these new environments, valuable system administrator time is now spent on more worthwhile, higher-order tasks vs. the drudgery of before.  

The database administrator in many regards is no different – their time is important as well, and they could greatly benefit from the same sorts of capabilities: far less time doing sequential, labor-intensive and occasionally error-prone grunt work; and far more time tackling the more interesting challenges and opportunities.

I haven’t had the opportunity to look at the new vFabric Data Director in gory detail, but from what I can see from the overviews, there appear to be the same sort of templates and automated workflow concepts you see in virtualized server provisioning workflows.

Ease Of Consumption

Today’s pooled and virtualized environments are designed to be easy to consume — that’s what the whole “as a service” thing is about.  

Popular request types can be easily exposed on a portal, and people can get what they need with an absolute minimum of human intervention.  Behind that, resource administrators now have powerful tools that help manage and control the pooled environment in aggregate.No such luck for most of the database world today.  

Getting (or changing) a database instance almost always involves tracking down a database administrator and asking them to do something on your behalf.  And, while database administrators have the tools to manage individual database instances, there’s not much out there that addresses their need to manage and control hundreds or thousands of database instances being delivered as a service.

That changes with the new vFabric Data Director.

Digging Deeper

I think once the novelty wears off, most IT thinkers will realize a few simple truths.

First, there’s a big and obvious problem to be solved here.  

I routinely meet customers who have hundreds and occasionally thousands of database instances swirling around their environment.  Telling people not to create new databases just means they’ll go elsewhere.  Not good.

And no one has the stomach for a massive “gee, let’s go consolidate a bunch of existing databases into a single humungous instance” project.  At least, not twice :)

The only viable approach for many?  Use virtualization techniques to lessen resource usage, control service delivery and manage the pool of database instances more efficiently.  Just like you do server instances.

Second, while the technology is capable of supporting demanding workloads, that’s not where it’s going to be used first.  Just as with server virtualization, the most appealing initial target will be non-critical database workloads vs. the big hairy stuff.   Make no mistake, that too will come — in time.

Third, the underlying hybrid cloud model is extremely relevant here.  If you think for a moment about external database and PaaS offeirngs (e.g. AWS, Azure, et. al.) there’s only one consumption option for each: their particular service.  Easy to get in to, somewhat more difficult to get out of …

Compare and contrast this with the vFabric Data Director approach where you’re free to set it up internally, use any number of compatible external service providers, or any particular combination that suits you.  

Fourth, I’ve met more than a few people that are looking for a different industry model to deliver database services to the business vs. buying more of what they already have.  Here’s a model that’s worthy of serious consideration.

Cisco incrementally improves their UCS stack with more advanced networking options and improved management software.

Not to be left out of the announcement fever that has gripped vendors recently, Cisco today announced several updates to their UCS product line aimed at easing potential system bottlenecks by improving the whole I/O chain between the network and the servers, and improving management, including:

• Improved Fabric Interconnect (FI) – The FI is the top of the UCS hardware hierarchy, a thinly disguised Nexus 5xxx series switch that connects the UCS hierarchy to the enterprise network and runs the UCS Manager (UCSM) software. Previously the highest end FI had 40 ports, each of which had to be specifically configured as Ethernet, FCoE, or FC. The new FI, the model 6248UP has 48 ports, each one of which can be flexibly assigned as up toa 10G port for any of the supported protocols. In addition to modestly raising the bandwidth, the 6248UP brings increased flexibility and a claimed 40% reduction in latency.
• New Fabric Extender (FEX) – The FEXC connects the individual UCS chassis with the FI. With the new 2208 FEX, Cisco doubles the bandwidth between the chassis and the FI.
• VIC1280 Virtual Interface Card (VIC) – At the bottom of the management hierarchy the new VIC1280 quadruples the bandwidth to each individual server to a total of 80 GB. The 80 GB can be presented as up to 8 10 GB physical NICs or teamed into a pair fo 40 Gb NICS, with up to 256 virtual devices (vNIC, vHBA, etc presented to the software running on the servers.

Taken as a whole, these improvements should help to communicate that Cisco intends to  allow their UCS environments to scale as customer requirements evolve, as well as nullifying some fo the competitor FUD about bandwidth limitations of UCS. They do not remove the fact that one of UCS’s weaknesses is that bandwidth scaling is decoupled from the scaling of the number of server nodes, but should move the bar higher for UCS users.

Additionally, Cisco announced UCSM 2.0, which includes features such as an integrated version of vMware vCenter and an express mode for simple configuration of smaller and remote UCS environments.

All in all, solid incremental improvement of an already strong product line. We expect that the upgrades of the I/O infrastructure will be followed at some point by improvements in compute nodes and possibly expanded scaling of the environment over time in order to respond to evolving customer demands.

The main gist of this article is EMC’s succession plan for their leadership, but there is some good information about the state of the storage market and how it impacts EMC’s business.

EMC business sitrep

The challenges EMC faces are, from one point of view, obvious. Storage for enterprise unstructured data and archive data looks set for a steady move to the public cloud, with an interim step being a move to private clouds. Then we can foresee hybrid clouds with some IT processing and storage done in-house and some done in the public cloud.

There will be fewer buyers of storage and and server-based software in the public cloud than in the private cloud, and they will wield more buying power, meaning lower prices. Large enterprise will be the slowest to move their enterprise-array-held data into the cloud so VMAX sales should hold up well into the future, especially if a common hardware platform can be developed for both the VMAX and VNX space.

Should EMC offer its own cloud storage service for enterprises? That would be the ultimate hybrid cloud, wouldn’t it? EMC storage in the private cloud and EMC storage and service being offered as a public cloud. Perhaps EMC should consider buying its way in to this market; Nirvanix looks a tasty morsel.

An aspect of the cloud is that tape is going to become more attractive as an archival data store, being cheaper than disk, and EMC may need to embrace tape rather than be cut out of that market.

Backup is still on a Data Domain/Avamar roll, but that can’t last forever and its future needs working out.

Data warehousing and business intelligence looks set to rock and roll for the foreseeable future, but it is a secondary business after the main business of storing data. Greenplum and that sort of thing are icing on the cake provided by VMAX and VNX.

Isilon and “Big Data” looks to be another revenue rocking and rolling area. The systems can ride the Intel processor curve, embrace flash and carry on riding the big data wave for a good few years.

The content management software area is a weak spot for EMC, but then it is a pretty weak area for everybody, as all it represents is a cost with ill-defined and ill-understood benefits. EMC may have to decide to merge it with Big Data and get some Big Data market star dust sprinkled over it.

The primary data storage array area is set to migrate to a 2-tier architecture: flash for hot data and SATA or SAS for the rest. This transition requires a smart product boss to oversee it.

Converging IT stacks

The converged IT stack area looks to be going gangbusters but EMC and Cisco and VMware, by the fact of their separate existences, may be at a disadvantage in integrating their vBlock echnology components into an all-singing, all-dancing, fast-performing IT stack as Oracle is set on doing. HP and IBM could follow Oracle’s path as they have the server hardware and HP also has its ProCurve networking gear.

Does EMC need a closer relationship with Cisco? Should EMC build its own servers? Should EMC buy a server company?

If it does decide it needs its own servers, then it may well face tensions with Cisco and end up needing a different networking supplier as well? How should it develop that capability?

Navigating through the IT converged stack minefield and dealing with Oracle will need a smart product person and an astute corporate relations person.

The Iomega consumer/SOHO product is also doing well. Our advice to the incumbent: Just leave it alone while ensuring it tucks neatly into EMC’s SME and enterprise product offerings, and don’t foul it up, that’s all that’s required.

Another point though is to tackle the decision about whether EMC needs its own large-scale services arm, to match Dell’s purchase of Perot Data Systems and the capabilities enjoyed by IBM and HP.

IDEAS International gives HP’s new converged infrastructure offerings the thumbs up.

In early June, HP announced an expansion of its strategy for HP Converged Infrastructure by rolling out a number of new products and services for implementing virtual, cloud and dedicated application environments using a common architecture and staged process. According to HP, “Converged Infrastructure is the HP strategy to help clients accelerate IT value, and do so at their own pace and preference. This is achieved through a systematic approach that enables clients to overcome the complexity, inflexibility, and high costs created by IT sprawl and turn technology assets into interoperable, shared pools of resources with a common management platform.”

With its extended strategy, HP now pursues a continuum of opportunities for its blade server platform, extending from pure vertical integration to more open integration, all supported by a common hardware architecture, management software, and security software.

The week of June 6, HP announced two new types of converged systems, and enhancements to a third, all of which are built on a single hardware foundation based on HP’s servers, storage and networking:

• HP VirtualSystem (New): This system integrates servers, storage, networking, virtualization software, and services into a system that is optimized for virtual server and virtual desktop environments from Microsoft, Citrix, and VMware.  It supports Microsoft Hyper-V, Citrix Xen, and VMware. An optional common security model, based on HP TippingPoint, delivers end-to-end security, and everything is managed through a single ‘pane of glass.’ In addition to TippingPoint, extensions to VirtualSystems include Site Scope, Data Protector, and Storage Essentials. Three preconfigured modular systems are available for virtual server, supporting 750, 2500, and 6000 VMs. A single system supporting 3000 virtual clients is also available.
• HP CloudSystem (Enhanced): This system, using the same infrastructure components as HP VirtualSystem, includes tools to build both private and hybrid clouds. CloudSystem, which was originally announced in January 2011 as the evolutionary follow-on to BladeSystem Matrix, adds intelligent automation and orchestration, along with ongoing management and security to support cloud multi-tenancy. On June 7, HP enhanced its CloudSystem offering by announcing a new service provider program, and a new array of cloud services. HP also announced “dual bursting.”  Bursting to local resources is available today and out of the box bursting to service providers will come later in 2011.  
• HP AppSystems (New): With this system, HP leverages Converged Infrastructure to create a number of application-specific configurations, based on solutions from Microsoft and Vertica. These configurations are fully configured, turnkey systems that can be brought online rapidly with minimal issues. Because HP AppSystems are all based on HP Converged Infrastructure, they can help customers transparently move these applications to virtual or cloud environments as appropriate. So far, HP has announced the following AppSystems (not all are yet shipping, though): HP Enterprise Data Warehouse for Microsoft, HP Business Decision Appliance for Microsoft, HP Messaging System for Microsoft, HP Vertica Analytics System, HP Database Consolidation Solution for Microsoft, HP Business Data Warehouse Appliance for Microsoft

Other vendors have their own versions of converged infrastructure, and HP is competing in an increasingly crowded and competitive market. Through the VCE (formerly Acadia) joint venture, Cisco, EMC and VMware had one of the earliest entries with the vBlock, which was announced in late 2009. IBM’s CloudBurst on System x and Power Systems is another tightly integrated, preconfigured solution that is similar in concept to HP CloudSystem. Dell’s recently announced vStart, a pre-packaged, tested, balanced, and fully validated virtual enterprise infrastructure solution. Finally, Hitachi’s Converged Data Center Solutions hit the market very recently. We expect to see even more competition going forward.

With this announcement, HP has not only shown that it has a solid vision and strategy, but an increasingly complete inventory of products, services and financing to back up their approach. HP has been demonstrating for years that it is very serious about converged infrastructure. Many potential customers had been holding back on implementing converged infrastructure because of the significant steps they must take to get there and the associated risks. By creating a staged approach that leverages common hardware, management, and security stacks, HP allows customers to move at their own pace from dedicated applications to virtualized applications, to cloud infrastructure, and finally to common modular datacenter components, thus minimizing risk. This announcement is an incremental, yet critical, step for HP as it strengthens its portfolio with vertically integrated systems that make it easy for users to take advantage of virtualization and the elasticity of cloud computing. We expect many more announcements like this as HP executes on its Converged Infrastructure strategy.

Peel open a big data appliance, and you’ll find an array of common-off-the-shelf (COTS) processors on blades, a very fast network backplane that’s good at virtualization, some custom load-sharing technology, and storage. That’s what the Cisco/VMWare/EMC marriage dubbed Acadia has in it, and it’s what’s in Oracle’s newly-announced Exalogic cloud-in-a-box, and it’s what Netezza makes. It resembles a legacy mainframe: elastic, shared, highly parallel, and very fast.

There’s a reason that the distributed, COTS data center is contracting into these high-performance appliances. A paper by the late Jim Gray of Microsoft that says that, compared to the cost of moving bytes around, everything else is free. That applies in data processing. It’s why Amazon’s S3 large-object store, not its EC2 compute service, is core to the company’s strategy: your computation goes to where your data is, not the other way around.