Recently in Cloud Management Category

On July 19, 2010, Rackspace led the announcement of OpenStack, with a goal of creating an open source cloud software solution for use on industry-standard hardware.  The initial releases contemplate solutions for both cloud compute and object storage.  While these are the first two releases, they are separate offerings.  Remember, cloud storage is not just the storage target for cloud computing, it is one potential storage target for cloud computing, and is in and of itself a stand alone cloud offering of programmable storage.

Now, I have purposely used a term from the clothing industry, "off the rack", to spend a moment looking at a framework for evaluating the opportunities this may present.  With dress shirts, you can buy off the rack, semi custom, or custom, each with a unique value proposition based on fit, choice and cost.   Interestingly enough, this may be a good lens through which to consider the possibilities of OpenStack, and in particular, OpenStack Object Storage.

Rackspace has made no secret of its motivations for leading this initiative, and its desire to focus on "fanatical" service as it's key differentiator versus the fundamental technology on which the service is based.  Fair enough, and so the question becomes, is the rapidly emerging and immature cloud marketplace already "mature" enough to seek homeostasis?  (Homeostasis is the property of a system, either open or closed, that regulates its internal environment and tends to maintain a stable, constant condition.)  Have enough models and innovations, from startups, academia, open source movements and large tech companies, been tested in the marketplace to the extent that we can already race to the common denominator?  Perhaps now is a good time to start, as long as you are willing to acknowledge that the desired results are a good ways off.

Before we jump off into "Off the Rack" software, a quick look back at open source is helpful.  For more reading on the open source software industry a good introduction is The Cathedral and the Bazaar. Six things are particularly interesting: 

1. An open source alternative can emerge as a follow on to a successful commercial technology and can become pervasive versus the commercial offerings it succeeded (LINUX versus UNIX is the reference case here).
   
2. A second result of this approach can also end up with a big success, although in more of a niche than a pervasive replace for the earlier commercial offerings (MySQL versus Oracle, IBM and Microsoft in the relational data base space).  
   
3. An open source effort can also emerge earlier in a technology cycle and come of age as a pervasive solution (Apache Web Server comes to mind here).
   
4. Open source generally requires very careful cultivation of the community of developers, with active interest by academia (and partnering with NASA is part of the formula here).  Commercially sponsored open source efforts are becoming more common, although it as of yet has not been proven as the typical "breeding ground" for most great open source successes.  Eucalyptus, with its roots at University of California Santa Barbara, seems to be a more traditional route.
   
5. Open source is not necessarily reflective of rapid commercial opportunities for success.  Eucalyptus is obviously beginning to maneuver towards a repeat of the commercialization model.  OpenStack is taking the approach most favored by other open source successes like Apache.  A couple of good reads here are this article from BusinessWeek and this. See also Derrick Harris' post over at GigaOm.
   
6. There are also hundreds of thousands of open source projects that had mixed success or languished altogether. A quick look at  SourceForge (an open source project hosting site) shows nearly a quarter million hosted projects. How many of these have languished or had little impact on the market.
   

So, the first issue is that there will exist for some time to come a real question as to the adoption potential of OpenStack.   I believe that adoption is driven by applicability to need.  In a moment we will address a serious issue which OpenStack Object Storage must overcome to be successful, at best, and at worst, will confine it to a niche market.  My views are very much directed at the Object Storage offering, versus the compute offering, which I believe exists in a different space and as a different type of solution.  With this backdrop, let's have a look at the cloud storage marketplace today, and use the analogy of off the rack, semi custom and custom:

• Off the Rack:  implement as is, one size fits all, each with unique approaches for performance, scalability, bit integrity, may or may not provide geo services.
• Semi Custom:  Select from storage types (DAS, SAN, NAS, JBOD), shared or distributed file systems and object systems, mix and match storage for different SLA and cost/usage patterns on the same infrastructure, multiple APIs, meta data and catalog abstracted from storage layer, geo services.
• Custom:  Generally a service only offering and not available as deployable infrastructure, specifics will vary widely based on service provider offering strategy.
  

Infrastructure	Type	Comments
Eucalyptus	Off the Rack	Limited S3 APIs
OpenStack	Off the Rack	CloudFiles APIs
Scality	Off the Rack	S3 APIs
Mezeo	Semi Custom	Mezeo ReST APIs and S3 APIs
NetApp	Off the Rack	Bycast APIs, NetApp storage
EMC Atmos	Off the Rack	Atmos ReST APIs, EMC storage
Service	Type	Comments
Amazon S3	Custom	S3 APIs
Microsoft Azure	Custom	Windows centric
Rackspace	Off the Rack	Is the basis for OpenStack
Nirvanix	Custom	SOAP APIs, multi node
Google	Custom	Offers S3 APIs
AT&T Synaptic	Off the Rack	Based on EMC Atmos
OpSource, SoftLayer, Layered Tech and others	Custom	Based on Mezeo

As you can see from the summary above, there exist as many views of what constitutes either a cloud storage service or a desirable cloud storage deployable infrastructure as there are service providers and vendors.  Note that a semi custom infrastructure results in a "custom" service as implemented.  "Off the rack" results in very similar services by those who utilize the same infrastructure unless they make their own major additions.  Any offering can be differentiated by service, and the degree and quality of service is critical to customer satisfaction and plays a strong role in value creation.

The OpenStack announcement as it regards Object Store and its approach to cloud storage seems to view cloud storage infrastructure as highly akin to an operating system (or at least a "hypervisor") and more similar to a selection of LINUX or Windows than that of an application or middleware layer.  While I agree that cloud compute is very close to this model, cloud storage is a service oriented architecture, with programmability for new applications that can tolerate Internet latency because of Web Services (like ReST APIs). The industry constantly overlooks this key point as it is consumed with the low cost, pay for use and thin provisioning capabilities of this storage tier.  Solutions for thin provisioning and low cost have been available far longer than cloud storage. Further, pay for use is more of a business decision than a technology. 

In the earliest days of cloud storage, there existed initial confusion that cloud storage was defined by cost, scalability, pay for use, and thin provisioning only and not programmable access (usually via ReST APIs).  ParaScale paid a huge price for not understanding that cloud storage requires Web services (like ReST API) access.  Now, with OpenStack Object Store, we see a follow on case of this same perspective, but with basic APIs for Put, Get and List.   Yes, it provides for Internet access via ReST APIs, but the focus continues to be primarily cost based versus new application enablement based.  It could be argued that the open source approach will provide for the appropriate additions of "advanced services" to be added.  However, even the use of the platform by NASA is more focused on cost of storage than on advanced functionality because NASA stores much more data than almost any institution or enterprise in the world.

I think Savio Rodrigues states this view very well in his post:

"Select products based on business needs, not license alone: It's also interesting to note that very few enterprises are in NASA's position with regards to size of IT investment and skills in-house. While NASA engineers were ready and willing to contribute new features into the Eucalyptus open source community, few companies have the skills or governance to consider allowing their developers to contribute to open source projects.  Summary trend number 7 from the 2010 Eclipse survey results highlighted this issue.

To suggest that NASA's buying or IT decision making patterns represents much more than the top 1 percent of IT buyers would be a stretch."

The overwhelming majority of enterprises would rather pay a vendor to deliver, maintain, support and enhance their private cloud software infrastructure than place that burden on internal IT staff. Whether the enterprise is paying for a closed source commercial product, a commercial product based on an open core product, or a subscription to an open source product, the product selection decision will be made based on business requirements much broader than 'is the product open source or not?' "

Keep in mind that cloud storage is a stand alone service associated with application delivery over the Internet and also associated with low cost, pay for use, scalable storage resources.  Social media applications and many Web based applications exploit these capabilities; for example publishing a file to a URL and significant tagging of files.

This view of cloud storage as nothing more than cost and volume-based ignores its extraordinary importance as a service-oriented architecture for new application enablement.  I believe both views are equally important and need to be equally served.  Will OpenStack, with its pervasive cost focus, be able to drive its community to this additional view of needed contributions of advanced services for cloud storage?  Lydia Leong of Gartner Group provides an interesting view of the open source community issues associated with this in her post:

"At the same time, open sourcing is not necessarily a way to software success. Rackspace has a whole host of new challenges that it will have to meet. First, it must ensure that the roadmap of the new project aligns sufficiently with its own needs, since it has decided that it will use the project's public codebase for its own service. Second, it now has to manage and just as importantly, lead, an open-source community, getting useful commits from outside contributors and managing the commit process. (Rackspace and NASA have formed a board for governance of the project, on which they have multiple seats but are in the minority.) Third, as with all such things, there are potential code-quality issues, the impact of which become significantly magnified when running operations at massive scale."

One last comment on this business of vendor lock in and cloud storage APIs (another focus of the OpenStack announcement).  I would submit that while a specific set of APIs has the potential to create vendor lock in, this is a much smaller problem than what is experienced in other technologies.  If you are really worried about it, you probably have never actually written a ReST API call.  It is written in many languages, and we have seen cases where applications that run on S3 run unchanged on Mezeo.  Others need very minor modifications, and still others are excited to take advantage of some of the unique Mezeo services.  It just is not a problem, and this is much more related to FUD (fear, uncertainty and doubt) and marketing zealotry than it is associated with technological reality.  The APIs of choice will shake out, and it is far to early to say if it will be S3, OpenStack, CDMI or a combination of all of these, and others, as yet unforeseen.  (At Mezeo, we have never believed there will be one winner, and instead focused on architecture to enable easy and effective delivery of whichever APIs stand the test of time.)

The interesting view that seems to be missing here is that marketplace competition by service providers already serves to drive down the price of cloud storage, so a commoditized stack embraced by most is unlikely to yield extraordinary incremental savings.  At the same time, while the competitive market conspires to drive cloud storage costs ever lower, the need to differentiate, and deliver solutions as well as a programmable storage to enable multiple new and exciting types of applications will rapidly replace the pure cost and scale focus of current cloud storage offerings.  Sometimes, the "new" application is simply enabling it in the cloud, to produce the same result at a lower cost!  This requires significant cloud storage functionality in order to make this easy and productive.  Amazon continues to prove this with their many additions and capabilities which differentiate their service.  Mezeo sees much the same view on the part of our customers.  The focus is on what cloud storage can do, what problems will it solve, what business opportunities does it create, what new applications can it enable and all of these views assume it will be competitively priced.

Cloud storage represents significant opportunities for institutions, the enterprise (see my recent post on the business case for enterprise cloud storage) and for the IT service provider.  Cloud storage is substantially different from cloud compute, and requires that you understand this difference in order to effectively evaluate the impact of this announcement, as well as your next steps.

There is no doubt that every enterprise has devoted some time and energy to evaluating how cloud technologies can best be put to work in their ongoing pursuit of cost reduction and to a lesser extent for potential improved service levels particularly around rapid provisioning of compute and storage resources.  Mezeo has recently begun to work with various enterprises, and I want to share some of the opportunities that appear to align strongly with these two goals.

In terms of cost, most enterprises are experiencing continued and significant growth in unstructured data.  As they look at the cost of this growth, it is more than just physical storage, data center occupancy, bandwidth utilization and power and the accompanying management demands; it is also the backup and disaster recovery requirements and the ability to quickly satisfy users who need more storage in order to execute whatever tasks and jobs they have.  Against this backdrop, the drumbeat of Amazon S3 and other public storage clouds advertises storage at costs that are generally below the internal "advertised" cost of the typical Fortune 500 company.  What gives?

First, cents/GB/month is only the tip of the iceberg, and bandwidth along with access charges gives a more realistic cost appraisal.  Next, real and legitimate concerns about data security exist (will someone gain unauthorized access, by accident or via an attack, to company data stored in a multi-tenant public storage cloud?).  Also, data integrity concerns are well founded (will the bits I store be returned, and will they be backed up and appropriate DR measures taken?).  Finally, can I absolutely trust the service provider to execute to the extent deemed necessary, and if they do, can they really save me any real money versus the assumed risk profile?  Private cloud computing is an appropriate strategy for addressing these issues.  

Not all unstructured data is a candidate for the latency of cloud storage as delivered from an IT service provider via the Internet.  So, while some tiers of data may be appropriate for a cloud storage service, it is a subset of the enterprise unstructured data requirement and not a lower cost panacea.  Hopefully, CIOs can easily make this case with their peers in senior management, although it may sometimes seem like they are making an excuse for keeping control and not exploiting new technologies.

Question one surrounds the cost proposition, and our analysis suggests that, even at sub petabyte initial cloud sizes, the enterprise can deliver economics for in-house cloud storage that compare very favorably.  In fact, it may even be lower than what is available from a service provider.  The Mezeo team comes from both a hosting and a cloud storage background, and this just reinforces our view that the cost proposition for private cloud storage has favorable economics.  However, if you are being forced to allocate capital for data center build outs, or you are otherwise CAPEX constrained, the hosted public cloud economics can be quite appealing.  Since businesses require positive margins, this further drives up the cost of cloud storage as hosted at a public service provider.

The case for improved user satisfaction is similar, regardless of public versus private, because the cloud gives users the capabilities they want.  First, with rapid provisioning of pay-as-you-go low cost cloud storage, the end user gets what they need when they need it via a frictionless interface.   Second, several benefits drive end user demand for cloud storage; including: avoidance of workstation storage upgrades, one solution for file sharing and collaboration, new capabilities and applications that exploit file search, tagging and publishing to a public URL, and the ability to access your storage anytime, anywhere and on any device.  Third, the solution is also ideal for implementing a workstation backup solution with sync.  It is not hard to see why end users would find all of these capabilities appealing.

Cloud storage clients, gateways and edge devices are also beginning to appear, and can solve many different issues.  For example, a client gives the end user access to multiple cloud storage accounts at multiple providers.  Why not replace that tape backup operation at a remote location with an iSCSI interface directly to a storage cloud, for a scheduled backup without local user intervention (get rid of the tape backup of your local file server, forever)?  Speaking of file servers, multiple solutions for replacing or even displacing file servers are coming to the market.  The savings from removing an entire layer of infrastructure are quite compelling.

New applications, including use of social media, may require file publishing.  Cloud storage allows you to store training videos, and make them easily available at every end user in the company.  Tagging and search offers new application capabilities, and new opportunities to support existing compliance requirements.  Secure file sharing, versus file publishing, may be a significant requirement as you work with customers and business partners.  Partner, customer and employee portals can reach new levels of capability with API accessible cloud storage, as the availability and the management of information is delivered via the cloud.

Our observation is that the early adopters have begun the move to cloud storage.  Why?  Simply, enterprise private cloud storage allows you to gain many of the benefits and set aside the security and data integrity concerns of public cloud storage.  At the same time, data tiering and private and public could solutions will drive "hybrid" cloud approaches that will allow the enterprise to exploit the best of both worlds.  In an upcoming post, we will offer up some tools to examine the cost and the benefits of cloud storage for the enterprise. 

Cloud Storage Strategy interviewed Gladinet co-founder Jerry Huang on cloud desktops, cloud gateways, and his company's business model. 

[NOTE: Gladinet is a customer of Mezeo Software.]

The definition of cloud storage has been on my mind lately, and I think some attention to this topic is still called for.  From an article in CXO Today:  Cloud storage is not a disk array that you own, lease, or manage neither is it a virtual logical unit number (LUN) from a larger disk array. It is in fact it is offered via an application programming interface (API) through which you can send and receive data without having to actively manage the storage.

I see many "Cloud Storage" services and vendors of cloud storage infrastructure products that do not do or provide for what is described in the preceding observation.  For example, some cloud storage services are really offerings of storage that are associated with cloud computing.  One major requirement is storing and retrieving cloud computing images.  Since these are "bootable" the typical storage approach is an iSCSI connected storage resource.  A cloud computing image may require files for its application, and these are often stored on shared storage systems, and may be accessed in a variety of ways, but not necessarily via Web services APIs. 

Often, IT service providers call this shared storage; however, when it is accessed by cloud computing images, it is often referred to as cloud storage.  Finally, block data, like  a data base, is often required for the application running on a cloud computing image, and is accessed via iSCSI, and may be referred to as cloud storage. 

So, where do these observations lead us?

There are many benefits of a storage cloud, and for the user these include ease of access (in a variety of ways) to various amounts of storage on an as needed basis, with instant or nearly instant provisioning, with little if any traditional storage management requirements for the user.  IT service providers, and enterprise IT organizations are fundamentally organized around the premise of service delivery.  So, for both of these entities, a service offering like cloud storage is an important business asset, and the primary differences in the deployed infrastructure is associated with multi-tenancy (which, among other things, drives different security requirements) and billing. 

For many months, I have relied on the following definition of cloud storage: a persistent storage solution for objects (also called files or unstructured data) accessed via Web services APIs via a network (LAN or WAN). 

Today, I would like to move forward and offer a new definition, more encompassing, and reflecting not just a purist view but attempting to capture what is truly important for an IT service provider, in house or as a focus of a business (hosters, telcos, and cloud providers): 

"Cloud Storage provides whatever amount of storage you require, on an immediate basis.  It is persistent.  It can be accessed in a variety of ways, both in the data center where the cloud is housed, as well as via the Internet.  If you obtain this from an external provider, it is purchased on a pay as you go basis.  You do not manage it, you use it, and the service provider manages it." 

Here is how we depict this at Mezeo:

 

I strongly believe that obtaining, using, and decommissioning persistent storage in a simple, easy way, available in any quantity on a pay-for-use basis, and accessible in a variety of ways, via the Internet or at the data center where your application runs, is the heart of the matter.  If you get that service in house, or from an IT service provider, it should include the aforementioned characteristics.  This is a very inclusive definition, and it provides for traditional access methods, as well as programmable access (Web services APIs). 

Finally, here are three more points that are very important:

1) By Web Services API access, we mean API access to stored content!  This is different from APIs for storage management and is specific to a way of working with stored content.

 2)  New applications, and retrofits, will ultimately expect "programmable" storage.  This is a classic "Innovators Dilemma" scenario, I see it every day, and it is coming. 

3)  HTTP access (Web services API or "programmable") is not slower than other access, but it does tolerate the latency of the Internet.  As a result, you will ultimately see that HTTP access of storage in a data center will be a preferred approach, because of it's "programmability" and the desired performance.  This will not happen overnight, but it will happen.

A hat-tip to Stephen Foskett is in order as well.  Take a look at this entertaining article in which he struggles to find an appropriate name for "cloud storage."

According to a recent Gartner press release, 20% of businesses will own no IT assets by 2012:

Several interrelated trends are driving the movement toward decreased IT hardware assets, such as virtualization, cloud-enabled services, and employees running personal desktops and notebook systems on corporate networks.

The need for computing hardware, either in a data center or on an employee’s desk, will not go away. However, if the ownership of hardware shifts to third parties, then there will be major shifts throughout every facet of the IT hardware industry. For example, enterprise IT budgets will either be shrunk or reallocated to more-strategic projects; enterprise IT staff will either be reduced or reskilled to meet new requirements, and/or hardware distribution will have to change radically to meet the requirements of the new IT hardware buying points.

This is a bold statement. If we believe Gartner, it means that we are at the beginning of an explosion in cloud-based services managed by trusted providers on behalf of the enterprise. Of course not all businesses will choose this path, but a substantial number of industries can and will. As I blogged about earlier, the message from the CFO office is clear. We will see adoption rates rise dramatically as the benefits of cloud services become more obvious to business leaders.

A second point of interest is the prediction that by 2012, India-centric IT services companies will represent 20 percent of the leading cloud aggregators in the market (through cloud service offerings).

Here’s the take-away:

Gartner is seeing India-centric IT services companies leveraging established market positions and levels of trust to explore nonlinear revenue growth models (which are not directly correlated to labor-based growth) and working on interesting research and development (R&D) efforts, especially in the area of cloud computing. The collective work from India-centric vendors represents an important segment of the market’s cloud aggregators, which will offer cloud-enabled outsourcing options (also known as cloud services).

We are witnessing examples of what GE innovation consultant Vijay Govindarajan calls reverse innovation in IT. Natarajan Chandrasekaran, the CEO of Tata Consultancy Services notes:

I’ve seen the new cloud-based computing models for applications and processes gaining currency in emerging markets. Rural cooperative banks and small and medium businesses in India are actually far ahead of their western counterparts in adopting these models. In fact, companies from emerging markets, buoyed by strong domestic revenues and revival in growth, have been making adjustments to their global strategies and fine-tuning their investments in order to be part of the recovery process in the west and build on their global expansion plans.

As the enterprise embraces the cloud, they’ll need a maturity model to help them on their journey. My next post will explore what the maturity model for cloud storage looks like. 

A recent report by Forrester's Andrew Reichman titled Business Users Are Not Ready For Cloud Storage: Current And Planned Adoption Of Storage-As-A-Service Is Minimal For Now paints a picture for cloud storage adoption, that at first blush, is not encouraging.

He states:

In Forrester's Enterprise And SMB Hardware Survey, North America And Europe, Q3 2009 survey, we asked businesses about their interest in "hosted storage capacity" offerings. Interest was minimal at best. Forty-three percent of all respondents said that they were simply not interested, and another 43% said that they were interested but had no plans to move forward.

While it could be argued that as a cloud storage supplier, I am necessarily bullish about the ultimate prospects, I believe the data is actually quite good and clearly represents what we are experiencing in the marketplace.  Now, Mezeo is engaged with many service providers, as well as the early adopters in the enterprise space as they begin their evaluations.

When I look at enterprise cloud-storage adoption based on Everett Rogers' diffusion curve I see a pretty clear view of the typical market place approach to adoption of disruptive technologies:    

For new, emerging, and potentially disruptive technologies, we should look for what the next practices are, i.e. the practices of the innovators and early adopters. The survey reflects the typical technology adoption cycle and re enforces what we are experiencing in the market place.

11% of companies are taking the plunge - these are the early adopters and innovators.  The early majority (43%) is interested, and watching.  The late majority is not in the game, yet.

So we are on track. And to prove it, let's look at one of these enterprise-level innovators: General Electric.

According to IBM storage expert Tony Pearson, GE has implemented cloud-based backups and archive for GE Corp, NBC Universal and GE Asset Management divisions running at only 32 cents per GB/month, representing a 40-60 percent savings over their previous methods. This includes backups of their external Web sites, archives of their digital and production assets, RMAN backups including development/staging databases. They plan to add out-of-region compliance archive in 2010. They also plan to monetize their intellectual property by offering "CloudStorage Manager" as a software offering for others.

We define hybrid cloud storage as utilization of private cloud storage at an enterprise data center, or a private cloud hosted by an IT service provider with some combination of additional IT service provider-based public and/or private cloud storage.  

In a recent post, Cloud Storage for the Enterprise - Part 1:  The Private Cloud, we covered the definition and requirements of cloud storage as an enterprise solution, and as a technology deployed within enterprise-owned data centers (or at least within their co- location racks and cages).  Fundamentally, a private cloud is also a non multi-tenant cloud (i.e., used by only one entity or related parties within an enterprise or a public sector agency) that is behind the firewall(s).  An additional solution that many enterprises are contemplating is the hybrid cloud, and we will look at the aspects of that solution in this post.  

Before we begin our investigation of hybrid cloud, let's review some of the basics.  The following diagram reviews the differences between public and private clouds:


Many enterprises are beginning their cloud evaluation with a "private cloud."  I extend the definition of private cloud to be a "single tenant" cloud, as some enterprises may chose to use a single tenant cloud hosted at a service provider, versus hosting their cloud within their own data centers.  In the following diagram, we show two private clouds, connected via policy-based replication in two data centers.  This provides the assurance of backup and disaster recovery that many enterprises require.  A third location could easily be added for even higher levels of backup and disaster recovery.


The growth of storage is driving increased costs, and the enterprise is on a continuous search to improve the way they can cost-effectively manage this growing data.  The primary difference between hybrid cloud and private cloud is the extension of service provider-oriented low cost cloud storage to the enterprise.  The service provider based cloud may be a private cloud (single tenant) or a public cloud (multi-tenant).  There are several implementations of hybrid cloud, and several examples are included.   The service provider cloud may enable enterprises to leverage the volume efficiencies of the service providers to realize additional savings. 

A hybrid cloud provides a way of securely using service provider-based cloud storage in combination with enterprise clouds.  Another implementation could be use of single tenant service provider-based private clouds at multiple locations. 

Some examples of hybrid clouds are offered for your consideration, although not every potential approach is covered herein:


Since the primary motivation for hybrid cloud is economics, let's begin the discussion with an understanding of the economics of cloud storage and then extend that discussion to the hybrid cloud environment. 

The primary cost components of cloud storage include:

1.    Data center occupancy - leased (co-location) or owned and depreciated.
2.    Data center environmental - utilities, cooling, heating, etc.
3.    Storage hardware (leased expense or capital requirements & associated depreciation).
4.    File system and storage management (may be bundled in the storage hardware).
5.    Cloud enablement or platform (discreet or bundled with the storage system).
6.    Systems management and operational overhead.
7.    Backup and disaster recovery.

While it can be argued that the economics at a large scale enterprise are very similar to those at a service provider, listed below are some of the most common reasons enterprises do turn to service providers for their technology solutions:

1.    Capital conservation.
2.    Distraction associated with infrastructure management.
3.    Desire to outsource functions that are required but not associated with core competency (focus dilution).
4.    Poor history of infrastructure management.
5.    Specific issues, for example, out of data center space and not projecting long term needs to add additional data centers, or unable to expand existing data centers and no desire for an additional site.
6.    Redundancy of networks available in data centers that may not be available in the enterprise with assuming additional costs.

Whatever the reason, service providers can solve these problems.  In each of the three hybrid cloud scenarios, there are costs and security tradeoffs that each cloud use-case will consider.  For example, in hybrid cloud variation #1, the economics can be quite appealing, but there are significant security concerns.  One approach to mitigate these concerns is to encrypting an object before replication to a public cloud might mitigate the threat.

Understanding where key functionality is applied in your cloud stack is critical for successful implementation and highly dependent on the cloud and storage subsystem technology, cloud interoperability capabilities, and data use case.  Critical technologies that provide benefits are: de-duplication, compression, encryption for data at rest and data in motion, geo location, geo replication, tagging and search capabilities, and cloud access methods.  I will address underlying cloud technology requirements for the enterprise in my next post.

Cloud Use Case Definitions:

Data Archiving - Storing data for retention management requirements (such requirements may be internally generated, or associated with regulatory and compliance needs).  Archive data must be highly secure, highly reliable over the archive period, and easily searchable.  Archive data is generally encrypted, compressed and stored in a proprietary format. Access to the data is usually very infrequent and thus typical enterprises have leveraged slower access, cheaper tape media or redundant NAS to control costs.  Typical data issues associated with archiving are maintaining the archive and eliminating what is known as bit rot of the data, which is where data becomes corrupt if stored in the same media for long periods of time and not accessed.

Data Backup - Storing data as a replacement copy in the event the original copy is somehow damaged or lost due to user error, system failure, or as a result of a disaster scenario.  Back up data may or may not need to be highly secure or easily searchable, but must be available for quick restore when needed.  This data is also generally encrypted, compressed and stored in a proprietary format. Access to the data is more frequent than with archive data and can be at any level of the organization.  A single file, user, server, site, or the entire enterprise could potentially need to be restored to proper service and backup data must support these highly variable access needs.

Data Access - Storing data in its original format for access by users or other applications.  This type of data is frequently accessed and is the superset of the data that comprise backup and archive data.  Access takes precedence over security, but needs to be easily and quickly searchable and retrievable by users and applications and thus highly available.  Typical issues with access data are the need for fast accessibility of frequently used data balanced against the overall cost associated with storing all the data.  Enterprises often implement tier strategies to stage data in progressively lower cost media based on frequency of access.


Hybrid cloud storage, which we have loosely defined as utilization of private cloud storage at an enterprise data center, or a private cloud hosted by an IT service provider with some combination of additional IT service provider-based public and/or private cloud storage, offers an approach that allows use case, economics and security to prevail when selecting the appropriate approach.  Implementation will also be driven by the technological capabilities of the three building blocks of cloud storage, the cloud abstraction layer, file/object system choice and storage subsystem hardware.

So, our discussion of hybrid cloud storage has likely demonstrated at least one significant additional aspect, and that is complexity.  Starting with use case definition and security requirements, combined with a clear understanding of the unique issues within each enterprise that effect cost, you can map a clear path to the cloud technology and selection of one or more cloud service providers.  Finally, the trusted service provider continues to be another significant requirement for exploitation of hybrid cloud.

1. Security will continue to be a big issue for the cloud, and, unfortunately, there will be at least one event this next year that is disruptive to Cloud Storage adoption, be it data loss or unauthorized data access.  Security will be an even more important point of evaluation for the use of specific Cloud Storage service offerings. The “trusted service provider“  becomes a requirement when selecting a cloud offering.
   
   
2. Cloud Storage will be characterized by a single word, “more”!  More adoption, more cloud storage offerings by more IT service providers, more variation in cloud capabilities, and more worries and concerns about the cloud.
   
   
3. The intersection of enhanced mobile devices with better wireless bandwidth will be combined with Cloud Storage to create exciting new work/life blended digital life applications. The user experience is of paramount importance.
   
   
4. Cloud Storage will see extraordinary adoption as a solution for backup, archiving and for policy-based georeplication for disaster recovery.

CloudStorageStrategy.com welcomes OpSource CEO Treb Ryan for an in-depth interview on cloud computing, from the perspective of the service provider.

NOTE: OpSource is a customer of Mezeo Software, the underwriter of this blog.

What are the opportunities you see in the cloud computing space, both for OpSource and your customers, and what impact has the downturn had on this?

It's interesting, but when people talk about cloud computing, they immediately go to the downturn and pricing - and cost being the big driver.  There's no question that cloud computing is cost effective, and it's accelerating adoption many times over, but what we're really seeing is something much more fundamental - a generation of users who are entering the workforce who've been using cloud computing all along; they've grown up on the Internet, and their interface to technology has always been through the Internet. 

As a result, this "Cloud Generation" has clear expectations of how technology should work:

1) it should be immediately available,
2) you do a search and get going,
3) it should be very flexible,
4) you should have ubiquitous access - anytime, anywhere,
5) sharing and collaboration - the expectation to collaborate and share anything they are working on.

This is not a generation which distinguishes between work data and home data - like my generation did. They've grown up with the concept of APIs and communities that grow around them; for instance, we see programmers who have grown up with Google and Facebook APIs, and now they expect that kind of thing in their work applications as well. So they're coming into the workforce and driving change in the workplace. They see technologies like client-server applications or hard-coded storage arrays pretty much the same way my generation saw green screens, mainframes, and mini-computers - as dated, inflexible, technology - hard to use, without nearly the power of cloud-based systems. So they have the day-to-day experience of the "consumer cloud" which they're now driving into business applications as well. 

To the Cloud Generation of programmers this means anything they can interact with on the Cloud they can program to through APIs. The idea of infrastructure being an item that can be addressed as part of the application, instead of something the application lays on top of, is a radical concept.  It has allowed not only for innovative applications, but also for true elastic computing making the Cloud environment even more flexible.



Great Cloud offerings have great communities around them. This is the aspect of Cloud computing that is so often missed - and even scoffed at - by the IT folks who think it's all about virtualization. One of the biggest gripes about Cloud computing is that support is done by the Community and not the vendor. While most will agree that far more proactive vendor support is necessary for Cloud computing, Community support is just as critical. For questions of configuration and usage tricks, the Community is a far better source of information than some call center employee with limited access. Often the Community devises more innovative solutions than the vendor ever could. And in addition to support, the Community can create third-party add-ins that make the Cloud even more useful.

The downturn has accelerated adoption from the top down as well.

We're seeing executives who have become enamored with this idea of the cloud - because of the ability to turn capital expenditures into operational expenses - and are pushing cloud computing into their organizations.  The CEO of one of our customers went so far as to tell his technical people - "now can you finally start using the cloud so I can get the board off my back?"

So, for different reasons, we have both top-down and grass-roots support for cloud-based applications, which makes this very interesting to say the least.

Which customer segments do you see leading the way in adoption?

Obviously, our traditional focus has been on ISVs and start-ups coming into Software-as-a-Service, business applications in the cloud, and we're seeing continued adoption of cloud infrastructure by those segments, but what has been interesting is that now that we offer the ability for any company to buy and use cloud infrastructure for any type of application, we're seeing a much broader spread of usage and adoption. Beyond the enterprise we also see widespread adoption by systems integrators, consultants, and VARs - upto 40% of our customer base - all without us targeting that segment at all.

How does OpSource differentiate its cloud offerings from other service providers?

We offer the best of the public cloud, combined with enterpise security and compliance, performance guarantees, and enterprise controls.

For instance, we offer:

• easy online sign-up & purchase with infrastructure provisioning in minutes
• pay by the hour and only for what you use, with no commitment (or purchase a monthly plan for a discount)
• a rich online community to share and collaborate with peers; get third party add-ins, images and configurations
• a web interface plus complete set of APIs

On the straight cloud, we provide a lot of the more robust, enterprise tools than you see from more consumer-based providers like Amazon, for example.

We focus on three different areas:

1) Security and Compliance: we provide a much more secure environment, because Opsource provides every customer with a Virtual Private Cloud within the public Cloud, allowing them to determine their own degree of public Internet connectivity. We also provide:

• Unique customizable security for firewalls
• VPN administration of all servers
• Unique username/password for each administrator
• Audit logs of all environmental changes
• SAS 70 audited
• 100% uptime SLA

2) Performance: we offer a multi-tier architecture with guaranteed latency in-between systems, sub-millisecond access time, industry standard technology, like VMware, instead of open-source, because that's where enterprise is comfortable.  Our 24/7 suppot also makes a diffence.

3) Control: today's cloud environment are single user environments, one user name and password, which is fine for individuals, but not so useful for the enterprise. We offer the ability to provision multiple users, do things like cross departmental billing, execute policy based control - which user can do what - and finally link all that back though an API to your existing management systems. So you can control how your users use the cloud same as you do your corporate datacenter.

So do you see any links into these large companies where they need to use ITIL for systems management?

Absolutely. OpSource has always focused on compliance as a major issue for our SaaS customers, eveything from SAS 70, PCI to European Safe Harbor, and even industry-specific ones like HIPAA, or government-specific certification, but in the cloud, we think about sophisticated  management techniques like federated authority and single sign-ons, and things like ITIL - while it's still in its infancy, it's shocking that most providers don't even have the ability to give their customers the critical capability to have more than one person manage the cloud for them - because they have a single user accounts. So while you can institute more sophisticated IT governance regimes like ITIL with the OpSource cloud, we give IT the capability to manage who does what, and track who did what, even if they aren't ready for something like ITIL.

So IT gets to do their own provisioning?   
  
Yes. So you want to know who provisioned what, how much it costs, and we give them that visibility instantly across their entire user community.  That way there are no surprises or charges they aren't aware of. It sort of reminds me of the controls I had to put in to alert me to my daughter's texting costs - so I'm aware of the charges before they get out of hand! I just blogged about this issue.

That's why you say that OpSource is what Amazon wants to be when it grows up... 

Absolutely.

And that's how you respond to cloud critics - the ones that say that the Cloud is not yet ready for the enterprise.

There are large parts of the cloud that are not yet ready for the enterprise. The cloud is still young, and it would be like asking that first 286 PC to run all of your corporate financials. However, a lot of these issues around enterprise adoption like security and compliance have been addressed, and are being taken care of, so as the cloud becomes more robust, we'll see increased adoption. We're seeing enterprise-level capabilities come to market that did not even exist six months ago.

We have just signed a partnership agreement under which OpSource will resell Gomez's Web performance management solution to our enterprise customers as well as use it to validate and monitor our own cloud performance service level agreements (SLAs). Through this partnership, we'll bring powerful performance monitoring to cloud computing, making it easier and more compelling than ever for enterprises to justify bringing their applications to the cloud.

Do you see infrastructure elements like storage growing now?

For true, full use of the cloud, we have to have the ability to access storage, go though the APIs to get to it, and give our customers a range of storage solutions, including cloud storage based on the specific application or need. We're giving our customers the widest range of choices.

What about agile programming? I heard you use agile methods to improve the customer experience.

Agile programming methods have helped us with not only development, but compliance and security as well. We talk to our customers to see how they are using our cloud offerings though our community, and we learn what's important to them.

We also test our offerings by having two programmers work on the same keyboard - literally  - one with the user story - so they can make sure that the customer is getting the exact functionality they need.

It's agile customer service.

Can you tell us a bit about your enthusiasm for composite applications (corporate mashups) and how they help your platform?

Of all the phenomenon in the cloud, we see the need for anytime-anywhere access and the idea that anything I can interact with I should also be able to program to.  So when Facebook enthusiasts start working in the enteprise, they bring their enthusiasm for integration as well.

So we see things in the cloud like direct access to the infrastructure as part of the application, which allows for all sorts of flexibility and robust usage.

We see real-time reporting applications of every kind you can imagine.  I myself am addicted to checking on everything that's coming out of our billing and customer systems tied into our Salesforce tabs.  So I'm always checking on the business in real-time via my iPhone.

I say this a lot, but integrating SaaS is a huge issue for today's enterprise. OpSource Connect can help SaaS companies -- of any size -- overcome integration hurdles and break out of the SaaS-only box. This speeds up adoption of SaaS in larger enterprise environments, opening the door for on-demand companies to cultivate business with large systems integrators. Plus, I'd say we're the only company providing Web operations from the ground up, addressing operational infrastructure, application management, and business operations. Today, integrations are expensive and one-to-one. For instance, while you can currently integrate your application with Google Maps as a composite application, OpSource Connect lets you integrate your app with many others, using just one platform. You can integrate your application with, for example, SAP, salesforce.com, Intuit QuickBooks, NetSuite, and a host of other SaaS and legacy applications. 

Everything is much more dynamic today, and programmers expect that. 

As the industry announcements on Cloud Storage APIs keep coming, the confusion surrounding what they mean keeps growing.

We have the Amazon S3 APIs, Eucalyptus APIs, Rackspace Cloud Files APIs, Mezeo APIs, Nivanix APIs, Simple Cloud API, along with the standards proposed by the Storage Networking Industry Association (SNIA) Cloud Storage Technical Work Group, and more. 

So what should you do or think about all this? What impact do these Cloud Storage APIs have on your decision-making? Just how important are they, and what's next?

Here's some information to aid your understanding of this emerging and important technology.  Let's begin by answering two basic questions: 

What is a Cloud Storage Application Programming Interface (API)?
    
A Cloud Storage Application Programming Interface (API) a method for access to and utilization of a cloud storage system.  The most common of these are REST (REpresentational State Transfer) although there are others, which are based on SOAP (Simple Object Access Protocol).  All of these are associated with establishing requests for service via the Internet. 

What is REST? 
REST is a concept introduced in the doctoral dissertation of Roy Fielding, and is widely recognized as an approach to "quality" scalable API design.  The actual API design and capabilities are very dependent on the actual capabilities of the underlying Cloud Storage System

One of the most important REST capabilities is that it is a "stateless" architecture.  This means that everything needed to complete the request to the storage cloud is contained in the request, so that a session between the requestor and the storage cloud is not required.  Why is this important?  The Internet is highly latent (it has an unpredictable response time and it is generally not particularly fast (when compared to a local area network (lan)).  Once you get a request, there is no guarantee that you can ask a "qualifying question" of the requestor in a reasonable time period.  So, REST is an approach that has very high affinity to the way the Internet works.  Traditional file storage access methods that use NFS (network files system) or CIFS (Common Internet File System) do not work over the Internet, because of latency.

One other thing we should clear up:  Cloud Storage is for files, which some refer to as objects, and others call unstructured data.  Think about the "files" stored on your PC, like pictures, spreadsheets and documents.  These have an extraordinary variability, thus "unstructured".  The other kind of data is "block" or "structured" data.  Think data base data, data that feeds transactional system that require a certain "guaranteed" or low-latency performance.  Cloud Storage is not for this use case.  IDC estimates that approximately 70% of the machine stored data in the world is unstructured, and this is also the fastest growing data type.

So, Cloud Storage is storage for files that is easily accessed via the Internet.  This does not mean you cannot access Cloud Storage on a private network or LAN, which may also provide access to a storage cloud by other approaches, like NFS or CIFS.  It does mean that the primary and preferred access is by a REST API.  (Here are other terms you will see, RESTful, or RESTlike or RESTstyle, which is geekspeak for how closely the API conforms to the REST approach.) 

Today, there are multiple definitions for Cloud Storage, and the one I prefer is "File Storage accessed through Web Services API's over a network".  This represents the key attributes of file storage that is cloud storage, versus other types of file storage.  Other key qualities of a storage cloud are:

• multi-tenant support (use by more than one unrelated user)
• geo location and geo replication, seamless and real time provisioning of accounts
• seamless and real time provisioning of accounts
  
• availability of "practically" unlimited amounts of storage "on-demand"
• "pay for use", which means that your payment is for actual storage used, over some time frame, usually a month. 
  

There are many who are still arguing about what I have defined above, but what I've said is generally accepted by the industry.  If it is a vendor doing the arguing I would suggest you check under their hood, usually you will find that they do not offer whichever of the above features they are trying to argue out of the definition.

Also, traditional storage vendors continue to proclaim the importance of local network access (like NFS, CIFS or ISCSI) for the purpose of Cloud Storage access by applications that today can only access via the older protocols.   This requires that the application making the request be on the same local network (think same data center) as the storage cloud.  Their reason for this view is that they are only just beginning to see application demand for storage cloud access via REST APIs, versus their traditional business model which serves an enterprise user with their own data center. 

This is why Cloud Storage has generally emerged as a service offering in the IT Service Provider  (also know as the WEB Hosting Industry) space first.  In this space, there is no doubting the importance and future of REST API access to storage clouds, it is only viewed as an adoption speed issue.  Note that within the data center, access to storage using an HTTP based protocol is not necessarily any slower than one of the more traditional protocols. API access has been labeled as being a slower form of access over NFS and CIFS. This view is largely due to the fact that it "may" be accessed over the Internet. In most cases, it is the network that adds the latency, not the means of access. Make no mistake, traditional storage vendors see this coming, and they will make offerings available in the near future.

REST APIs are language neutral and therefore can be leveraged, very easily, by developers using any development language they choose. Resources within the system may be acted on through a URL. So, an API is not a "programming language" it is the way a programming language is used to access a storage cloud.  This is part of the basic understanding of APIs that is required to discuss the dreaded "vendor lock in" and upcoming "cloud lock in" discussions and understand the issues that surround these assertions.

REST APIs are also about changing the state of resource through representations of those resources. They are not about calling web service methods in a functional sense. The key differences between different Cloud Storage APIs are the URLs defining the resources and the format of the representations.
 
The Cloud Storage space is very young and everyone has their opinions on how things should be represented and accessed. Efforts are underway by organizations like SNIA, with their Cloud Data Management Interface (CDMI), to standardize both the resource structure and the representations. However, standards are not developed overnight and customers are demanding programmatic access to Cloud Storage now.

Current Cloud Storage vendors have produced a basic set of APIs that are accomplishing fairly similar things, and other APIs that expose the underlying unique functionality of the Cloud Storage platform supplying the storage cloud.  You should expect that, over time, most storage clouds will provide the basic functions in somewhat similar ways, and further that additional advanced functions will be adopted and expected to be in every storage cloud offering. 

Finally, you should look for a taxonomy of APIs, that includes basic file functions, advanced functions, Provisioning APIs, Billing APIs, and Management APIs.  Storage clouds that become successful will offer all these capabilities, to increase the efficiency of their use.



 
Several efforts have been made to simplify the transition between vendors by providing an abstraction layer on top of the vendor's APIs. In this approach, a program library is created, for use in the application that needs cloud storage access, and this API translates (for the given program language) a single API into the API that is specific to a Cloud Storage offering.  So, the application, which is using this library, writes their APIs once, and achieves portability between storage clouds that are supported by this approach.

This approach has been largely programming language specific and may take advantage of the language it was designed for. Good examples of this are jClouds, an open source cloud storage abstraction library written in Java, and Simple Cloud API, a collaboration of vendors including Microsoft, Rackspace, Nirvanix, IBM and Zend which provides a simplified Cloud Storage interface for PHP developers. While extremely useful for developers, these abstractions tend to expose the lowest common denominator relating to Cloud Storage functionality and may omit critical features, for example only providing namespace object access as opposed to ID access.

So, let's discuss lock-in, the term used to express concern that once a vendor has gotten you to exploit their architecture and technology, they will recognize that you are committed to them and cannot easily move away.  As a result, they will then raise their prices and take advantage of your lock in status, keeping their price just below the amount that would encourage conversion away from their technology and towards a more "open" set of capabilities.  Let's look at all the "dreaded" examples that have been surfaced around cloud storage and as a reason to slow it's adoption:

1.    API lock in, which means your interaction with a storage cloud uses the APIs of that storage cloud, and suggests that you cannot easily move to another providers cloud with their own, different APIs.

2.    Vendor lock in, which means that since you are condemned because of your application development activity with specific APIs to use only a cloud from a specific supplier.

3.    Device lock in, meaning that you developed a cloud storage based program utilizing the APIs of that specific cloud, for a specific device (generally a PDA) that has specific functionality.  This is double lock in, both the device programming methodology and the API selection.

4.    Browser lock in, meaning that programming to specific APIs can also be rendered unique based on the Web browser that is selected.

5.    Programming language lock in, which means that you have written the APIs in a language like Python, or JAVA, or .NET, or whatever.

6.    API wrapper lock in, which means that you incorporated libraries into your application that allows your application to write generic APIs, which are then translated by these APIs to the correct API for the desired storage cloud (this is what Simple Cloud API is).

So, as you can see here, utilizing cloud storage could ultimately have you locked in on at least six levels! 

With this much opportunity for vendor abuse, why are developers rushing to write Web based applications that utilize cloud storage services via API access?  Are they simply uncontrolled, unthinking rebels who will shortly learn the error of their ways?  Have they made a fatal error?  Or do they know something you don't?

First, learn about Cloud Storage APIs.  What they do is make storage programmable, and they abstract storage from the application.  They offer advanced functionality (the programmable word) that makes it faster and easier to write the applications that are scalable versus the traditional storage access approaches.  When you add these two capabilities to the storage cloud offering of low cost, availability in multiple locations, seamless provisioning, ease of adding additional storage, and the pay for use model, the case for the cloud has become compelling.

Where are we seeing early adoption:  at service providers, because they host Web based applications and SaaS (usually Web based) applications, and this is where the developers who recognize the opportunity are focused. 

What is coming: the introduction of this technology into the enterprise, complete with the adoption of the RESTful API technology.  This will ultimately lead to a level of cooperation between service providers and the enterprise that has long been predicted.  Enterprises will move to an IT modeled on an OPEX model, and expect their applications to be provisioned and interacting with service provider clouds, via APIs.  IT Service Providers are racing to build the clouds to provide for this emerging business opportunity.

So, what about the lock in mentioned above.  Sit down with your developer, they will show you why they don't feel "locked in".  They will show you that you can quickly recraft your current APIs, in the programming language of your choice, to utilize the new APIs of the desired cloud.  For this reason, Simple Cloud API will likely be a short term measure, which precedes base case APIs that are extremely similar, and goes through a market led process to identify "best practice" APIs for both base case and advanced function, as well as all the other API led capabilities as mentioned above.  In short, vendor lock in is not the problem for this technology that it has been for others.  Also, the ingenuity and resourcefulness of all the suppliers, standards groups, and market adoption scenarios will continue to mute your ability to be lock in free. 

Your real challenge is not lock -in, but rather how to adopt this new set of capabilities, and solve problems and create opportunities with your IT solutions as rapidly as possible.  Standing on the sidelines waiting for this one to resolve will keep you out of a great opportunity, because we still have several meaningful years of rapid change associated with this technology adoption cycle.