Recently in The Personal Cloud Category

The definition of cloud storage has been on my mind lately, and I think some attention to this topic is still called for.  From an article in CXO Today:  Cloud storage is not a disk array that you own, lease, or manage neither is it a virtual logical unit number (LUN) from a larger disk array. It is in fact it is offered via an application programming interface (API) through which you can send and receive data without having to actively manage the storage.

I see many "Cloud Storage" services and vendors of cloud storage infrastructure products that do not do or provide for what is described in the preceding observation.  For example, some cloud storage services are really offerings of storage that are associated with cloud computing.  One major requirement is storing and retrieving cloud computing images.  Since these are "bootable" the typical storage approach is an iSCSI connected storage resource.  A cloud computing image may require files for its application, and these are often stored on shared storage systems, and may be accessed in a variety of ways, but not necessarily via Web services APIs. 

Often, IT service providers call this shared storage; however, when it is accessed by cloud computing images, it is often referred to as cloud storage.  Finally, block data, like  a data base, is often required for the application running on a cloud computing image, and is accessed via iSCSI, and may be referred to as cloud storage. 

So, where do these observations lead us?

There are many benefits of a storage cloud, and for the user these include ease of access (in a variety of ways) to various amounts of storage on an as needed basis, with instant or nearly instant provisioning, with little if any traditional storage management requirements for the user.  IT service providers, and enterprise IT organizations are fundamentally organized around the premise of service delivery.  So, for both of these entities, a service offering like cloud storage is an important business asset, and the primary differences in the deployed infrastructure is associated with multi-tenancy (which, among other things, drives different security requirements) and billing. 

For many months, I have relied on the following definition of cloud storage: a persistent storage solution for objects (also called files or unstructured data) accessed via Web services APIs via a network (LAN or WAN). 

Today, I would like to move forward and offer a new definition, more encompassing, and reflecting not just a purist view but attempting to capture what is truly important for an IT service provider, in house or as a focus of a business (hosters, telcos, and cloud providers): 

"Cloud Storage provides whatever amount of storage you require, on an immediate basis.  It is persistent.  It can be accessed in a variety of ways, both in the data center where the cloud is housed, as well as via the Internet.  If you obtain this from an external provider, it is purchased on a pay as you go basis.  You do not manage it, you use it, and the service provider manages it." 

Here is how we depict this at Mezeo:

 

I strongly believe that obtaining, using, and decommissioning persistent storage in a simple, easy way, available in any quantity on a pay-for-use basis, and accessible in a variety of ways, via the Internet or at the data center where your application runs, is the heart of the matter.  If you get that service in house, or from an IT service provider, it should include the aforementioned characteristics.  This is a very inclusive definition, and it provides for traditional access methods, as well as programmable access (Web services APIs). 

Finally, here are three more points that are very important:

1) By Web Services API access, we mean API access to stored content!  This is different from APIs for storage management and is specific to a way of working with stored content.

 2)  New applications, and retrofits, will ultimately expect "programmable" storage.  This is a classic "Innovators Dilemma" scenario, I see it every day, and it is coming. 

3)  HTTP access (Web services API or "programmable") is not slower than other access, but it does tolerate the latency of the Internet.  As a result, you will ultimately see that HTTP access of storage in a data center will be a preferred approach, because of it's "programmability" and the desired performance.  This will not happen overnight, but it will happen.

A hat-tip to Stephen Foskett is in order as well.  Take a look at this entertaining article in which he struggles to find an appropriate name for "cloud storage."

We see a lot of coverage about cloud storage these days - and why it is or is not being adopted. One way to look at cloud storage adoption is to view it as an evolutionary process which changes over time, as both the organization matures and becomes adept at leveraging the new technology, and as the technology itself evolves to meet the real needs of the end-user.  The common name for this sort of thinking is a "maturity model."

With that in mind we developed this simple maturity model for cloud storage, based on the actual cloud storage adoption process we're witnessing in the industry. We'd like to hear your thoughts - are you seeing the same trends?


PHASE ONE: Public Cloud Storage

Description
There remains significant marketplace confusion about what constitutes cloud storage.  Cloud storage is a persistent storage for unstructured data accessed via Web services APIs over a network (LAN or WAN), with the additional  characteristics of rapid provisioning of both new accounts of any size as well as rapid provisioning of increases (or decreases) in account size, along with a pay for use model, Some believe that cloud storage is just the provisioning and pay for use model with access method being varied between older technologies (CIFS/NFS) and http (Web services API access).  Public, multi-tenant storage clouds as delivered by service providers clearly meet our definition, as traditional access methods like CIFS/NFS are not useful over the Internet.

Many technologists and almost all non technologists, make the initial mistake that cloud storage is simply the storage used when using cloud computing.  In fact, a cloud computing image (CCI) may very well be provisioned and stored when not in use on traditional iscsi type storage systems, and is often dependent on very high speed access associated with a locally attached device.  Many times, the data needed for the application supported by the CCI is often stored on shared storage devices within the same data center as the CCI, for application performance reasons.  The data for these CCIs may also be block, or data base data.  This is storage for cloud computing, but it is not "Cloud Storage"!  This confusion permeates the marketplace in Phase One.  Many vendors, particularly traditional storage vendors, have confused the marketplace by claiming to be cloud storage based on "thin provisioning" attributes with traditional data center access versus HTTP access. Cloud storage may also be accessed and utilized by CCI based applications, but that is not a defining attribute of cloud storage.  Cloud storage is accessed by applications on both CCIs and dedicated servers, as well as clients on PDA's and PC's, wherever they are and whenever they need access.  The use cases are very tolerant of the latency associated with the Internet. The thin provisioning and pay for use model of cloud storage does deliver the important cloud storage attribute of transferring storage costs from a CAPEX to an OPEX basis, if you are acquiring your cloud storage form a service provider on a pay for use basis.

 The IT service provider space is the earliest adopter of cloud storage, for both offensive and defensive purposes.  Many service providers are hosting workloads on dedicated or virtual servers (CCIs), and the workloads are new applications that utilize cloud storage from companies like Amazon S3, Rackspace Cloud Files, Nirvanix, and SoftLayer CloudLayer. Since the amount of data can be very large, it is difficult to move without downtime. And since the processing is relatively easy to move, IT service providers recognize the need for their own cloud storage service in order to provide a complete offering to their customers and to promote retention.  Without the associated cloud storage, the application server workload can easily move, usually to the provider who provides the storage cloud.  This is the defensive argument for service providers to offer their own storage cloud.  On the offensive side, cloud storage is growing rapidly in terms of adoption, provides a new revenue stream, can attract new hosted workloads (cloud or otherwise), and drives increased (and very profitable) bandwidth use.

The web hosting industry also saw the initial development activities associated with adoption of Web services APIs, which provide many programming capabilities that are now resident in the storage, and easily enabled new applications that are delivered via the Web.  These services, including tagging, searching and filtering, sharing, publishing, and collaboration, all exist within the APIs of a storage cloud, and are easily implemented within the application.  While the enterprise has not yet adopted this new functionality, it has become quite pervasive within social networking apps, enabling new apps on mobile devices, file sharing services, and online file services, and backup and archive services.

Cloud storage is currently offered by only a few service providers including Amazon (S3); SoftLayer (CloudLayer); Rackspace (CloudFiles), Nirvanix, and is only available as a service.  Enterprise adoption is limited to development only, primarily testing, and enterprise adoption has not yet occurred, primarily because of security concerns.

Key attributes

Adoption Drivers:
Business drivers: low cost, rapid scalability and on-demand capacity
Technology enablers: New programming capabilities

Adopters:
- SMBs/ SMEs
- Developers
- Consumers

Use Cases:  
- Testing and application development
- SaaS (Consumer & SME/SMB users: Backup, file sharing, additional device storage, rich media)

Differentiators:  
- SLA variability
- Pricing elements

PHASE TWO: Public & Private Cloud Storage

Description: As large enterprises start to fully comprehend the benefits of cloud storage, their interest grows.  While security concerns keep them from adopting the public cloud, they begin building private clouds behind their firewall. A private cloud provides them with the level of control and security that they are comfortable with and improves the utilization rates of their existing storage infrastructure, because of thin provisioning and potential for technology reuse. Enterprises start to roll out advanced capabilities such as file sharing and collaboration to their employees and their partners. The initial use of storage cloud services allow the enterprise to begin initial development of storage cloud based applications.  They also start to move backup and archives into their  own clouds. Since these applications do not require the highest performing storage, enterprises are able to reuse decommissioned hardware. This effectively starts the process of "tiered storage." 

At the same time, the public cloud storage offerings continue to grow.  The availability of deployable solutions to create your own storage cloud begin to arrive in the market, enabling IT service providers to quickly implement storage clouds versus being faced with a roll your own development effort.  Public storage cloud service offerings become more pervasive and better accepted as security and awareness increases.

Key attributes (Private Cloud Storage)

Adoption Drivers:
Business drivers: low cost, rapid scalability, high security and control
Technology enablers: new programming capabilities, cloud gateways (such as Blue Thread, Entropy)

Adopters:
- Enterprises

Use Cases: 
- Application Development
- Testing
- Backup
- Archiving
- File Sharing and Collaboration

Key attributes (Public Cloud Storage)

Adoption Drivers:
Business drivers: Low cost, rapid scalability, on-demand capacity
Technology enablers: new programming capabilities, cloud gateways generating multi-cloud usage

Adopters:
- SMBs/SMEs
- Developers
- Consumers
- Enterprise Evaluators

Use Cases: 
- Testing and application development
- Backup
- SaaS (Consumer & SME/SMB users: Backup, file sharing, additional device storage, rich media)
- Personal cloud storage with access clients
- Backup and archiving using cloud gateway
- Special use cases enabled by cloud gateway
- File server replacement
- Availability of CIFS/NFS access within the data center

Differentiators: 
- SLA variability
- Pricing
- Scalability and performance
- Access options

PHASE THREE: Public, Private and Hybrid Cloud Storage

Description: The maturity of the cloud (both private and public) has enabled many new applications which now require all the advanced services of a storage cloud (Web services API access, tagging, search, sharing, collaboration, etc).  Capabilities such as Geo Access (accessing files from a repository closest to the requester) and Geo Replication (policy driven replication across geographies to facilitate disaster recovery) are realized.  As Internet latency is constantly improving, more and more applications become "cloudy" in terms of storage, and cloud location becomes slightly less important as associated with performance.  Cloud storage is now a requirement of developers and development platforms.  Most SaaS applications also expect the availability of cloud storage.  Everyone is storing everything!  Most importantly, the improved security in public storage cloud offerings begins to blur the distinction of importance of security as being where data is stored (in public or private clouds).  Instead, applications utilize both public and private clouds, for reasons associated with location of data, disaster recovery and backup, and CAPEX versus OPEX.   Only the most sensitive data still retains a private cloud requirement.  Performance is a more salient driver of where the data is stored, does it need to be on a LAN in the same data center as the application?

This use of both public and private clouds as solutions for storage, often by the same application, becomes what we refer to as the Hybrid Cloud.

Key attributes (Private Cloud Storage)

Adoption Drivers:
Business drivers: Low cost, high security and control, rapid scalability, compliance and forensics
Technology enablers: New programming capabilities, cloud gateways
 
Adopters:
- Enterprises

Use Cases:  
- Application development
- Backup
- Archiving
- File sharing and collaboration
- Geo access

Key attributes (Public Cloud Storage)

Adoption Drivers:
Business drivers: low cost, rapid scalability, on-demand capacity, clouds become more pervasive
Technology enablers: new programming capabilities, cloud gateways generating multi-cloud usage

Adopters:
- SMBs/ SMEs
- Developers
- Consumers
- Enterprise evaluators

Use Cases:  
- Testing and application development
- SaaS (Consumer & SME/SMB users: Backup, file sharing, additional device storage, rich media)
- Personal cloud storage with access clients
- Backup and archiving using cloud gateway
- Special use cases enabled by cloud gateway
- File server replacement
- Availability of CIFS/NFS access within the data center

Differentiators:
- SLA variability
- Pricing elements
- Scalability and performance
- Access options
- Multiple clouds vs. single cloud

Key attributes (Hybrid Cloud Storage - a mix of Public and Private Cloud Storage)

Adoption Drivers:
Business drivers: lowered average cost obtained via a mix of public/private cloud, reduction of DR/BC costs, optimized mix of capex and opex
Technology enablers: improved security

Adopters:
- Enterprises

Use Cases:  
-  Incorporates use cases for private and public clouds

Differentiators:
-  SLA variability
-  Pricing elements
-  Scalability and performance
-  Access options
-  Multiple cloud vs. single cloud

PHASE FOUR: Federated Cloud Storage

Description: With the advent of greater security, flexibility and interactivity, users will demand applications that provide real time dynamic interaction within their supply chain. Regardless of where their data may reside, partners, customers, employees and consumers will want a seamless, transparent access capability. Enter the Federated Cloud. Through a common management layer, Federated cloud will connect private and public clouds exposing all storage as a single name space. Through federated identity management and creation of trust relationships amongst various vendors and enterprises, authorized users (human or programmatic) will be able to authenticate to their cloud and be able to access information that resides anywhere across the globe. Excess capacity will be easily pushed over a grid and be sold and consumed as a true utility. Ultra-high utilization rates will be achieved, and within the trust circle security and compliance requirements will be defined and met. Interoperability will be ensured by continued maturity and standardization of APIs and applications.

This truly will culminate in a meaningful internet of knowledge and commerce.  The "Semantic Web" has arrived!  Note that, for matters of very high security, agencies and enterprises will continue to use private clouds.

Key attributes (Federated Cloud Storage)

Adoption Drivers:
Business drivers: need for real time dynamic interaction with partners/customers on different clouds, ability to sell excess capacity within the trust circle, optimized infrastructure utilization, establishment of trust relationships
Technology drivers: federated authentication and provisioning across clouds, streamlined cross-cloud management, standardized APIs  
'
Adopters:
- Service providers
- SMEs/SMBs
- Consumers
- Enterprises

Use Cases:  
- Supply chain management
- Ad-hoc capacity capacity enhancement
- Non-sensitive and sensitive data hosting

Differentiators:
- SLA variability
- Pricing elements
- Scalability and performance
- Access options
- Security
- Governance and regulation compliance

Based upon our experience in the marketplace, a large majority of the organizations are still in the first two phases. There is an undeniable appetite by the early adopters to be at the forefront, however, unlike many other emergent technologies, cloud storage comes equipped with a very compelling economic model and that is really helping justify the move into the cloud.

There are relatively few options for early adopters to implement private clouds that deliver the appropriate capabilities.  This is why Mezeo focused on a deployable platform versus only offering cloud storage as a service.  With the deployable platform, enterprises can implement their own in house cloud, and also take advantage of a "private" cloud hosted on their behalf at a service provider.  See my discussion of this topic in my post: Cloud Storage for the Enterprise - Part 2: The Hybrid Cloud

In summary, those of us who hail from the IT service provider industry are very comfortable with cloud storage.  We see the adoption as proceeding, and the issues are being knocked off as they arise.  We are in an early technology cycle but with innovative early adopters we see a bright future.

A recent report by Forrester's Andrew Reichman titled Business Users Are Not Ready For Cloud Storage: Current And Planned Adoption Of Storage-As-A-Service Is Minimal For Now paints a picture for cloud storage adoption, that at first blush, is not encouraging.

He states:

In Forrester's Enterprise And SMB Hardware Survey, North America And Europe, Q3 2009 survey, we asked businesses about their interest in "hosted storage capacity" offerings. Interest was minimal at best. Forty-three percent of all respondents said that they were simply not interested, and another 43% said that they were interested but had no plans to move forward.

While it could be argued that as a cloud storage supplier, I am necessarily bullish about the ultimate prospects, I believe the data is actually quite good and clearly represents what we are experiencing in the marketplace.  Now, Mezeo is engaged with many service providers, as well as the early adopters in the enterprise space as they begin their evaluations.

When I look at enterprise cloud-storage adoption based on Everett Rogers' diffusion curve I see a pretty clear view of the typical market place approach to adoption of disruptive technologies:    

For new, emerging, and potentially disruptive technologies, we should look for what the next practices are, i.e. the practices of the innovators and early adopters. The survey reflects the typical technology adoption cycle and re enforces what we are experiencing in the market place.

11% of companies are taking the plunge - these are the early adopters and innovators.  The early majority (43%) is interested, and watching.  The late majority is not in the game, yet.

So we are on track. And to prove it, let's look at one of these enterprise-level innovators: General Electric.

According to IBM storage expert Tony Pearson, GE has implemented cloud-based backups and archive for GE Corp, NBC Universal and GE Asset Management divisions running at only 32 cents per GB/month, representing a 40-60 percent savings over their previous methods. This includes backups of their external Web sites, archives of their digital and production assets, RMAN backups including development/staging databases. They plan to add out-of-region compliance archive in 2010. They also plan to monetize their intellectual property by offering "CloudStorage Manager" as a software offering for others.

As the industry announcements on Cloud Storage APIs keep coming, the confusion surrounding what they mean keeps growing.

We have the Amazon S3 APIs, Eucalyptus APIs, Rackspace Cloud Files APIs, Mezeo APIs, Nivanix APIs, Simple Cloud API, along with the standards proposed by the Storage Networking Industry Association (SNIA) Cloud Storage Technical Work Group, and more. 

So what should you do or think about all this? What impact do these Cloud Storage APIs have on your decision-making? Just how important are they, and what's next?

Here's some information to aid your understanding of this emerging and important technology.  Let's begin by answering two basic questions: 

What is a Cloud Storage Application Programming Interface (API)?
    
A Cloud Storage Application Programming Interface (API) a method for access to and utilization of a cloud storage system.  The most common of these are REST (REpresentational State Transfer) although there are others, which are based on SOAP (Simple Object Access Protocol).  All of these are associated with establishing requests for service via the Internet. 

What is REST? 
REST is a concept introduced in the doctoral dissertation of Roy Fielding, and is widely recognized as an approach to "quality" scalable API design.  The actual API design and capabilities are very dependent on the actual capabilities of the underlying Cloud Storage System

One of the most important REST capabilities is that it is a "stateless" architecture.  This means that everything needed to complete the request to the storage cloud is contained in the request, so that a session between the requestor and the storage cloud is not required.  Why is this important?  The Internet is highly latent (it has an unpredictable response time and it is generally not particularly fast (when compared to a local area network (lan)).  Once you get a request, there is no guarantee that you can ask a "qualifying question" of the requestor in a reasonable time period.  So, REST is an approach that has very high affinity to the way the Internet works.  Traditional file storage access methods that use NFS (network files system) or CIFS (Common Internet File System) do not work over the Internet, because of latency.

One other thing we should clear up:  Cloud Storage is for files, which some refer to as objects, and others call unstructured data.  Think about the "files" stored on your PC, like pictures, spreadsheets and documents.  These have an extraordinary variability, thus "unstructured".  The other kind of data is "block" or "structured" data.  Think data base data, data that feeds transactional system that require a certain "guaranteed" or low-latency performance.  Cloud Storage is not for this use case.  IDC estimates that approximately 70% of the machine stored data in the world is unstructured, and this is also the fastest growing data type.

So, Cloud Storage is storage for files that is easily accessed via the Internet.  This does not mean you cannot access Cloud Storage on a private network or LAN, which may also provide access to a storage cloud by other approaches, like NFS or CIFS.  It does mean that the primary and preferred access is by a REST API.  (Here are other terms you will see, RESTful, or RESTlike or RESTstyle, which is geekspeak for how closely the API conforms to the REST approach.) 

Today, there are multiple definitions for Cloud Storage, and the one I prefer is "File Storage accessed through Web Services API's over a network".  This represents the key attributes of file storage that is cloud storage, versus other types of file storage.  Other key qualities of a storage cloud are:

• multi-tenant support (use by more than one unrelated user)
• geo location and geo replication, seamless and real time provisioning of accounts
• seamless and real time provisioning of accounts
  
• availability of "practically" unlimited amounts of storage "on-demand"
• "pay for use", which means that your payment is for actual storage used, over some time frame, usually a month. 
  

There are many who are still arguing about what I have defined above, but what I've said is generally accepted by the industry.  If it is a vendor doing the arguing I would suggest you check under their hood, usually you will find that they do not offer whichever of the above features they are trying to argue out of the definition.

Also, traditional storage vendors continue to proclaim the importance of local network access (like NFS, CIFS or ISCSI) for the purpose of Cloud Storage access by applications that today can only access via the older protocols.   This requires that the application making the request be on the same local network (think same data center) as the storage cloud.  Their reason for this view is that they are only just beginning to see application demand for storage cloud access via REST APIs, versus their traditional business model which serves an enterprise user with their own data center. 

This is why Cloud Storage has generally emerged as a service offering in the IT Service Provider  (also know as the WEB Hosting Industry) space first.  In this space, there is no doubting the importance and future of REST API access to storage clouds, it is only viewed as an adoption speed issue.  Note that within the data center, access to storage using an HTTP based protocol is not necessarily any slower than one of the more traditional protocols. API access has been labeled as being a slower form of access over NFS and CIFS. This view is largely due to the fact that it "may" be accessed over the Internet. In most cases, it is the network that adds the latency, not the means of access. Make no mistake, traditional storage vendors see this coming, and they will make offerings available in the near future.

REST APIs are language neutral and therefore can be leveraged, very easily, by developers using any development language they choose. Resources within the system may be acted on through a URL. So, an API is not a "programming language" it is the way a programming language is used to access a storage cloud.  This is part of the basic understanding of APIs that is required to discuss the dreaded "vendor lock in" and upcoming "cloud lock in" discussions and understand the issues that surround these assertions.

REST APIs are also about changing the state of resource through representations of those resources. They are not about calling web service methods in a functional sense. The key differences between different Cloud Storage APIs are the URLs defining the resources and the format of the representations.
 
The Cloud Storage space is very young and everyone has their opinions on how things should be represented and accessed. Efforts are underway by organizations like SNIA, with their Cloud Data Management Interface (CDMI), to standardize both the resource structure and the representations. However, standards are not developed overnight and customers are demanding programmatic access to Cloud Storage now.

Current Cloud Storage vendors have produced a basic set of APIs that are accomplishing fairly similar things, and other APIs that expose the underlying unique functionality of the Cloud Storage platform supplying the storage cloud.  You should expect that, over time, most storage clouds will provide the basic functions in somewhat similar ways, and further that additional advanced functions will be adopted and expected to be in every storage cloud offering. 

Finally, you should look for a taxonomy of APIs, that includes basic file functions, advanced functions, Provisioning APIs, Billing APIs, and Management APIs.  Storage clouds that become successful will offer all these capabilities, to increase the efficiency of their use.



 
Several efforts have been made to simplify the transition between vendors by providing an abstraction layer on top of the vendor's APIs. In this approach, a program library is created, for use in the application that needs cloud storage access, and this API translates (for the given program language) a single API into the API that is specific to a Cloud Storage offering.  So, the application, which is using this library, writes their APIs once, and achieves portability between storage clouds that are supported by this approach.

This approach has been largely programming language specific and may take advantage of the language it was designed for. Good examples of this are jClouds, an open source cloud storage abstraction library written in Java, and Simple Cloud API, a collaboration of vendors including Microsoft, Rackspace, Nirvanix, IBM and Zend which provides a simplified Cloud Storage interface for PHP developers. While extremely useful for developers, these abstractions tend to expose the lowest common denominator relating to Cloud Storage functionality and may omit critical features, for example only providing namespace object access as opposed to ID access.

So, let's discuss lock-in, the term used to express concern that once a vendor has gotten you to exploit their architecture and technology, they will recognize that you are committed to them and cannot easily move away.  As a result, they will then raise their prices and take advantage of your lock in status, keeping their price just below the amount that would encourage conversion away from their technology and towards a more "open" set of capabilities.  Let's look at all the "dreaded" examples that have been surfaced around cloud storage and as a reason to slow it's adoption:

1.    API lock in, which means your interaction with a storage cloud uses the APIs of that storage cloud, and suggests that you cannot easily move to another providers cloud with their own, different APIs.

2.    Vendor lock in, which means that since you are condemned because of your application development activity with specific APIs to use only a cloud from a specific supplier.

3.    Device lock in, meaning that you developed a cloud storage based program utilizing the APIs of that specific cloud, for a specific device (generally a PDA) that has specific functionality.  This is double lock in, both the device programming methodology and the API selection.

4.    Browser lock in, meaning that programming to specific APIs can also be rendered unique based on the Web browser that is selected.

5.    Programming language lock in, which means that you have written the APIs in a language like Python, or JAVA, or .NET, or whatever.

6.    API wrapper lock in, which means that you incorporated libraries into your application that allows your application to write generic APIs, which are then translated by these APIs to the correct API for the desired storage cloud (this is what Simple Cloud API is).

So, as you can see here, utilizing cloud storage could ultimately have you locked in on at least six levels! 

With this much opportunity for vendor abuse, why are developers rushing to write Web based applications that utilize cloud storage services via API access?  Are they simply uncontrolled, unthinking rebels who will shortly learn the error of their ways?  Have they made a fatal error?  Or do they know something you don't?

First, learn about Cloud Storage APIs.  What they do is make storage programmable, and they abstract storage from the application.  They offer advanced functionality (the programmable word) that makes it faster and easier to write the applications that are scalable versus the traditional storage access approaches.  When you add these two capabilities to the storage cloud offering of low cost, availability in multiple locations, seamless provisioning, ease of adding additional storage, and the pay for use model, the case for the cloud has become compelling.

Where are we seeing early adoption:  at service providers, because they host Web based applications and SaaS (usually Web based) applications, and this is where the developers who recognize the opportunity are focused. 

What is coming: the introduction of this technology into the enterprise, complete with the adoption of the RESTful API technology.  This will ultimately lead to a level of cooperation between service providers and the enterprise that has long been predicted.  Enterprises will move to an IT modeled on an OPEX model, and expect their applications to be provisioned and interacting with service provider clouds, via APIs.  IT Service Providers are racing to build the clouds to provide for this emerging business opportunity.

So, what about the lock in mentioned above.  Sit down with your developer, they will show you why they don't feel "locked in".  They will show you that you can quickly recraft your current APIs, in the programming language of your choice, to utilize the new APIs of the desired cloud.  For this reason, Simple Cloud API will likely be a short term measure, which precedes base case APIs that are extremely similar, and goes through a market led process to identify "best practice" APIs for both base case and advanced function, as well as all the other API led capabilities as mentioned above.  In short, vendor lock in is not the problem for this technology that it has been for others.  Also, the ingenuity and resourcefulness of all the suppliers, standards groups, and market adoption scenarios will continue to mute your ability to be lock in free. 

Your real challenge is not lock -in, but rather how to adopt this new set of capabilities, and solve problems and create opportunities with your IT solutions as rapidly as possible.  Standing on the sidelines waiting for this one to resolve will keep you out of a great opportunity, because we still have several meaningful years of rapid change associated with this technology adoption cycle. 

BMC Software's announcement that it has entered into a definitive agreement to acquire privately-held Tideway Systems Limited (Tideway), a provider of IT discovery solutions, can be interpreted as an extension of BMC's commitment to cloud-computing.

Here are two important statements in the press-release:

1. BMC will deliver unmatched visibility into the data center and rapidly reduce the time and resources required to model, manage and maintain applications and services. This is critical for IT organizations that are transitioning applications and services to cloud computing environments.

2. With the acquisition of Tideway, BMC adds the industry's leading application discovery and dependency mapping capabilities to manage and maintain complex data center environments including distributed, virtual and mainframe IT platforms and further extends its leadership in business service management.

So let's see what this could mean. 

It gives BMC the critical capability to discover and map complex data environments which are both physical and cloud-based.

This acquisition also puts BMC in a strong position to build a cloud-based CMDB.  While that might not happen right away, it is clearly now a key capability if they decide to pursue it. It also allows them to build a federated CMDB - and manage the hybrid cloud - private and public - across enterprise and hosted data centers. 

The evolution towards cloud-based ITIL continues.

We've discussed ITIL and Cloud Computing and the role of trust as a differentiator for service providers. Yes, we see the evidence that IT Hosting companies and managed service providers are closer to their customers and we see that their differentiation is their commitment to serving the customer.

But Amazon, Google, and Microsoft aren't going away. As they pressure customers to make the switch to the cloud, traditional service providers must find new ways to compete. Step one, of course, is providing alternatives - cloud services, like storage for example.  Step two is to highlight their customer commitment - the relationships they already have and defend this "advantage" by becoming even more responsive. 

So how do you build trust? According to Stephen Covey Jr. trust is built through behavior. His work has identified 13 behaviors which build trust:

1. Talk Straight
2. Demonstrate Respect
3. Create Transparency
4. Right Wrongs
5. Show Loyalty
6. Deliver Results
7. Get Better
8. Confront Reality
9. Clarify Expectations
10. Practice Accountability
11. Listen First
12. Keep Commitments
13. Extend Trust

But how do these behaviors translate to a cloud service delivery model? 

To answer this question, I dug up an old model for assessing service quality - SERVQUAL -  which was introduced to the world of service and retail back in 1988 (those were the days before ITIL).  SERVQUAL has its share of detractors, but even recent research reminds us that it is still a useful model.  In particular, I'm interested in how it can be used to help service providers improve and extend their intangible advantages over the more impersonal big shops.

Over the years, the SERVQUAL instrument has been a popular methodology used to measure consumers' perceptions of service quality. Its five generic dimensions or factors are still valid:

(1) Tangibles: physical facilities, equipment and appearance of personnel.
(2) Reliability: the ability to perform the promised service dependably and accurately.
(3) Responsiveness: willingness to help customers and provide prompt service.
(4) Assurance: includes competence, courtesy, credibility and security; the knowledge and courtesy of employees and their ability to inspire trust and confidence.
(5) Empathy: includes access, communication, understanding the customer; caring and
individualized attention that the firm provides to its customers.

None of these dimensions will change in the cloud, with the exception that some of these dimensions are now virtual and must be proven online (customer support, for example) or through superior automation of work processes.

Let's also analyze the SERVQUAL "gap model," as it was called, and see how it applies to service delivery in the cloud:

Let's look at the meaning of each "gap" - the possible breakdown areas in service delivery:

Gap 1: Customers' expectations versus management perceptions: caused by the lack of a marketing research orientation, inadequate upward communication and too many layers of management.

Gap 2: Management perceptions versus service specifications: caused by an inadequate commitment to service quality, a perception of unfeasibility, inadequate task standardization and an absence of goal setting.

Gap 3: Service specifications versus service delivery: caused by role ambiguity and conflict, poor employee-job fit and poor technology-job fit, inappropriate supervisory control systems, lack of perceived control and lack of teamwork.

Gap 4: Service delivery versus external communication: caused by inadequate horizontal communications and propensity to over-promise.

Gap 5: The discrepancy between customer expectations and their perceptions of the service delivered: caused by the influences exerted from the customer side and the shortfalls (gaps) on the part of the service provider. In this case, customer expectations are influenced by the extent of personal needs, word of mouth recommendation and past service experiences.

Gap 6: The discrepancy between customer expectations and employees' perceptions: caused by the differences in the understanding of customer expectations by front-line service providers.

Gap 7: The discrepancy between employee's perceptions and management perceptions: caused by the differences in the understanding of customer expectations between managers and service providers.

Three of these gaps are directly connected external customers: Gap 1, Gap 5 and Gap 6.  Service providers will find their optimal "trust-building" opportunities here.  Apply Covey's 13 behaviors to each one of these gaps to build on your commitment to your customers.

Amazon, Google, and Microsoft aren't building a high-touch responsive model for their cloud services. But you, the service-provider, already have a high-touch relationship. Your cloud-based SLAs must reflect this advantage. The security issue is just a small part of this reality.

Service providers who dedicate themselves to closing the gaps will succeed in this new world.

The quest for quality service didn't start yesterday. I highly recommend that service providers give Delivering quality service: balancing customer perceptions and expectations by Valarie A. Zeithaml, A. Parasuraman, Leonard L. Berry, a second look.

With all due respect to Cory Doctorow, he's wrong.

In his article Not every cloud has a silver lining (Guardian) he states:

There's something you won't see mentioned by too many advocates of cloud computing - the main attraction is making money from you.

And I suppose all the vendors of physical storage, the hard drives, etc., are interested in your spiritual well being!

Here's the heart of Doctorow's beef with cloud computing:

Rather than buying a hard-drive once and paying nothing - apart from the electricity bill - to run it, you can buy cloud storage and pay for those sectors every month. Rather than buying a high-powered CPU and computing on that, you can move your computing needs to the cloud and pay for every cycle you eat.

The point he misses is that cloud computing exists because it answers a real need.

We aren't prohibiting you from buying physical hard drives, I assure you. In fact, we think physical and cloud storage will work together, complementing each other. It's not one or the other.

Our focus is to provide specific services which deliver quantifiable value for both individual consumers and business users. As we mentioned earlier, the basic promise of cloud computing: instant access to your data anytime, anywhere, on any device, is already a reality. It's all about convenience, ease-of-use, cost, and of course, value-delivered. The cloud is a disruptive innovation.

Let's review the benefits of cloud computing:

For Businesses
In addition to cloud storage, the cloud brings game-changing pricing and service capabilities to disaster recovery, fault tolerance, geographic redundancy, and other solutions that have been prohibitively expensive to everyone except for the largest organizations in the world. Here are some specific drivers of business value:

Financial Benefits: The very nature of "pay per use" makes large upfront financial outlays a thing of the past. So your CFO won't bug you about capital expenditures.  You'll simply have to pay a monthly fee for renting the data center and the services you choose. And yes, that's a monthly operational cost.

Better use of Human Resources: Your IT people don't have to spend time doing repetitive tasks like provisioning and  setting passwords.  That will be done in an automated way by your service provider.

Agile Provisioning: "Time to value" is greatly accelerated using the cloud.  Softlayer, for example, allows lets you deploy on-demand computing instances running enterprise-grade and open source operating systems in as few as five minutes. Can your IT department do that today?

Scalability and Flexibility: The cloud provides customers with the capability to start small and grow with demand, in real-time. Cloud "burstability" allows for rapid scaling to meet demand caused by usage spikes.

Leaner and Greener Infrastructure: The cloud allows companies to outsource their IT infrastructure, and maximize utilization of the computing power of their service provider. This makes for a leaner and greener IT infrastructure for all.

Service Oriented Architecture:  Cloud Storage accessed via RESTful Web Services APIs provides new capabilities for developers.  For the first time, an abstracted, services rich storage layer is a true SOA implementation.

For Individuals
I already hear commercials on TV:

"Are you tired of lugging your laptop everywhere? Are you tired of transferring your files every time you switch devices? Are your running out of space for your endless downloads of videos, songs, and movies? Do you want to access your files anytime, anywhere, on any device? Try cloud computing, and your life will never be the same."

The flexibility cloud computing offers individuals is unparalleled. Again, it is the user-experience which will determine cloud use by the consumer.  And as we see more and more personal files (videos, music, photographs) explode, we'll see a bigger and bigger role for cloud computing.

A final statement. We do want to make money. Like everyone else in the market, we're going to have to deliver value to earn your trust and dollars. And if you find that you get more value from buying your own physical storage or owning and operating your own datacenter, go ahead.  We're betting we can show you a better way, a way that complements your local storage.

When we posted our ParaScale Fact Check blog post earlier, it was specifically oriented to a situation where there were specific, unsupported claims by ParaScale. 

A fact check should be based on facts, not marketing claims or opinions.

The claim that ParaScale offers Web services APIs, among other claims, initiated our response.  Now, ParaScale not only does not have them, but claims that you should not use them, unless and until there are S3 APIs.  Furthermore, they have now initiated a Mezeo fact check.  Except they are making marketing claims, they do not, nor can they offer "facts" that are claims by Mezeo that are not in fact true.

Here are their claims and our responses:

Mezeo does not give you the storage economics to compete with Amazon.

Mezeo response:  ask our many customer references in the hosting industry, they are competing and winning against Amazon.  By the way, we came from the hosting industry, and we know how much it costs to host storage. They don't state a fact, this is their opinion, and it is unsupported by facts.

Mezeo is not adaptable to your customer's data access needs of standard protocols.

Mezeo response:  We have offered WebDAV support since Q1 of this year, our Windows Native client has been available since we launched our company, as well as REST APIs.  CIFS and NFS are easily with the scope of our capabilities, and we will be advising our customers and prospects of our plans in this area. It is unclear to me that this demonstrates a "lack of adaptability".

It is simply proprietary REST API with custom clients. 
Mezeo response:  This is not a fact.  In fact, we now have to fact check ParaScale again.  The Mezeo platform offers significant, extensible services that go beyond those on competitive public cloud offerings.  These include secure sharing, public sharing, collaboration, tagging, notifications, permissions, and numerous other services that make Mezeo a desired platform for Web developers.  Our SPML based provisioning integration; APIs for billing and bandwidth utilization, and Acceptable Use Management further differentiate the offering.  Many consider our ability to accommodate both industry standard file systems and clustered file systems (like ParaScale, for example) for the storage target of a Mezeo based cloud as an advantage.  You can mix and match storage offerings in a Mezeo cloud to achieve different offerings of price, performance and availability, all on a single infrastructure.  Finally, access, via APIs, WebDAV and our white label clients are a critical differentiator, and we deliver all of this today!

PROPRIETARY REST API:  With an understanding of REST and APIs, you understand that this is not the critical issue.  The critical issue is that Web developers want to develop against platforms that offer REST APIs.  The minimal changes required on the APIs to move from one cloud to another has never been raised as an issue of significance, versus the services and features of specific clouds.  

To this very point, Lydia Leong of Gartner, in a recent blog post, asks the question:  Are Multiple Cloud APIs Bad?

"... I believe that it's too early in the market to seek commoditization. Universal commitment to a particular API at this point clamps standardized functionality within a least-common-denominator range, and it restricts the implementation possibilities, to the detriment of innovation. As long as there is rapid innovation and the market continues to offer a slew of new features -- something which I anticipate will continue at least through the end of 2011 and likely beyond -- standardization is going to be of highly limited benefit."

Mezeo is also engaged with the SNIA Cloud Storage Technical Working Group (I talk about the group here) to work with the industry to sort out the requirements for APIs that will allow for transportability, and there are other vendors that build wrappers that provide for easy portability amongst APIs.  

Finally, here's an opinion: if I were calling myself cloud storage for the IT hosting industry, and I did not offer REST Web Services APIs, I would likely argue that they are not needed, or at least that they are the wrong ones if they are not like S3.  When I am with my colleagues in the Web Hosting Industry, they find this argument against APIs amusing to say the least, and that's a fact you'll have to accept my opinion on! 

My last post on REST generated some attention.  Since it is an important topic, I wanted to share some additional links for those who are trying to improve their understanding of REST:

- Stefan Tilkov's Intro to REST presentation and When is an API RESTful?
- Dare Obasanjo's Explaining REST to Damien Katz
- Paul Precod's Second Generation Web Services, REST and the Real World, SOAP, REST and Interoperability
- Tim Bray's The Sun Cloud and REST, as in Take It Easy
- More stuff from Roger Costello
- Ryan Tomayko's How I Explained REST to My Wife
- Roy T. Fielding's Dissertation: Chapter 5

REST reflects the architecture of the Web.  One of its most important characteristics (and there are many) is that it is "stateless".  That means that a REST style command from a requestor to a responder has everything in it the responder needs to know in order to take an action.  No further handshaking is required.  Very efficient and Web like.  Since it is "stateless" it works very well with a "stateless" server architecture, in order to achieve Web scale.  In this way many clients can interact with many servers against a large pool of objects to accomplish many interactions, well, you get the point, Web scale.  That's one reason we use RESTful Web Services API commands to access the Mezeo Cloud Storage Platform servers, which are also stateless architectures, implemented via Linux.  Web scale, one of the requirements of cloud.

REST is also highly efficient, so that interactions between requestors and responders via a network can be done with a minimum of overhead.  If you ever download a 500 gigabyte file via a cable modem based internet connection, you will likely appreciate any efficiency that can be achieved.  Speaking of efficiency, REST also accommodates caching, at both the client and the server, which can dramatically improve the efficiency of your interactions with the "object" (an object could be, for example, a file, like a picture, or a pdf). 

Developers who utilize RESTful Web Services APIs to create applications appreciate the efficiency and capability of the APIs.  Expect, over time, to see more commonality among base case APIs and other APIs that expose storage cloud specific advanced services.  For example, Mezeo based clouds offer a secure share, collaboration, notifications, and nested files and folders, for example.  Some clouds may have such a unique set of APIs that others will create translators (wrappers, for the IT guys in the audience) for them, and we will continue to make headway on openness.

RESTful APIs are a critical part of new application development, and represent the delivery of Service Oriented Architecture infrastructure for storage.  Storage is now programmable.  And I bet you thought cloud storage was just a utility computing model applied to storage, for scalability and pay for use.  Both are necessary, but not sufficient for cloud storage.

Most of us in the Cloud Storage industry strongly believe that a key capability of a storage cloud is the REST style Web Services API.  Many of the most popular storage cloud services include or exclusively use REST, including SoftLayer's CloudLayer, Amazon S3, Nirvanix SDN and Rackspace Cloud Files.

Other access methods that are most often associated with Cloud Storage access include cifs, NFS and WebDAV,  NFS and cifs are not particularly usable via an Internet connection and therefore useless in public cloud offerings.  While WebDAV is very useful for an Internet connection, it is similarly limited, in that all three protocols support traditional file operations like store and retrieve, versus the robust set of services that Web Services APIs can deliver.

Amazon introduced S3 with REST style API access only.  Cloud Files from Rackspace also utilizes REST style APIs. Nirvanix SDN utilizes both REST and SOAP APIs.  Mezeo offers REST APIs.   Various groups are also engaging on the issue of what representations of REST should be common across cloud offerings.  The SNIA, (the Storage Networking Industry Association) has assembled a technical Cloud Storage working group for further refinement of REST style implementations for several purposes.

So, what is the purpose of the other, older access protocols?  When deployed with API based Cloud Storage offerings, they provide additional options for legacy applications to expose their objects (files) to the advanced services of the Cloud, and further make these files available to the new API based applications.

Why all the excitement about RESTful APIs?  Cloud Storage is more than a utility business model applied to traditional storage. It is storage that is accessed via Web Services APIs, over a network. Developers utilize these APIs because they are easy to use and they expose significant capabilities and services from the storage cloud, far beyond scalability, performance and pay for use.  As I have said before, scalability and pay for use are as much a business decision about how you sell storage, as they are a technology implementation of storage.  If there were no need for the API based services, the older and well used protocols would persevere.  This is clearly not the case.

I have carefully avoided the use of the word "standard" associated with the REST  style or architecture.  Here is an interesting view on that topic from Roger Costello:

REST is not a standard. You will not see the W3C putting out a REST specification. You will not see IBM or Microsoft or Sun selling a REST developer's toolkit. Why? Because REST is just an architectural style. You can't bottle up that style. You can only understand it, and design your Web services in that style. (Analogous to the client-server architectural style. There is no client-server standard.)

Cloud storage service providers understand that a new storage infrastructure has emerged, as an embodiment of Service Oriented Architecture, with a set of services that are delivered via APIs.  Scalability, performance and pay for use are attributes of traditional and cloud storage solutions, but Web services APIs are the distinguishing feature of cloud storage.  Accessing storage via Web services APIs represents a revolutionary change in storage, not a simple generational change. REST APIs are the embodiment of the way the Web works and are necessary to expose storage as a "storage cloud"!

What should you expect in relation to these API issues?

Most of us expect that over time, there will be a base set of specifications that are jointly developed within the marketplace and by various industry organization, resulting in a well accepted set of representations for REST style Web Services APIs.  At a panel at Hosting Con earlier this week, both Emil Seyegh of Rackspace and myself confirmed that when the industry gets further clarity on this specification, it will be relatively easy to introduce those APIs into our offerings, and that they can co exist with our current APIs.

REST is a topic that you will continue hearing more about.  You'll most certainly hear more about it from me in future posts.