CloudOps

Storage Intelligence - about time!

I was reading recently an article about Backblaze releasing storage designs. This is a 180TB NAS device in 4U! Absolutely huge! A 42U rack would be able to have around 1.8Petabyte in a single rack.

Blog-pod30-header

When thinking about Petabytes, one thinks about the big players in storage, EMC/NetApp/HDS, selling tens of storage racks covering substantial parts of the datacenter floor space and offering a fraction of this capability.

Vmax_images Fas_index

Clearly, the storage profile of what the large monolithic enterprise arrays offer is different. However, Backblaze clearly highlights the ability to get conventional "dumb" storage easily and at low cost! Packing some flash  cache or SSD in front would already bring these boxes to the same I/O capacity;-)

This makes the case that storage per se is not really a challenge anymore. However, making storage aid in the overall performance equation; making sure that storage helps in specific workload acceleration is going to be critical going forward. Basically Intelligent Storage!

Questioning "Accepted Wisdom"

Many IT shops still think of storage as a separate part of their estate. It should simply store data and provide it back rapidly when asked - politely. The continuing stifling of innovation in datacenters due to having a single answer for all questions - namely VMware/hypervisors and server virtualization - tends to stop any innovative thinking that may actually aid an organisation to accelerate those parts of the application landscape leveraging revenue.

Some questions that came to mind and also echoed by clients are:

  • Disk is cheap now. SSD answers my performance needs for storage access. Is there something that together with software actually increases the efficiency of how I do things in the business?

  • For whole classes of typical applications, structured data persistence pools, web servers etc what would "intelligent" storage do for the physical estate and the business consumers of this resource?

  • How can enterprise architecture concepts be overlaid to intelligent storage? What will this mean to how future change programmes or business initiatives are structured and architected?

  • How can current concepts of intelligent storage be used in the current datacenter landscape?

We are seeing the first impact of this type of thinking in the structured data / database world. By combining the database workload with storage and through software enablement we get  intelligent acceleration of store/retrieval operations. This is very akin to having map-reduce concepts within the relational database world.

Further combining storage processing, with CPU/RAM/Networking offload of workload specific storage requests, facilitatest unprecedented scale-out, performance and data compression capabilities.

Oracle's Engineered Systems, the Exadata Database Machine in particular, represents this intelligent storage concept, amongst other innovations, for accelerating the Oracle database workload.

These workload specific constructs foster security of precious data assets, physically and logically. This is increasingly important when one considers that organisations are using shared "dumb" storage for virtual machines, general data assets and application working data sets.

In the general marketplace other vendors (IBM PureSystems + DB2, Teradata, SAP HANA etc) starting to use variations of the technologies for intelligent storage. The level of maturity varies dramatically, with Oracle having a substantial time advantage as first mover.

2013-2015 will see more workload focused solutions materializing, replacing substantial swathes of datacenter assets built using the traditional storage view.

Why is this important for the CIO, CTO & CFO?

Intelligent workload-focused storage solutions are allowing CIO/CTOs to do things that were not easily implemented within solutions based on server virtualization technology using shared monolithic storage arrays - dumb storage - such as in the VMware enabled VCE Vblock and HP CloudSystem Matrix - which are effectively only IaaS solutions.

Workload specific storage solutions are allowing much greater consolidation ratios. Forget the 20-30-40 Virtual Machines per physical server. Think 100s of workloads per intelligent construct! An improvement of 100s of percent over the current situation!

It is important to verify how intelligent storage solutions can be a part of the CIO/CTO's product mix to support the business aspirations as well as simplify the IT landscape. Financing options are also vastly simplified with a direct link between business performance and physical asset procurement/leasing:

  • Intelligent storage removes architectural storage bottlenecks and really shares the compute/IO/networking load more fully.

  • Intelligent storage ensures those workloads supporting the business revenue generating activities are accelerated. Acceleration is linked to the cost of underlying storage assets. As cost of NAND flash, SSDs and rotating disks drop, more is automatically brought into the storage mix to reduce overall costs without disrupting the IT landscape.

  • Greater volumes of historic data are accessible thanks to the huge level of context sensitive, workload-specific data compression technologies. Big data analytics can be powered from here, as well as enterprise datawarehouse needs. This goes beyond simple static storage tiering and deduplication technologies that are unaware of WHAT they are storing!
  • Workload-specific stacking supports much higher levels of consolidation than simple server virtualization. The positive side effects of technologies such as Exadata include the rationalization of datacenter workload estates in terms of variety, operating systems can be rationalized and generally have net-net healthier estate. This means big savings for the CFO!

Intelligent storage within vertically engineered workload specific constructs, what Gartner calls Fabric Based Infrastructure present a more cogent vision of optimizing the organizational's IT capability. It provides a higher level of understanding how precious funding from CFOs is invested to those programmes necessary for the welfare of the concern.

CIO/CTOs still talking about x86 and server virtualization as the means to tackle every Business IT challenge would be well advised to keep an eye on this development.

Intelligent storage will be a fundamental part of the IT landscape allowing effective competition with hyperscale Cloud Providers such as Google/Amazon and curtailing the funding leakage from the business to external providers.

Disclaimer

The opinions expressed here are my personal opinions. Content published here is not read or approved in advance by my current employer and does not necessarily reflect the views and opinions of my employer.

Inflexibility of Datacenter Culture - 'The way we do things round here' & Engineered Systems

With a focus on large enterprise and service provider datacenter infrastructures, I get the chance to regularly meet with senior executives as well as rank and file administrators - top-2-bottom view.

One of the things that as always struck me as rather strange is the relative "inflexible" nature of datacenters and their management operations.

As an anecdotal example, I recall one organization with a heavy focus on cost cutting. At the same time the datacenter management staff decided that they would standardize on all grey racks from the firm Rittal. Nothing wrong here - a very respectable vendor.

The challenge arising was:

  • The selected Rittal racks at that time were around 12,000 Euro each approximately
  • The racks that came from suppliers such as Dell, Fujitsu, Sun etc were around 6,000 Euro each

See the problem? 50% savings thrown literally out of the window because someone wanted all grey racks. When we are talking about a couple of racks - that is no big deal. With say 1000 racks we are looking at over-expense of 6 million Euro - before anything has been put into those racks!

Upon asking why it was necessary to standardize the racks, the answers I got were:

  • independence from server manufacturers
  • create rack and cabling rows before servers arrive to facilitate provisioning
  • simpler ordering
  • perceived "better-ness" as enclosures is a Rittal specializ

Sounds reasonable at first glance - until we see that racks are all engineered t support certain loads, and typically optimized for what they will eventually contain. Ordering was also not really simpler - as the server vendors made that a "no brainer". Perception of quality was not validated either - just a gut feel.

The heart of the problem as came out later was that the datacenter would benefit from having everything homogenous. Homogenous = standardized for datacenter staff.

The problem with this is that datacenters are not flexible at all, they have a focus on homogeneity and ultimately cost lots of money to the business financing them.

In an age where flexibility and agility means the literal difference between life and death for an IT organization, it is incumbent on management to ensure that datacenter culture allows the rapid adoption of competitive technologies within the datacenter confines.

Standardized does not mean that everything needs to be physically the same. It involves the processes o dealing with change in such as way that it can be effected quickly, easily and effectively to derive key value!

I indicated the recent trend of CIOs reporting to CFOs and this would have provided financial stewardship and accountability in this case - getting staff and managers to really examine their decision in the light of what was best for the organization.

Questioning "Datacenter Wisdom"

The focus on homogenous environments has become so strong that everything is being made to equate a zero-sum game. This happens in cycles in the industry. We had mainframes in the past, Unix, Linux (which is Unix basically for x86), Windows - and the latest is VMware vSphere and all x86!

Don't get me wrong here - as a former EMC employee - I have the greatest respect for VMware and indeed the potential x86 cost savings.

What is a concern is this is translated to "strategy". In this case the approach has been selected without understanding why! It is a patch to cover profligate past spending and hoped that magically all possible issues will be solved.

After all - it is homogenous. All x86, all VMware, all virtualized. Must be good - everyone says it is good!

See what I mean - the thinking and strategizing has been pushed to the side. Apparently there is no time to do that. That is really hard to believe, as this is one of those areas that fall squarely into the CIO/CTO/CFO's collective lap.

There are other approaches, and indeed they are not mutually exclusive. Rather they stem from understanding underlying challenges - and verifying if there are now solutions to tackle those challenges head-on.

Why is this important for the CIO?

At a time of crisis and oversight, it is incumbent on the CIO to question the approach put on his/her table for Datacenter Infrastructure transformation.

The CIO has the authority to question what is happening in his/her turf.

At countless organisations, I have performed strategic analysis of macro challenges mapped to the IT infrastructure capability of an organization to deal with those changes. Time and again, when discussing with the techies and managers (who were from a technical background but seem to struggle with strategy formulation itself) it was shown that the marginal differences in technologies were not enough to justify the additional expenditure - or that there were other approaches.

Engineered Systems, in Oracle parlance, are one such challenge. They do not "fit" the datacenter culture. They can't be taken apart and then distributed into the slots that are free in racks spread over the datacenter.

From a strategy perspective, a great opportunity is not being exploited here. Engineered systems, such as Exadata, Exalogic, SPARC SuperCluster, Exalytics,  and Oracle Data Appliance represent the chance to change the Datacenter culture and indeed make the whole datacenter more flexible and agile.

They force a mindset change - that the datacenter is a housing environment that contains multiple datacenters within it. Those mini-datacenters each represent unique capabilities within the IT landscape. They just need to be brought in and then cabled up to the datacenter network, power, cooling and space cabilities of the housing datacenter.

There are other assets like this in the datacenter already - enterprise tape libraries provinding their unique capability to the entire datacenter. Nobody tries to take a tape drive or cartridge out and place it physically somewhere else!

Engineered Systems are like that too. Taking Exadata as an example. This is clearly assembled and tuned to do database work with the Oracle Database 11gR2 platform. It is tuned and tweaked to do that extremely well. It breaks down some of the traditional barriers of datawarehouse and OLTP workloads and indeed allows database workloads to be "stacked".

Taking the idea of what a datacenter really should be (facilities for storing and running IT infrastructure) and being flexible - Exadata should literally be placed on the floor, cabled to the main LAN and power conduits and the database infrastructure platform is in place. After that databases can be created in this mini-Datacenter. The capability is literally available immediately.

Contrast this with creating lots of racks in rows where it is not certain what will be in those racks, put VMware everywhere, add lots of SAN cabling as I/O will always be an issue - and then spend ages in tuning performance to make sure it works well.

The CIO should identify this as a waste of time and resources. These are clever people who should be doing clever things for the benefit of an organisation. It would be similar to the idea of buying a car to drive around in OR getting the techies to buy all the components of the car and trying to assemble.

This loss of value inherent in the Exadata when taking a x86/hypervisor route to creating many virtual virtual machines running each a separate database make no real sense in the case of databases.

The CIO can use valuable organizational knowledge gained over many years regarding the functions the business needs. If, as in this example, it is the ability to store/retrieve/manage structured information at scale - the answer should literally be to bring in that platform and leverage the cascading value it provides to the business.

Neither x86 or a certain operating system is "strategically" relevant - in that it is a platform - and the normal DBAs can manage this using tools they already know. This mini-datacenter concept can be used in extremely effective ways and supports the notion of continuous consolidation.

CIOs can get very rapid quick-wins for an organization in this way. Datacenter infrastructure management and strategy should be considered in terms of bringing in platforms that can do their job well with software and hardware well tuned to run together. Further, they should reduce "other" assets and software that is needed.

Exadata does with not needing a SAN switch/cabling infrastructure - it encapsulates paradigms of virtualization, cloud and continuous consolidation. This will drive deep savings and allow value to be derived rapidly.

Challenge datacenter ideas and culture in particular. Agility requires being prepared to change things and being equipped to absorb change quickly!

Disclaimer

The opinions expressed here are my personal opinions. Content published here is not read or approved in advance by Oracle and does not necessarily reflect the views and opinions of Oracle.

The Practical Cloud - Road to Cloud Value Derivation (CVD)

After a change of residence and a change of job - it is high time to write another blog!

In talks with many customers and indeed feedback on this blog site, I receive a lot of indication that the "current Cloud" the pre-2011 industry was marketing was simply tackling the infrastructure side of things. Much of the focus was on consolidating the x86 server estate and delivering features such as migration of live virtual machines across physical servers.

While this is fine as the initial steps to deriving some form of value - it is typically too little. Many business leaders and IT managers indicate that "we have bought into the Cloud, we have virtualized and we can even offer a level of VM automation in terms of provisioning" - "so where is this taking us and where can I highlight the ongoing value to the business?".

This is a very valid line of questioning. The client has millions of $$bucks$$ of equipment sitting on the floor, they have attended training and have done everything they were told to do. The result - they can create a virtual machine with software in minutes as opposed to hours or days. Cool!

That is not a lot to show the business for all that investment and evangelism. This is typically (and incorrectly) lauded as a solution and a great win for all!

IaaS alone, in my opinion, was always too little value.  The approach of simply putting servers, storage, network with a thin veneer of hypervisor magic has limited value in itself. This was incidentally the main haunt till 2011 for mainstream hypervisor purveyors.

This type of datacenter transformation using pre-assembled hardware for the sole purpose of consolidating x86 is too simple and let's face it - too dumb. Clients are cleverer than that. Clients have persisted in following the virtualization wave, and that is good. They have somewhat resisted the Cloud marketing till now as it was simply focused on replacement of their existing hardware and hypervisor stack.

Towards the tail end of 2011 we started seeing a stronger focus on provisioning enterprise software and environments - DB as a Service (DBaaS) which was nothing more than installing a database instance on a virtual machine through a browser provisioning page. Well that is better - but still does not smack of value! Indeed, if you want many big instances of databases with say 64 virtual CPUs per VM you were out of luck! AND yes there are customers that do this!

In 2011, we started to see the emergence of the appliance. This was an entire hardware and software stack that was factory installed. In some cases, such as the EMC GreenPlum appliance, this was built using the components with functional tuning to undertake the task. Others such as Oracle with Exadata Database Machine (which has been around since 2008 incidentally - but first used Sun intellectual property acquired in 2010) not only took the idea of virtualization but actually embedded it into all the components in the stack.

Through innovation, integration, best-of-breed technology and the simple idea that a system should do what it is designed for to the best of its ability, Exadata represents, in my opinion, a new approach to transformation that makes real business impact.

I am sure that during 2012 we will see a move away from the generalized Cloud stacks, such as VCE VBlock, Dell prepackaged servers with VMware installed and something similar from HP Virtualsystem for VMware. These systems are all focused at helping the hypervisor - in this case VMware vSphere, perform its job well. However, the hypervisor only lets you manage virtual machines! It does not do anything else!

That is also the reason that I see the move away from expensive hypervisor software solutions towards open source solutions or systems having the hypervisor embedded as a functional technology to support an enterprise software stack - with no $$ for the hypervisor per se.  

The Race to Business Value

One of the issues that has been stagnating business value derivation through Cloud technologies has been the lack of business as a driving stakeholder. Business should be  driving the IT roadmap for an organisation. Business defines what it wants from developers in the form of functionality. Why not the same for IT infrastructure?

You see the value of Business is that it thinks differently. Business tends to think at levels of enterprise architecture holistically as a driver and motor for business value generation! They think frameworks and they think (with developers and architects) in terms of enabling software platforms upon which to further their unique selling points.

The real Cloud value to be derived in that case is based on the software Cloud platforms leveraged to facilitate global service/application delivery with quality of service baked in. These platforms in turn are used to create further value!  

The real business case for the Cloud comes in the form of Platform-as-a-Service (PaaS). I think that Exadata hits this nail on the head. I don't just want to be able to setup a virtual machine running the database inside, I want the functionality of the database itself! Exadata delivers just that through a clever blend of components!

Why is this important for the CIO?

CIOs have set the agenda for Cloud in 2010-2011. They have seen that it has an effect on the delivery of IT services - but not necessarily a direct impact in the culture of the business or indeed the value the business derives. The early gains have been achieved, and it is time to move on to business focused IT.

CIOs look beyond the mainstream hype. They verify through intensive research and peer-level networking the effect of IT strategies on business value. The CIO pioneers and sets the agenda for deep intelligent consolidation. Not just doing more with less - BUT gaining greater business insight and leverage with fewer more effective resources!

Exadata, and engineered systems of that ilk, with embedded technology are paving the way for scale-up/out with extremely high performance and gathering in the benefits/innovations of the IT industry over the last years e.g. unified networking with Infiniband, high performance SSD storage, deduplication, compression, tiered value-oriented storage, big data capable file systems and indeed open source.  

That is a very potent mix, and Oracle customers are actively leveraging this. They have been using Linux and Oracle Solaris 11 to support those enterprise workloads needing that level of reliability and speed. They have been consolidating hundreds of database and middleware servers - yes - hardware, mixed OSs, non-x86 systems, licenses, management tools, script frameworks and so forth. This is real consolidation!

Further, they have used the well respected Oracle 11g enterprise capable platform to power their Java applications, drive the backend of their middleware platforms, created new value by delivering through the Exadata platform applications to the mobile space (iPads, Androids, Browsers, OS independent applications). 

Indeed, if the Java virtual machine (JVM) is one of the ultimate forms of virtualization, it makes perfect sense that as a business which has elected to use that technology you create the underlying infrastructure AND platform ecosystem to support those efforts at scale.

The Corporate Cloud Strategy can be dramatically refreshed and aligned with the ability to deal with all data needs in a single well managed platform. Exadata provides in this case the ability to deal with all database needs that an organisation has from the smallest to the largest. It provides significant front-end direct value. 

Other Exasystems have started to arrive to deal with specific challenges such as big data and middleware. These use the same magic source of Exadata Database Machine, but are tuned/enhanced for their specific functions. Deep lasting transformation can be achieved and the very nature of these Exasystems means the Business must be included as a principal stakeholder - they can truly see what the value of extracting a business insight means in hard $$ terms!

Look out for these paradigms that directly affect business value and indeed allow new business insight to be gained by easily manipulating petabytes of information in near-realtime! They provide the ability for the business to rapidly come to market with new products, support directly application developers, are built on industry-proven technologies - and best of all - retain the key know-how of your developers and DBAs - they will be up and running with little change to their operational routine!    

 

Disclaimer

The opinions expressed here are my personal opinions. Content published here is not read or approved in advance by Oracle and does not necessarily reflect the views and opinions of Oracle.

The Role of Defragmentation in the Cloud – Releasing trapped value

Imported from http://consultingblogs.emc.com/ published November 21 2010

One of the questions that we get from clients when moving to the Private Cloud is do we still need to do things as we did in the physical world?

Defragmentation of the file systems (FS) within guest operating systems (OS) containerized in a virtual machine (VM) always comes up. From my own enterprise messaging and database background, this was a very important question to get answered upfront. There could be tremendous negative consequences from not doing proper housekeeping and performing defragmentation (either offline or online while service was running). This essentially represents trapped value to the business.

There are many great vendors out there in this VM defragmentation space, for example, Diskeeper’s V-locity2 or Raxco’s PerfectDisk. They make a good case for defragmentation, essentially pointing to the fact that:

 

1. Many industry sources point to the negative impact of fragmentation

 

2. Fragmentation increases the time taken to read/write a file

 

3. Extra system workload resulting from fragmentation

 

4. Free space consolidation is very important for improving write operations

 

5. Fragmentation contributes to higher I/O bandwidth needs

 

6. Resource contention for I/O by VMs

 

7. VM disks perpetually growing even when deleting data

The fragmentation issue, whether of files or free space, has symptoms analogous to the performance issue VMware identified with misalignment of Guest OS partitions and indeed of VMFS itself. Essentially, much more unnecessary work is being done by the ESX host server and the corresponding storage elements.

The array and vSphere 4.1 features, help reduce the impact of these issues through I/O coalescing and utilizing array cache to get larger sequences of updates bundled – contiguous writing – EMC VMAX can provide 1TByte of cache currently. Multipathing tools such as EMC PowerPath/VE alleviates the increased I/O load through queue balancing and utilizing all paths to the storage array concurrently.

Thin provisioning ensures ‘Just-In-Time’ space allocation to virtual disks. This is heavily enhanced at the array hardware with complementary technologies as EMC's FAST to further optimize storage price/performance economics. This is also changing through the VMware vStorage VAAI such that vSphere is offloading storage to, surprise suprise, storage tiers that simply do the job better.

However, these do not proactively cure fragmentation within the guest OS or indeed at the VMFS level.

Indeed when we start thinking about environments with hundreds of thousands of VMs, such as in desktop virtualization, using VMware Linked Clones, this issue needs to be tackled. Virtual disk compaction represents an important element here to ensure online compaction capability; space reclaimed, and trapped value released.

The ability to use FAST, can support defragmentation scenarios by shifting the workload onto Solid state drives (SSD) for the duration of the high I/O activity. The array will then move the corresponding sub-LUN elements back to the appropriate tier later. Many customers do this with scripts.

Essentially, using Storage vMotion, the VM could be moved to a datastore on high performance disks, and then use the guest OS internal defragmentation tools. Once completed, the VM is storage vMotion’d back to its datastore. Seems easy enough to do for small numbers of machines, but does not scale to Cloud levels - doing this continuously for large VM volumes.

The whole area of scheduling defragmentation cycles, across an entire virtual infrastructure Cloud estate, is also no trivial task. Tools are needed. The current tool generation operate within the Guest OS. VMFS also warrants an examination, although with the ability to utilize 8MB block sizes, there is less fragmentation taking place at the VMDK level – but this is still worlds away from a self-defragmenting file system!

After all, in a busy Cloud environment, the datastores are heavily used. VMs are created and removed. This eventually causes fragmentation. Whether that is an issue for the Cloud environment – well it is still too early to say I believe.

My own view is that some of the best practices regarding defragmentation of the past are still relevant, but need to be updated with the current generation of applications. For example, Exchange 2000/2003 issues are different in scale than in Exchange 2007/2010. It’s the application stack that still counts as that is delivering service to end users. On the other hand, implementing thousands of defragmenting tools in guest OS VMs is also not my idea of fun, and cost may well be prohibitive. Side effects such as large growth in redo log files of any sort when defragmentation takes place also needs to be considered.

I’d like to see VMware create a defragmentation API integrated with the vStorage VAAI APIs for array awareness, much as they have for anti-virus scanning using the VMSafe API. This would allow the defrag engines to hook into the ability of the hypervisor itself and get the array to offload some of these tasks. That would also provide a consistent interface for vendors to design against, and defragmentation can then be a thing of the past - regardless of the guest OS running in the VM. The Cloud should just deal with it, when it is needed!

Disclaimer

The opinions expressed here are my personal opinions. Content published here is not read or approved in advance by EMC and does not necessarily reflect the views and opinions of EMC.