Why are my backups still killing me?Kelly J. Lipp, CTO, STORServer, Inc. Highlights
|

Let’s agree to make 2009 the year we stop having to ask this question! If we accomplish nothing else this year, we’ll probably chalk it up as a win.
So why are backups still a problem? I think the problem stems fundamentally from the need of IT folks to continually re-invent the wheel. For some reason, we operate under the misconception that our IT shop is so totally unique that an already tried and true solution just won’t work. How in the world can that be?
If you look at most shops today you’ll find a moderate number of servers, perhaps 50-100 (or less depending on your virtualization success) and a moderate amount (by today’s standards) of data, perhaps 10-50TB, mostly Windows, some flavors of Unix, running databases like Oracle or SQL, email servers like MS Exchange, a couple of file servers and perhaps some homegrown applications. Oh, and a smattering of infrastructure things like DNS, Active Directory and domain servers. All connected by GigE networks, sharing storage in a SAN. You get the picture.
Does this sound like your shop? Close enough? OK. Thousands of folks have solved this problem to a greater or lesser extent in a hundred different ways. Each way being somewhat unique (though there are only so many technologies and technology permutations available), but each attempt taking far longer than it should have.
Hundreds of enlightened folks have solved this problem by installing an appliance. Nothing unique as each appliance looks essentially the same. There is a backup server, some disk, some tape and some software (common in the thousands of solutions too). One fundamental difference, though, is that the appliance vendor put all the hard stuff together before it ever arrived in the data center, ensured that it all functioned together and performed as designed and that the customer was ready to implement. The big difference was that the solution was completely implemented in a week or less. In all cases.
How can this be? Most enterprise backup solutions take months to implement and even then many systems aren’t participating or key aspects of system restore or disaster recovery haven’t even been tested. Why? Months are spent re-inventing the wheel, that’s why! Most IT folks will implement one or two enterprise wide backup solutions in their career. The Appliance vendor does this for a living, perhaps having implemented thousands of such solutions. Who has a better chance of getting it done correctly in your shop? That was a rhetorical question.
Backup Solution Time-Line
All solutions have a similar set of steps required to implement:
1. Define specific business problem to solve. In the case of backup this is pretty simple: provide a reliable copy of all mission critical data available for restore in the event of data loss. This should include the loss of a single file or directory up to a complete data center catastrophe.
2. Define Recovery Time and Recovery Point objectives per data class.
3. Analyze existing technology offerings from a variety of vendors to determine which lend themselves to solving the requirements listed in steps 1 and 2.
4. Optional: Test some of these technologies in your shop.
5. Design a solution using the technology analyzed in Step 3 and 4.
6. Execute RFI, RFQ, RFP to solicit vendor input on price, etc.
7. Evaluate the responses.
8. Choose a solution or the parts of the solution.
9. Order the parts.
10. Wait for the parts.
11. Assemble the parts.
12. Configure the parts with an eye toward solving problems from Step 1 and 2.
13. Place the tested and configured solution in the IT infrastructure.
14. Install client and agent software on all participating systems.
15. Verify backups.
16. Test restores.
17. Done.
Steps 3 through 12 are where the wheel re-invention takes place. These activities have been done to death. Nothing your folks are going to do in these steps will be unique. And they will consume months of time in the best case and yield a result already arrived at hundreds of times. Oh, and by the way, repeating these steps didn’t contribute at all to furthering the objectives set forth in Steps 1 and 2.
How does buying an appliance help? The steps below show how much time is spent on each step and highlight how the appliance vendor can short circuit the timing.
Step 1 – Determine Objectives
Your IT Folks-Assemble stake holders in the organization to determine goals of new backup solution
Elapsed Time-1 Week
Your Appliance Vendor-Has done this many times and can facilitate discussions. Knows what questions to ask and how to get meaningful responses.
Elapsed Time-1 Week
Step 2 – Determine RPO, RTO per data class
Your IT Folks-Classify all the data in the environment and work with stake holders to determine what each recovery point and time objective should be.
Elapsed Time-2 Weeks
Your Appliance Vendor-Templates based on industry standards can be applied by the appliance facilitator to determine best practices per data class. Gain consensus.
Elapsed Time-2 Days
Step 3 – Analyze Technology
Your IT Folks-Read product literature, white papers. Discuss options with vendors and other IT shops that have implemented solutions of this nature. Determine how each technology applies to RP and RT objectives determined in Step 1 and 2. Determine which technologies require testing and may be applicable. Determine evaluation criteria for those requiring more analysis.
Elapsed Time-How long? Very difficult to determine. In many cases, shops get bogged down right here as they really have no idea the impact technology can have, positively or negatively, on the RP and RT objectives. They simply do not have enough experience using the various technologies to know how well they might fit. 4 Weeks or longer.
Your Appliance Vendor-Knows all the applicable technologies and can easily match them with the RP and RT objectives already established. Technology is treated as an arrow in the solution quiver. Has a selection of preferred, tested technologies to draw from.
Elapsed Time-1 Day
Step 4 – Test Technologies
Your IT Folks-Work with vendor to obtain hardware or software for test. Develop testing methodology with an eye toward solving a specific problem. Install and configure the HW or SW. Test and evaluate the results and decide if the HW or SW is applicable.
Elapsed Time-Again, a good place to get bogged down. Frankly, testing hardware and software can be fun and many IT folks enjoy it! It’s one of the few times where playing is encouraged. However, it doesn’t improve the company’s bottom lines. Unless of course technology evaluation is their business…2 Weeks or longer.
Your Appliance Vendor-Already tested. They know what works.
Elapsed Time-0 Days
Step 5 – Design a Solution
Your IT Folks-Determine which technologies should be used and design a complete solution based on these technologies. Determine interface types between the devices, the OS required, interconnect to the network and the SAN. Integration into the overall site infrastructure.
Elapsed Time-1 Week
Your Appliance Vendor-Based on RT and RP objectives and other information gathered during the pre-sale effort and by using best practices knowledge and “arrows in the quiver” in a pre-determined way, solution design is very quick. Experience counts!
Elapsed Time-1 Day
Step 6 – Determine Pricing
Your IT Folks-Get pricing information for each of the chosen components. In some cases, you might decide to bid these out to various vendors to get the best pricing. Weigh this pricing information against the project’s budget.
Elapsed Time-1 Week
Your Appliance Vendor-Develop a solution proposal.
Elapsed Time-1 Day
Step 7 – Evaluate Pricing Responses
Your IT Folks-Decide which vendors are the winners in the pricing wars. This can have an impact on the design so there is a possibility that a loop back to Step 5 is required. For instance the cost of the disk part of the solution is too high and thus more tape will be employed.
Elapsed Time-1 Week
Step 8 – Choose/Finalize the Solution
Your IT Folks-Probably involves presenting to management
Elapsed Time-1 Week
Your Appliance Vendor-Appliance vendor has pricing on the solution and is ready to present it to you.
Elapsed Time-1 Day
Step 9 – Order the parts
Your IT Folks-Cut purchase orders to a number of different vendors.
Elapsed Time-1 Week
Your Appliance Vendor-You cut just one purchase order to just one vendor for the entire solution. Once the appliance vendor has the order, they will begin to put your solution together.
Elapsed Time-1 Day
Step 10 – Wait for the parts
Your Appliance Vendor-Often the appliance vendor has your appliance ready to ship.
Elapsed Time-Less than four weeks
Step 11 – Assemble the parts
Your IT Folks-Unpack, rack, power, install, configure, etc., all the parts. Contact the various vendors when problems arise during integration of components from many different vendors. Weigh through the finger pointing inherent in implementations of this type.
Elapsed Time-2 – 4 Weeks
Your Appliance Vendor-There are no integration problems or issues. The appliance is a fully integrated, scalable solution. It arrives ready to plug in to the network.
Elapsed Time-1 Day*
Step 12 – Configure the parts
Your IT Folks-Determine how the various components of the solution, such as disk and tape, hang together. Configure the software to use these resources appropriately with an eye toward solving RP and RT objectives. Test the configuration. Tweak as necessary.
Elapsed Time-1 – 2 Weeks
Your Appliance Vendor-There are no integration problems or issues. The appliance is a fully integrated, scalable solution. It arrives ready to plug in to the network.
Elapsed Time-1 Day*
Step 13 – Install the solution
Your IT Folks-It’s been tested and works as designed. Now integrate it into the IT infrastructure.
Elapsed Time-1 Day
Your Appliance Vendor-There are no integration problems or issues. The appliance is a fully integrated, scalable solution. It arrives ready to plug in to the network.
Elapsed Time-1 Day*
*Steps 11-13 are accomplished in a single day.
Step 14 – Install Client and Agent Software and set up automated tasks
Your IT Folks-Push to all systems or install individually. Test the installation by performing a backup and restore.
Elapsed Time-Time depends on number of hands and number of systems. 1 – 2 Weeks or longer
Your Appliance Vendor-Same steps as without the appliance, but much more experience shortens the time.
Elapsed Time-1 – 3 Days. Longer for large environments. Might require multiple trips.
Step 15 – Verify backups
Your IT Folks-As clients are added to the backup infrastructure, ensure that scheduled backups are completing in a timely fashion and that they are correct and resolve errors.
Elapsed Time-1 – 2 Weeks to wring out all the kinks.
Your Appliance Vendor-Basically the same, but deep knowledge and experience shortens the time.
Elapsed Time-1 – 3 days and coincident with ongoing installation tasks.
Step 16 – Test Restores
Your IT Folks-Usually restores are tested when a restore is needed. This is probably not a good idea. Ideally, complicated restores are tested and re-tested to ensure they can be done in a timely manner. In some cases, a restore test will indicate a loop back to Step 5 and in even more extreme cases back.
Elapsed Time-1 – 2 Weeks, if done at all!
Your Appliance Vendor-Experience with different types of restores and deep knowledge and experience generally makes this step easy. An appliance vendor has seen most of the restore scenarios. Since the appliance approach shortened the overall time so dramatically, more time can be spent here. Here is where the real savings are gained.
Elapsed Time-1 Week
Bottom Line
IT Shop-Elapsed Time-24 Weeks on the low end and 30 Weeks on the high end.
Appliance Vendor-Elapsed Time-8 Weeks. And in many cases this can be shorter with a willing customer bent on solving the problem quicker.
So what does this mean? Without a doubt an in house developed and implemented solution will always take longer to implement and in most cases will not yield as good a result. The appliance vendor has so much more experience implementing solutions across a wide variety of environments, they can very quickly design and implement a solution that solves upwards of 95% of a business’s problems within a couple of weeks and achieve 100% within a month by providing one stop solution support.
If solving the backup problem is the number one goal, doing it quickly and correctly should be imperative. If time has no value, then taking four to five times longer is not an issue. However, if you factor in the cost of time and more importantly the potential cost of data loss during that time, implementing a Backup Appliance makes much more sense than re-inventing the wheel.
Comments:









