EMC Avamar v5.0 Storage Based Backup Implementation Closed     Case # 10048     Affiliated Job:  New Trier Township District 2031 Opened:  Tuesday, February 1, 2011     Closed:  Monday, February 28, 2011 Total Hit Count:  51075     Last Hit:  Tuesday, April 23, 2024 8:57:28 PM Unique Hit Count:  11017     Last Unique Hit:  Tuesday, April 23, 2024 8:57:28 PM Case Type(s):  Server, Vendor Support Case Notes(s):  All cases are posted for review purposes only. Any implementations should be performed at your own risk. Project: After discussion of the future of our backup strategy utilizing Backup Exec; we finally decided to move forward with an EMC Avamar solution. EMC performed an assessment of our network and determined we would be best fit with an Avamar "DS7" grid consisting of a Utility node, Spare node and 5 Data nodes. Under a new platform of Avamar v5.0, each data node includes 3.3 TBs of available storage space. Since we have nearly virtualized our entire server environment in our array of 5 host servers running the latest and greatest ESX 4.1 vSphere (which we have since learned of EMCs plans to do away with ESX and shift to ESXi for version 5) along with the storage of our infrastructure on an EMC CX-120 - we are good to move forward with a storage based backup solution. We hired the help of a certified partner of EMC to perform the major install; however, the partner consulting company encountered a great number of difficulties with the hardware provisioned and after a week of troubleshooting, ended up replacing two nodes along with deploying EMC teches to return the grid to a fresh from scratch state. Therefore, in our planned week worth of time we were only provisioned two actual days, "Day 1" for install and "Day 2" to briefly go over the high level details & job configuration. So below are lessons learned through work I have had to perform to troubleshoot and self-educate various implementations either left uncompleted or missed during the "install." The consultant performed the install on the nodes individually, then through the "wizard" brought the grid online into a collective state; set the Avamar utility node to trust the root certificate of the vCenter server, setup the initial Avamar proxy (which captures the image level backups of a virtualized machine); imported the VM server environment into Avamar; created the requested datasets, schedules, retention policies; explained the various agents to be installed for File based (rather than image based backups which permits file exclusions, restore permissions with files etc.), SQL, Exchange; LDAP authentication into our A.D. infrastructure and SMTP notifications of "Call Home" to EMC as well as sending us server notifications. What the consultant failed to explain or setup were a couple items we then figured out on our own. When he finished (on "Day 2") we began an "On Demand" on a couple backups to begin while keeping all others in a disabled state. And before the consultant left we had arranged a return date for him to explain the "how to" restore, review the backed up jobs (success/failure), Avamar reporting, change root/admin passwords, shutting down the grid, backing up of external NAS resources, setup of additional Avamar proxy servers (each proxy performs a single thread - to enhance backup time EMC suggests a proxy per VM host), review of the "System State" File Agent backups on Windows 2003 (Which we discovered were NTBACKUP bkf files being placed on the C volume and in some cases were consuming what little space was left) To address the above, a great majority had to be performed on our own - simply because our consultant had already spent the 5 days budgeted on our contract building the original units only to find the failed hardware components and therefore had to spend an additional two days to reconfigure the two new nodes with the original 5 other nodes all of which were returned to factory settings after repair. Because of this, the consultant neglected to return on the date arranged and instead the questions were briefly explained through email rather than demonstrated. Action(s) Performed: Total Action(s): 4 Action # Recorded Date Type Hit(s) User Expand Details 10165 2/28/2011 2:37:07 PM Server 3785 contact@danieljchu.com Image Level File Level Restores Some of the lessons learned, most i  More ... 10166 2/28/2011 2:36:07 PM Server 3459 contact@danieljchu.com Add Additional Avamar Image Backup Proxies The creation of addition  More ... 10168 2/28/2011 2:35:07 PM Server 3280 contact@danieljchu.com Image Level Backup Failures Image level backups on VMs that have a   More ... 10167 2/28/2011 2:34:07 PM Server 3573 contact@danieljchu.com Backup Performance A couple things to mention, when performing back  Collapse ... Last Hit: Tuesday, April 23, 2024 8:57:20 PM Backup Performance A couple things to mention, when performing backups of Windows 2003 using the file agent based backup, it also offers the ability to backup the System State (using NTBACKUP); our consultant suggested turning this off because a couple of our physical boxes configured for backup were running out of C drive space after the System State backup was created. The "SystemState.bkf" file is created at: c:\Program Files\avs\var. Instead, we are using the "Backup the system profile" option which is specific to Windows 2003. Windows 2008 uses the VSS option to backup it's system state. This is all true of Avamar v.5... I am also told if you wish to redirect the Windows 2003 system state backup (know this requires the seaprate Avamar Windows 2003 System State agent) you can set an advanced option in the Avamar client for the data set using: systemstatefile = "Path\systemstate.bkf" - "PATH" being on the local server where the systemstate.bkf file should be placed. In backing up of our main file server, the file agent level backup took 88 hours on its initial backup of 4.1 TB with about 4 hours on each follow up backup. In comparison of file agent level versus image level through a Avamar proxy, this same server took 103 hours to backup 7 TB of image data also with 4 hours on each follow up as well. Two files that also exists at c:\Program Files\avs\var is a "f_cache.dat" & "p_cache.dat" file, there appears to be a 512 MB limitation on these files which I am told can be expanded. On our largest file server it is so far 360 MB. Just something to keep an eye on.