7. January 2010 by Myke.

This is an follow up and update to the following post;  http://homerun-networks.com/2009/12/10/commvault-simpana-8-saving-lives-disk-space-and-relieving-stress/

We have been running Commvault Simpana 8 for 4 months and to be quite honest, it has been flawless and great.  As I mentioned before, we were concerned with future growth and what it would cost us for hardware for backups and more importantly, could we actually get backups the way we needed them?!?  Well, we nailed everything and then some.

We have had to restore multiple files including Exchange (single message and multiple messages), Exchange store (testing purposes), Server 2008 DC (testing purposes), VMWare virtual server (testing purposes), SQL database, Server 2008 system state (testing purposes) and multiple files on file shares.  Every single restore took less than 5 minutes except for the testing recoveries.  The testing recovery is part of an on-going plan to prepare for a major project, but it was still rock solid and flawless.

Domain rebuild recovery - We are in the planning process of re-building the entire domain and infrastructure of our company and I have begun the testing of Server 2008 recoveries and disaster recoveries.  So far I have tested recovering a 2008 domain controller after deleting multiple users and groups and replicating the change.  Easy as pie my friend, everything went into place and the replication took place and the domain was back up and running in minutes.  This domain rebuild has allowed me to test for just about every disaster possible and to document exactly every step in the case I get hit by a bus/train and the boss man has to take over for me.

All in all I could not be happier with our choice of moving our backups to Commvault.

*We received some e-mails in regards to Commvault and we have included those below*

What was your installation process like and how much time did it take you to convert from Symantec to Commvault?
-  The process itself was very easy and simple.  We disabled all the Symantec software and services and installed Commvault.  The install itself took about 2 hours, which included getting all the clients installed.  Once the suite was up and running we set the backup policies and that took another 2 hours.  All in all, it was smooth and very easy.

Did Commvault pay you for this article? (we got about 15+ of these emails)
-  No.  I firmly believe that sharing IT/IS knowledge with others in the industry is key to making all of our lives easier.  If I find something that rocks, I will tell everyone that wants to listen.  If I find something that sucks, I will tell everyone that wants to listen.  Any product that makes life as a systems engineer/systems admin/network engineer easier, why not share it with others?!  p.s. If Commvault is reading this article…I wear XXL shirts, love fast Italian cars, good beer, steak dinners and Amazon.com gift cards.

What do you think of the Commvault deduplication?
-  Do you remember Smeagol and his precious?  Well consider me being Smeagol and Commvault deduplication being my precious.  We must haves it, haves the precious we must.  We are currently getting a savings of 90.25% on physical storage space.  Or even easier to understand, we are getting an average of 13.4TB of backup data on 1.3TB of physical space.  Must haves the precious!

As always, if you have any questions please feel free to ask away.

posted by:Myke Reinhold

Posted in Backups | No Comments »

21. December 2009 by Myke.

Everyone knows that surfing the web can/is/will always be a dangerous thing to do.  As a systems engineer/administrator we always have the task of protecting end users who are educated on the security risks and the end users who have no clue at all.  No matter how much knowledge you have as an end user you can always get hit by doing something very innocent on the Internet.  But what can be done to help prevent this?  For myself, I registered with the elite group over at MalwareURL and started importing their database into my firewall.  Now this does not protect me 100% but it sure helps to say the least.  To date they have 33,944 domains listed and 8,787 IP addresses listed.

Here are the two best reasons to check out MalwareURL.  First of all, you can use their information to infect a virtual/physical machine to practice clearing out nasty little bugs and teaching yourself how to reverse engineer problems.  Just remember to infect a test machine, not a production box.  Second, you can also report any sites you find that are not listed yet.  This helps build the database and the best way for us to protect ourselves is to share information with each other.

Posted in Security | No Comments »

10. December 2009 by Myke.

I recently implemented Commvault version 8 with a company that was running Symantec Backup Exec.  The Symantec software was having trouble backing up the Exchange mailboxes (Exchange 2007) and this was a mission critical issue for the executives.  The Symantec software was also having a difficult time backing up Server 2008 and Citrix Xen Server.  After numerous calls and emails (18 calls and 22 emails) to tech support it was still not resolved.  So now the company was missing a massive amount of data and could not get the software to backup to an IP NAS device (Seagate Black Armor).  In the end this would have spelled disaster for the IT team and there would have been some very bent employees with a very bad taste in their mouth for the IT staff.  I made one suggestion…Commvault.  I used it in the past as a stand alone and in conjunction with Exagrid disk storage.  I loved it very much and wanted to get it in house ASAP.

First step was getting the management staff on board after showcasing it for the Director of IT/IS.  The Director loved it and only had one thing to say.  “Prove it in the first month of use and I am sold forever.”  The budget was approved and the purchase was made.

Second step was scheduling the fun of turning off Symantec and kick starting Commvault.

It was a warm fall day in 2009 and Myke the Master Geek went to work in his workshop.  I started by disabling the Symantec service on all servers and disabling the software on the backup server.  Next up, getting my Commvault Media Agent and Commserve on-line and ready to go.  The Media Agent was a new Dell R710 loaded with Server 2008 64 bit with a Powervault connected to it.  Then we added 2 Seagate Black Armor 4.5TB devices for the disk storage.  The Commserve was actually a VMWare virtual server loaded with Server 2003 32 bit.  Once the devices were loaded, connected and talking…it was on to deploying the agents on each server to be backed up.

The ultimate goal was to have about 4 weeks of backup data on disk and then a weekly full backup on tape.  We had a decent size of data that was backed up daily so we purchased the deduplication license with our Commvault software.  This would allow us to deduplicate our data and use less disk space for our backups.  With that in mind we expected a disk savings of about 50% to 60%.  We were wrong and wrong big time.  After running the Commvault backups with deduplication for about 2 months, we were getting a disk savings of 89.88%.  We were storing 10.074TB of data on 1.019TB of actual disk space.  That was saving us 9.055TB of disk space.  We were very excited about this as this gave us a great amount of room for growth and gave us a baseline to look forward to in the future.  Needless to say, the Director of IT/IS was very happy.

So with backups running to disk and tape now we had to verify that everything worked as planned.  So I began to test restores of data.  I started by restoring data from disk back to file servers, mailboxes and SQL servers.  Everything worked as planned and with great speed.  Now I began the tape restore process.  I selected a file and the software came back and told me what tape it need and bam, there it was…restored.  I tested about 35 different files ranging from SQL to Exchange to general Office file types.  Everything worked as planned and promised.

To this day everything has worked perfect and we have been very happy with our backups since.  This has saved on restless nights of sleep having nightmares about backups and restores and it has dropped our stress level by a huge margin.

Thanks Commvault!

I would also like to say that the new Dell R710 server runs like a champ and is a solid server.  We are also very pleased with our low cost NAS devices from Seagate, Black Armor 440.

p.s.  If you are from Symantec or really like Symantec and find this post to be offensive…good.  That is exactly what it was meant to be.  Once Symantec bought Backup Exec, the software fell apart and has fallen way behind the times and needs of the IT/IS world.

posted by: Myke Reinhold

Posted in Backups, Storage | 4 Comments »

24. November 2009 by Myke.

This post contains information on how to edit and modify your Windows Registry.  It is always recommended that you take a backup of the Registry before editing any of the values because any improper editing can cause strange behaviour and at worst could even corrupt your operating system completely, requiring you to re-install Windows.

We encourage you to try out the registry changes,  but only if you know what you are doing and if you do it with care.

After building a brand new Windows 7 ENT x64 laptop I ran into some issues.  The issues started shortly after finishing some updates.  Explorer.exe kept crashing every time I would right-click on an icon or try to use anything that used explorer.exe.  After searching the web for hours I found nothing that actually resolved the issue.  Pretty much everything out there pointed to doing a full restore or a clean installation.  I also found a couple posts that said once they deleted their profile and rebuilt it, everything worked.  Each of these is true but why waste the time and effort.  I am not sure about you but hearing from a Microsoft employee and having them tell you to do a clean install because it is hardware related or due to 3rd party software is getting real old.  Well you are in luck folks, because I have a solution that does not harm the machine and it can be done within 2 minutes.

Here is the error we were getting in our event logs;
The program Explorer.EXE version 6.1.7600.16404 stopped interacting with Windows and was closed. To see if more information about the problem is available, check the problem history in the Action Center control panel.
Process ID: 810
Start Time: 01ca6d1f1aca747c
Termination Time: 0
Application Path: C:\Windows\Explorer.EXE
Report Id: 3fe9620d-d913-11de-8a55-00242cbe9d84

I ran every application I had that would point me in a direction of figuring out what was causing it and found nothing.  I decided to go through the 34 updates I had applied the day before and found an issue finally.  One of the updates was forcing the CEIP to execute.  *Dear Microsoft, why place something like this in an OS when you know it causes problems?*

The cause of the Windows Explorer crash is related to the SQM Client, which is part of the Customer Experience Improvement Program (CEIP). Under the default setting, where MachineThrottling is enabled in the registry, any calls to WinSqmStartSession in ntdll.dll file will cause Explorer to crash, or Windows Installer installation to fail.

So instead of waiting for a hotfix or an update from Microsoft, just remove and delete the MachineThrottling registry entry from system registry. The MachineThrottling registry entry is located inside the following registry key: HKLM\SOFTWARE\Microsoft\SQMClient\Windows\DisabledSessions

*NOTE*  If you do not know what you are doing within the registry, stop and do not proceed.  Ask someone for help that knows what they are doing and can recover your registry if a failure occurs.

To make it easy you can just create your own little batch file with the following command;
reg delete HKLM\SOFTWARE\Microsoft\SQMClient\Windows\DisabledSessions /v MachineThrottling /f

At this point you can close the registry and right-click on your file or icon and you should be good to go.

posted by: Myke Reinhold

Posted in Scripting, Registry, Microsoft, Desktops, Laptops | No Comments »

6. October 2009 by Myke.

Going back to an old school issue.  What do you do when you switch out an end user’s computer and they freak out because all of their auto fill addresses in outlook are no longer there?  Easy, switch over their .nk2 file to the new computer and call it a day.

Do you miss the convenience of Outlook automatically completing people’s names as you begin to type them on your new computer? Are you upgrading to a new computer and don’t want to lose all the names stored in your Outlook AutoComplete feature? Wouldn’t it be nice if Outlook installed on your new computer just “remembered” the names and filled them in for you?

You can copy the names in AutoComplete from your old computer to your new one.

Copy the names in AutoComplete to another computer

Important  You must exit Outlook before starting the following procedure. The names will be included in AutoComplete when you restart Outlook.

1. On the computer with the saved AutoComplete names, go to drive:\Documents and Settings\user name\Application Data\Microsoft\Outlook.Note  Depending on your file settings, this folder might be hidden. To view the files in this folder, do one of the following:

   Microsoft Windows XP

   1. Click Start, and then click My Computer.
   2. On the Tools menu, click Folder Options.
   3. Click the View tab, and then, under Advanced settings, under Hidden files and folders, click Show hidden files and folders.

   Microsoft Windows 2000

   1. Double-click My Computer on your desktop.
   2. On the Tools menu, click Folder Options.
   3. Click the View tab, and then click Show hidden files and folders.

2. Right-click profile name.nk2, and then click Copy.Tip  You can copy the file to removable media, such as a floppy disk or a CD, and then copy the file to the correct location on the other computer. Or you can attach the file to an e-mail message and send the message to yourself. On the new computer, open the attachment in Outlook, and then save it to the correct location.
3. On the computer where you want to populate the AutoComplete feature, copy the file to drive:\Documents and Settings\user name\Application Data\Microsoft\Outlook.
4. If the Outlook user profile name is different on the computer where you are moving the .nk2 file, you must rename the file with the same Outlook user profile name after you copy it to the correct folder. For example, if you move Kim Akers.nk2 from the original computer with an Outlook user profile name of Kim Akers, and you copy the Kim Akers.nk2 file to the new computer, you must rename it with the Outlook profile name being used on the new computer.
5. When prompted about replacing the existing file, click Yes.
6. Open Outlook to view changes.

source: Microsoft Office Online

Posted in Technical Questions, Microsoft, Desktops, Laptops | No Comments »

30. July 2009 by Myke.

As simple and easy as this task is, we received about 10 emails over the last 2 weeks asking, “I have my users set up to use a mapped drive for their personal data stored on the network.  How can I make that available to them while they are not on the network?”

Easy, open up My Computer and right-click on the mapped drive and select “Make available offline”.  That’s it.  Once the wizard pops up you can detail what you want the offline files to do and once you complete the wizard, it will begin the sync of the files to the local PC.  You are now done.  Cheers.

Posted in Technical Questions, Networking, Microsoft, Desktops, Laptops, Storage | No Comments »

24. July 2009 by Myke.

Enable-ExchangeCertificate : The certificate with thumbprint XXXXXXXXX was found but is not valid for use with Exchange Server
(reason: PrivateKeyMissing).
At line:1 char:27
+ Enable-ExchangeCertificate <<<< -Thumbprint XXXXXXXXX -Services “IIS”

The above error is a result of a glitch with Exchange 2007. This issue does not happen all the time as it is completely random, but when it does happen no certificate can be installed or removed through the Exchange Management Shell (EMS). For whatever reason it may be, the system forgets where it placed the Private Key or the certificate store is damaged.

Repair Damaged Certificate Store:

1) Open MMC (Microsoft Management Console) to the Certificate Manager (Certificates Snap-in) for the Local Computer account.
2) Double-Click on the recently imported certificate (It will be missing the golden key).
3) Go to the Details tab.
4) Click on the Serial Number field and copy down that number. (Leave window open)
5) Open up the command prompt (DOS Prompt — CMD.exe)
6) Type: certutil -repairstore my “SerialNumber”( SerialNumber is that what was copied down in step 4.)
7) After running the command, go back to the MMC and right-click Certificates and select “Refresh”.
One should now see the golden key associated with the certificate.
9) Double-check in the Exchange Power Shell with: Get-ExchangeCertificate

Alternatively if the above does not work try the following:
Note: Follow these steps if running Windows Server 2008 only

1) Open MMC (Microsoft Management Console) to the Certificate Manager for the Local Computer account. (Certificates Snap In)
2) Look in the Personal section of the Certificate Manager and there should be icon(s) without a little golden key. (Those with the key have the private key bonded to them.)
3) Delete the icons without the golden key.
4) Go back to the EMS.
5) Run the Import-ExchangeCertificate and Enable-ExchangeCertificate in one line like so: [ Import-ExchangeCertificate -Path c:\exchange.comodo.com.crt | Enable-ExchangeCertificate -Services “SMTP, IMAP, IIS, POP” ]
*** Please modify the command according to your needs. ***

Posted in Security, Exchange, Microsoft | No Comments »

14. July 2009 by Myke.

“I cannot put my Citrix server into install mode.  It keeps saying I am not an administrator and my account is a domain admin.  WTF gives man?” 

Over the last few weeks we have received multiple e-mails asking why they cannot get a Terminal/Citrix server in install mode.  Each time they do this they encounter the following error, “Only members of the Administrators group may enable Install Mode”.  The problem is not due to your account and the privileges it has.  These new Operating Systems have elevated security as compared to Server 2003 and Windows XP.  If you just right-click on CMD.exe (located at C:\Windows\System32) and select “Run as Administrator” and then place the server or machine into Install Mode (change user /install) you should be fine.  This is annoying to a point but at the same time a nice security feature.  As always with Microsoft, it irritates us at first but we soon learn to do it out of habit.

Note - If you have renamed and/or disabled your built-in Administrator account, you can still run the task above.  If you still have any questions please feel free to let us know.  Cheers.

Posted in Citrix, Technical Questions, Microsoft, Servers | 1 Comment »

11. July 2009 by Myke.

…an office near you.  Are you prepared for it?  Can you handle it?

Posted in Funny, Microsoft | No Comments »

24. June 2009 by Myke.

There I was living the life of luxury with my backups running via CommVault software over to my ExaGrid disk storage and then to tape.  Backups were fast and the deduplication was AMAZING!  I basically set it then forgot about it each day.  I would get my daily reports telling me “hey, your backups worked of course and they are still kicking butt and taking names”.  Then came D-Day for my career at that job.  I walked away proud of my environment and what I left behind.  I was proud of where I took my job to, technology wise.  Servers went from white box to HP, server software went from 2000/2003 to 2003/2008.  i created virtual server after virtual server.  I put in an HP Blade Enclosure and all power was controlled via APC.  It was a true Enterprise setup for an Enterprise company.

I took the next 4-5 weeks pulling myself to the conclusion that I will never work on that equipment again.  I started with a very small group with my new career that has potential to become the largest Gold mining/production/exploration/reserve companies in the World.  Backups are controlled by an unnamed software vendor and it is not all that great.  The backups are about 30% fail to 70% complete each day and they are slow.  They eat tapes like there is no tomorrow and it is only going to get worse each day as we grow.  So what do I do to make my new job run as streamlined as my previous job?  How do I make backups a non-issue?

Simple, I work a budget that allows me to move forward with each piece one step at a time.  My first goal of 2010 will be to get ExaGrid in house to help create a disk based backup solution with some serious deduplication abilities.  The deduplication process will basically take my backups and compress them into smaller chunks of data (sorry for the non-IT explanation).  Take for instance, you have about 20TB of backup data and you run it through the ExaGrid…you will get about 15:1 (based upon my very own first hand knowledge) on space usage.  What does that mean to the non-IT person who is in charge of reducing backup costs???  That means I will get about 15TB worth of raw data and have it backed up on an ExaGrid using 1TB of actual disk space.  That’s right folks…15:1 on an average and that is being generous on the small side.  I have seen deduplication results close to 60:1 and 70:1.

Some folks have asked if I sell ExaGrid or CommVault or if I get paid by them or receive free hardware…no.  Like we have said before, we share the truth with no biased answers or results.  We want what works best for the IT World and truthfully, ExaGrid is the King of disk based backup solutions.

Links to previous ExaGrid posts.

http://homerun-networks.com/2008/12/17/exagrid-announces-customer-focused-enhancements/

http://homerun-networks.com/2008/07/22/exagrid-deduplication-update/

http://homerun-networks.com/2008/07/22/exagrid-announces-enhancements-with-latest-version/

http://homerun-networks.com/2008/07/21/exagrid-and-commvault-a-perfect-backup-solution-update/

http://homerun-networks.com/2008/07/21/exagrid-and-commvault-a-perfect-backup-solution/

p.s.  Dear ExaGrid - It would be very nice if I could borrow a couple 1TB units for about 1.5 years or so.  I promise to love and care for the units at all times. 

Posted in General Hardware, Servers, Backups, Storage | No Comments »