SEPM Database Fun

Tuesday morning I received an email no Symantec Endpoint Manager admin wants to receive

From: [email protected] [mailto:[email protected]]
Sent: Tuesday, September 27, 2011 12:13 AM
To: Roger
Subject: Database is down

Message from:
    Server name: asdfasdf
    Server IP: x.x.x.x    
The Symantec Endpoint Protection Manager database has gone down and needs immediate attention.

I went through several likely candidates in the Symantec KB but couldn’t find anything to fix the issue.   The database wouldn’t start.   As a side note, has anyone else had issues with many search results in the Symantec KB beign a “file not found”?

I ended up reinstalling SEPM and restoring a previous backup because I couldn’t get anything else to work.

The fun didn’t end there.   The next day at the same time (midnight) the database died again.   This time I called support first thing rather than after me trying many solutions.   It was the same as the day before.   Really nothing they knew how to do with the database down.   I did the same uninstall/reinstall database restore to get services back for the end users.   After hours, I installed from scratch and configured much of it by hand.    If you find your database backups are corrupt and need to do this.

1.   Export all the policy files and any other setting that is exportable.
2.  Make sure your configuration is up to date.   There are a lot of screens in SEPM but you’ll be glad you screenshot every last one of them and kept it up to date.
3.  Even without the database, you can use the recovery file so your clients are still able to check in.   Otherwise they’d need a reinstall or a sylink.xml.
4.  In the tomcat/etc directory under the SEPM install, edit conf.properties and change scm.agent.roupcreation to true.   Restart SEPM.   This allows clients to create the groups they were previously assigned to.   Otherwise all clients would end up in the default group.   Even after creating a new group, the group ID wouldn’t match and you would be stuck moving all clients manually.

I spent three long nights on this issue.    I was very glad to have “Essential” support so I could get support on the line outside business hours.    Hopefully this was a one time issue.   I suspect the database was a little hinky after the upgrade to 12.1.

6 Comments

  1. I just experienced the same exact thing this week. After rebuilding, the next day it happened again just as it did to you. I did not import the database after the second time. I did not perform your step 4, but the clients are still coming back into their original groups, so there must have been another setting prior to this mess that is allowing that to happen.

    • Interesting. In my environment the clients up in the default group if I didn’t either restore the database or make that settings change.

      glad you got your environment back working.

  2. I’ve encountered the “file not found” error on the Symantec KB, but I just figured it was an issue with firefox with noscript or blocking some cookies or something.

  3. HI all,

    Had the same e-Mail. Simply rebooted the server to resolve this issue but I do not trust it. So far I have only had one instance of this alert. I will back up the policies and database in preparation for the worst. Just in case….

    Thank you all for your informative submissions

    Have a nice day !!!!!

  4. Had the same thing happen when I upgraded to SEPM 12.1. We ended reinstalling 11.6 and restoring the DB backup.

    I did notice you said this happened around midnight. I started getting those emails at the same time. I think I’ll talk to Symantec support and see if there is something within SEPM around midnight that would cause the corruption. Seems like more than a coincidence that we all started getting the emails around the same time.

    Can anyone remember during setup, I thought it said it would preform so DB maintenance around midnight. I have done some test installs without upgrading and havent run into DB corruption, so I’m wondering if tit might have to do with the conversion process during an upgrade.

    Luckily, I had done a disaster recovery backup prior to the upgrade so reinstalling and getting 11.6 back up and running wasn’t too bad, except for sending all day on the phone with Symantec. 😉

    I have since gone through and exported all my policies and firewall rules as well as making sure I have the server certificates and anything else I can think of!

Comments are closed.