Posts tagged ‘Backup’

Over Logging

We had a big storm roll through on Wednesday afternoon. About 6 hours later I lost my cable modem connection. Since the storm was long over, I’m guessing the power backup at the cable modem head end ran out.
I was pretty fortunate compared to some people at work didn’t have any power at home. The power was still out Friday so I borrowed a EVDO card just in case. That turned out to be unnecessary as my cable modem came backup at 4pm.

Today I read this story on the front page of Saturday’s Washington Post about one families travails from not having a internet connection (oh yeah, they lost power as well.
Bethesda eighth-grader Jacob Rasch could not do his history homework assignment on the Compromise of 1877 on Thursday because, he said, he couldn’t look it up on Wikipedia.
Wow, that’s right up there with “my dog ate my homework.” Doesn’t he have a textbook? Doesn’t he have an encyclopedia? How about a CD ROM Encyclopedia. How about a library.
His mother could not e-mail health forms so Jacob can play baseball in high school this fall because severe thunderstorms that rolled through the Washington region this week took down the family’s power and their Internet connection.”
Oh the humanity.
And his father couldn’t fix the generator outside the house because he couldn’t visit HowTo.com to find out how to clean the carburetor so that the generator would spring to life and power, among other things, the wireless router to their computer network
Look on the brightside, at least he didn’t accidentally kill them with carbon monoxide.
So the Rasch family packed a laptop Thursday evening and moved to a hotel, where they could log on and feel plugged in. .
I thought about heading over to Wegmans, Panera or McDonalds for some free wifi, but decided I could go an evening without Internet. If I were out of power, I admit like these guys I might go somewhere else for a hot shower, but for Internet…these people would fit in with the South Park episode linked.
“We couldn’t connect to the outside world without the power and the Internet,” Jacob Rasch said. “We had no idea what was going on.”
Better upgrade your disaster readiness kit to include a radio with batteries and a hand crank.
Now this does make me think that replacing my land line with VOIP or a cell might be a bad idea if I want 100% uptime.

The Case of the Backup Software DoS.

Our vulnerability scanner is causing the server backup software’s we use on to crash.
After examining a crash dump, a developer for the backup software replied

“Looking at the logs it we are getting some corrupted packets and that is causing the to try to allocate huge memory and that is the reason for the failure.
Does this security scanner corrupt our packets to test some of its features? If yes then they will have to stop it.”


While not sending corrupt packets would stop the crashing, I’m not sure a bad guy would be so kind as to respect at request. I also wonder if there is a remote exploit in this defect.
To take it out of the realm of the vulnerability scanner, I used nmap’s service fingerprint option to crash the service. Reviewing the packets with wireshark shows that nmap with the -sV option set is also throwing a corrupt packet. The hardest part in reproducing this is the backup software not staying on a predictable port.
Vulnerabilities in backup software are frequently targeted. Backup software often runs with full admin or system rights. Exploiting vulnerabilities in backup software can lead to information disclosure or an attacker fully compromising import servers. SANS has backup software vulerabilities in the SANS Top 20 list.

The Case of the New DC and the LM Hash

While reviewing the results of the latest windows domain password audit, I noted that there was an increase in the number of lanman hashes stored. We had two domain controllers blow up recently and they had to be rebuilt from scratch rather than restored from backup. I correctly figured that on one or both of those DCs the disable lan man setting had not been implemented correctly.
I knew that on a Windows 2000 domain controller this setting needed to be added manually. The Group Policy setting only effects XP and Windows 2003 computers. I didn’t remember what the registry setting was so I sent to http://support.microsoft.com/kb/299656,
I read

To add this key by using Registry Editor, follow these steps: 1. Start Registry Editor (Regedt32.exe).
2. Locate and then click the following key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa
3. On the Edit menu, click Add Key, type NoLMHash, and then press ENTER.
4. Quit Registry Editor.
5. Restart the computer, and then change your password to make the setting active.

In my haste, I forgot about the difference between a Key and a Value. I saw that the domain controller had HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa with Nolanman hash set to dword value 1. I compared that to the other domain controllers and didn’t see why that domain controller wasn’t working.
It took a second to realize that was the Windows 2003 setting set by Group Policy. For Windows 2000, you need to go to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa and create a key of nolmhash. That isn’t the same thing at all. A quick check verified that this setting was missing on the new DCs and existed on the old DCs. We set the registry key and scheduled a reboot.

Shmoocon 2008 Day 2

Here are some notes from Shmoocon day 2. Today was a return to the traditional Build It, Break It, and Bring it on tracks. Here are some notes/summaries from the sessions I attended. It was another fun day.
Active 802.11 Fingerprinting, Bratus, Cornelius and Peebles
How can you identify if an access point is legitimate or rogue? Does two way RSA crypto solve the problem of a rogue AP? The speakers would argue that if you are communicating with a rogue AP, the use of certificates could actually cause more information to be given away to the rogue. You could certainly be exploited in your communication as well if your wireless drivers have vulnerabilities.
Just as with OS fingerprinting through TCP, the wireless protocol can be abused to send unexpected traffic to the AP and fingerprint how it responds. They built a tool called Baffle using Ruby to perform this test. They were able to verify that the access point was using the driver that is expected.
If you’re expecting a linksys AP and I set up a rogue linksys AP, this isn’t going to help you, at least from my understanding of the talk. An audience member asked if this could be used with adhoc (client-to-client) connections as well. It cannot be used for that because the APs are much more chatty and have more negotiation.
The remainder of the time was a presentation on access point hiding. I did not catch the presenters name. Basically anything that has some room inside and has sufficient power could be refashioned to contain an AP. This assumes that you need to be stealthy about placing a rogue AP in the first place. The take home for me from this section of the talk was the question, “if an AP enabled itself at 2 am (either to let the hacker in, or to move some data out) would you catch that.”
Smarter Password Cracking; Weir, Glodek
Not a lot new here.
Password cracking is getting tougher. Sometimes users are forced to pick better passwords. Often developers are throwing in a salt or hashing multiple times. A salt makes a precalculated table attack difficult. Multiple hashes attempt to increase the calculation penalty when trying a offline password attack. For example while Word’s password mechanism was once trivial to break, Word now uses 5000 SHA1 and a huge salt.
In the last year or two several password troves have become available to all. In the past researchers didn’t have a way to report on user password selection. After a myspace phishers collected passwords leaked, researchers now had a large collection of legitimate passwords. Many of the passwords were tremendously weak and thus not comparable to the enterprise password.
When setting out to crack passwords, it is helpful to figure how how the users select the passwords. This allows the cracker to have a better chance at success.
I was hoping to take from this lecture a script to analyze a list of passwords and display the tendencies found. I would like to be able to easily run a report that says: 30% of users passwords were reveals in testing. Of those 90 percent were in the format Aaaaaa11 (A=upper, a=lower, 1=any number). I don’t see that script on his website, I’m going to check back later.
They’re hacking Our Clients, Why are we focusing only on servers; Beale
This talk had two major sections. The need for patching clients, and a poor man’s way to find clients that need patching.
In the first section Beale said that in pentesting engagements they now attempt to get to the internal network through client side attack. Often they are limited by engagement rules to the computers belonging to IT staff or security folk. Even with this set of users they are consistently able to perform attacks on the browser, mail client, Office, Adobe Reader, etc. Core Impact and Metasploit are two tools mentioned.
The bad guys moved to client side attacks years ago. Their biggest problem is managing all their owned boxes.
The question is asked, isn’t this just social engineering. There are two responses to this. No, sometimes attacks autorun without user interaction. Yes, but the human firewall is imperfect. Even the most educated users get fooled. Its still appropriate for a pentest.
Comment from the audience – Once it reaches the user, freakin game over.
The attackers only have to find one vulnerable human or one vulnerable software install.
Isn’t this a patch management problem, Beale asks rhetorically.
He says yes, but not every organization has patch management.
Also patch management, needs know about every system to patch it. It needs rights. It often doesn’t patch every product. Most people don’t have that complete an inventory of what is on their network.
To address these issues, the speaker proposed using User-Agent strings to self identify vulnerable systems. That information could be collected in HTTP proxy logs, and email servers. Vulnerable clients could be denied further access.
While you could do further things such as implement something like the Master Reconnaissance Tool to gather browser plug-ins, there is still vulnerable software that you don’t address in this way.
Another idea is to look at the metadata for recently created files on your fileserver, sharepoint, in email. Apparently you can determine the version of the software used to create the document. A vulnerable version and a recently created document equal a problem that needs to be addressed.
Since I do vuln scan all online systems, and I do have a patch management system, the second part of the talk wasn’t as interesting. It seemed like a lot of work just to catch a small number that missed the patch management and vuln scanning. I do see the usefulness in a University or other similar environment.
VOIP Hopper; Ostrom and Kindervas
This was strong talk demonstrating their new version of their voiphopper program. Most people outside that room think that a vlan is a security separator. The talk showed how easy it is to get onto the voice vlan. In IT there is also a low awareness of VOIP threats. People think, “you can’t access corporate data from an IP Phone.”
voiphopper now includes a Cisco Discovery Protocol generator making it really easy to pretend to be a VOIP phone.
Mitigation-
1. Use Cisco’s phone CDP Security provided in 12.2.36 SE. This requires a phone to have power or it will shutdown the port. (one wonders how that would work in my case where a bad blade wasn’t providing power for some ports, and I was given a brick for my phone instead of using power over ethernet).
2. MAC address filtering
3. Disable the pc port on the phone. (this is the lobby phones that should be have a pc plugged into them).
Got Citrix? Hack it!; Gupta
One audience member correctly asked for less IE vulnerabilities and more about Citrix I agree. The vulnerabilities presented all existed because Windows was not secured for the role the system was playing.
Gupta has a good point that people think putting something behind Citrix is equal to securely serving it.
We did not get to see a couple of demos because the wireless network was down during this session. I’d recommend either not relying on a unreliable medium for a presentation or have a video backup. We were left with a session cut short, and a feeling of disappointment.

IRPStackSize

I have a whole bunch of Windows XP sp2 systems that give me an error when I attempt to connect to their c$ or admin$ shares: “Not enough server storage is available to process this command.”

The remote system’s event log records: Event ID : 2011 Source : Srv Description: The Server’s configuration parameter “IRPStackSize” is too small for the server to use a local device. Please increase the value of this parameter.

I checked a couple of Microsoft Knowledgebase articles and did a bunch of googling searching the Internet. It seems that a lot of people have latched onto http://support.microsoft.com/kb/177078 as the only cause and concluded if you have the error message “Not enough server storage is available to process this command” than it must be Symantec’s fault. As I searched, I found person after person with this error message being told they needed to uninstall symantec. The person with the issue responded they had another antivirus product, they never had Symantec installed and they still had the issue. The Symantec blame had specifically to do with NAV 7.6 and 8 which hardcoded the IRP stack size to 8, roughly half of its default value in Windows XP. That doesn’t have a lot to do with the issues i’m having. I dont have that registry value at all.

http://support.microsoft.com/kb/285089 is a more helpful article. It describes what the IRP Stack is and why you might have a problem with it. The problem is, you’re left guessing at what “an appropriate value for my network is”. I also wondered if I could configure this setting globally instead of having to manually configure it on systems exhibiting issues.

I spoke with a Microsoft contact and decided that we were having the problems because of the high number of file filtering applications (AV, AS, encryption, backup, etc) and concluded it is safe to adjust this globally. Currently we’re using SMS to change the IRPStackSize to 18 (decimal).

This error is really a big problem. Its not very noticeable by itself. But on the systems with the error, SMS seemed to not be working. This effects software update distribution. It also hurts the vulnerability scanners ability to check file versions. Hopefully we are on are way to fixing this problem on a permanent basis.

Mozying along

Last month, I read a blog entry over at zatznotfunny about Mozy that got me thinking. Perhaps its time to give in to best practice and backup my stuff. I last backed up my home computer in 1995. It was an AST computer with a built in tape drive of some sort. That computer has been in a closet for 8 years.
Backing up to a USB (or preferably eSATA) hard drive is fine, but if you don’t take the drive to another location you still have potential data loss issues. Once you’ve done that, how do you guarantee a reasonable schedule for backing up?
Some people suggest that I back up to the extra disk space provided by my web provider. If I did that, I would have to somehow schedule backing up, encrypting the data and copying it to the remote server. My web provider’s Terms of Service state that the storage space is for files necessary to the website. So that is not allowed anyway. Others mention Google Mail or Amazon’s S3 service as a great way to store data cheaply. I think its important to have software that you can count on to back the files up. I don’t want a kludge.
So that brought me to Mozy. Free for the first 2 GB of data or 4.95 per month for unlimited. That sounded pretty good. If you exclude your media the free account may be good enough. If you want to backup the videos of the kids first recital, than cough up the dough for the unlimited account. ArsTechnica had a review in July of several similar products and Mozy came out on top. After checking out their site, I googled to get the other side. A CNet blogger doesn’t like it, but I think he’s being unusually picky.
As I mentioned, data privacy is a concern when you send you data away. With Mozy there is an option to backup with their key or with a key you provide. The more paranoid would say that since it is their software doing the encryption, either key could really be known and stored by them. I chose to go with them picking the key for easier recoverability. I’ll choose to trust their privacy policy that they do not look in data files. Hopefully controls are in place to prevent low level, uncleared employees from obtaining access.
My data is encrypting now. So far I’m pretty pleased. I’ll have to test recovery (they say it may take some time to create the recovery set for you).
As I say, I just installed it, so I’m not giving a full recommendation. However, you do need to be doing something with backup. If you do choose to try out Mozy, please use this link https://mozy.com/?ref=M447CB. If you sign up from that link and begin backing up data, we’ll both get a free256 MB bump up.

Symantec Steps into software as a service

I thought this article was interesting, Symantec Steps into Software as a Service.

The Cupertino, Calif.-based company said that the launch of its Online Backup Service, which provides outsourced data storage and disaster recovery services to SMB customers, is merely the first piece in a wider set of offerings it will introduce dubbed Symantec Protection Network, which will eventually include a full range of hosted security tools.

SANS 2.2 Desktop Encryption

This is a 5 company report on their lessons learned and experience.
Rhonda Maluia from the Naval Special Warfare Development Group spoke on their use of hardware based encryption. They use Flagstone which is a British company (opening U.S. offices shortly).
I took less notes on this talk due to the dark background of the slides. Encryption on the hardware device is a very interesting concept that takes encryption out of the hands of the user completely. They don’t even need to know its going on.
They were seeking a secure solution with ease of use and the ability to fail securely.
They defined a secure solution as FIPS compliance AES 128 bit Full Disk Encryption with pre-boot authentication, tamper evidence and it works.
The more the user has to do, such as putting data in a “secure” folder, the less a solution works. The wanted minimal user intervention and moving parts. A low learning curve and good performance.
The device locks after 5 failed logon attempts. After 5 recovery attempts, the data is gone.
Obviously you still need antivirus, personal firewall, antispyware, etc.
Monty McDougal is speaking on behalf of TrueCrypt. This is a free open source solution for Linux and Windows.
I didn’t take a lot of notes because I’m not interested in this product. One thing that I think would be true across the board is that unexpected power outages can be devastating to the file system. This is harder to recover from with full disk encryption. Backups are key.
Matt Norris
Matt uses Netapp Decru to address the problem of Tape Backup Encryption.
Most people are not addressing the issue of tape backup encryption. This is a real issue.
q. Do you encrypt all backups
a. yes
Tape backup encryption is tough. We’ve all heard stories of needing to recover from 10 year old backups and trying to install the backup software and find the license key. Now imagine that with encryption.
Regarding performance issues, he says that tapes aren’t wired speed anyway.
The netapp appliance connects to the fiberchannel switch and is passed the data.
I don’t have any notes on the other two speakers.

SANS Session 1.4 Top Ten Things to Look out for in Laptop Encryption

These are my notes from a talk Eric Cole gave at today’s SANS Secure Storage and Encryption Summit. If you have a chance to hear Eric talk on any subject, run do not walk to sign up. I dint have a lot of security heroes but he is someone I admire.
Again these are my notes. I am not copying the slide deck due to obvious copyright concerns. But I hope these notes are still somewhat useful as it does take some time to convert from handwriting. If nothing else it allows me to review the material while its still fresh.
Gartner has a Magic Quadrant for desktop encryption. Most of the providers in the “good” quadrant are only 1-2 years old. Food for thought.
With encryption you might not know for 10 years if the implementation is valid. So you should do some basic checks. Boot from a CD, mount the hard drive and see what can be discovered.
Credant is great for mobile devices and PDA, but on the laptop they focus on specific folders leaving hibernation files vulnerable.
PC Guardian encrypts everything but doesn’t have the integration the bells and whistles (your mileage may vary).
Histogram – I kind of missed this part. It has to do with looking at file size over time and determining if something or other is too predictable.
It is certainly worse to think you’re secure when you really aren’t.
Eric likes to encrypt at the folder level. If you encrypt full disk, then when you log in everything is accessible. He likes to be able to leave his consulting directory encrypted while working at a SANS conference. Further backups remain encrypted when you do folder level encryption.
Many people deploy encryption without fixing up the security of their computer at all.
Deploying without a screensaver lock is like leaving the door open on a safe.
Same goes for deploying with a bad password policy. Eric says quit messing around. Set the minimum length to 30 and be done with it. That will force users to use a phrase. They cant write something like that down, its more trouble than just learning a phrase.
Like Alan said, you need to look at data protection solutions as well as encryption.
If encryption was easy everyone would be doing it. Its been around a long time.
Because of laptop theft and data leakage press, and regulation, crypto has become the hammer of choice. Crypto is seen as the solution to every problem. Ever hear the phrase “when all you have is a hammer, everything looks like a nail”? Pass me the crypto-hammer.
1. Protection of the key is paramount.
the strength of the key is based on the strength of the password that protects it.
If your users have admin rights, your ability to succeed in this deployment drops by 80%.
2. Understand what risk is being mitigated and what isn’t.
take protections commensurate with the exposure. A $10k per day body guard is nothing if the wrong people want you dead.
3. Encryption doesn’t prevent inference attacks.
Several friends of mine have spent time in unfriendly countries. I asked them about using hushmail or PGP. They said that if the bad guys suspected you of hiding email traffic it would only cause trouble.
Eric told a story where they suspected theft of trade secrets. They did some egress monitoring and found one guy who only used encryption when sending email to one address. That certainly raised suspicion.
Steve Jobs apparently has a bodyguard for his computer. If he isn’t within 5 feet of the computer, than the guard needs to be.
(skipping some steps)
6. Know the problem you’re trying to solve.
Its no longer a laptop when its got 80 Gb of data on it. Its a portable server.
“The only silver bullet is found in a bar”

SANS Section 1.3 Top Mistakes in Deploying Mobile Data Encryption

Again these are my notes from the SANS Secure Storage and Encryption Conference. In Session 1.3 four companies discuss their experiences deploying encryption.
JP Morgan Chase – Guardian Edge EPHD
48k laptops deployed.
They found problems due to standardization issues and multiple support teams.
Key Challenges
- If your goal is to encrypt data on laptops specifically you need to be able to find the laptops and know how many you have.
- multiple support organizations
- New login for users
I didn’t quite understand the login issue. Are their users now faced with a dual login where they authenticate to the encryption software and then again to Active Directory?
Reports! Produce reports showing install rates. Highlight the departments doing good.
Your biggest problem will be the guy who likes to screw around with hacker tools even though its not part of his job.
You need to be able to validate that encryption has occurred and continues to occur.
Backups are crucial.
They found that if you boot to safe mode and run defrag you will kill your master boot record. I wonder what that says about booting to safe mode to fix spyware issues. HMMMM.
People think this will slow down their PC. They wont do it on their own. (I would say that the users who have customers demanding it will do it.).
Q – How do you deal with the engineer/hacker wannabe who thinks they know better
A – Log agent with central aggregator.
Northrop Grumman – also using Guardian Edge
High level buy-in is key
They had lots of pushback initially, but the installs turned out to be not that big of an issue.
You don’t want your customer coming back to you and saying your encryption isn’t good enough. That is why they did full disk AES 256.
They spent a lot of time with legal on export control issues. We all know about the axis of evil countries where you cant send export software. But what about less known laws where bringing an encrypted laptop in can cause problems. They have a list of 20 countries that they cant go with their computer. Corporate Security and the Travel office coordinate so people going to these countries dont have sensitive info and use a vanilla PC without encryption.
Communication is key in the deployment. The initial encryption time can be an issue.
Northwest Mutual - Safeboot, Credent Mobile Guardian
q – how did you verify that the solution is installed
a – They used altiris to look for specific EXEs.
Q – how did you handle multi-user pcs
a- I didn’t quite get this. It sounded like you have to assign each user the rights to logon.
use full disk encryption – you dont want to leave the decision in the user’s hands.
users would reboot on their way out for the day. As a result unattended SMS installs did not work. They had to change user behavior.
FDIC Credent Mobile Guardian
Credent does GINA Chaining
In your project you need to give users the confidence that you aren’t going to disrupt them.
Don’t go for the big bang. Test in small groups and deploy.
Lessons Learned -
-Confirm product’s ability to encrypt data regardless of location type and structure. Fill in the gaps where necessary. ( my comment. it can be a real issue when the project scope is defined one way and people start asking about other features)
- Don’t deploy to many things at once. Everything will get blamed on the encryption.