I thought I would share a little story about my HUGE security oversight with my home SBS 2003 system. I thought it might be good to both to educate and to elucidate how easy it can be to compromise a system. (when you don’t pay attention) I should preface this blog entry with the fact that I am no slouch at computer security and have worked extensively with ISA server for many years. I used to run SBS 2000 at home and have run SBS 2003 ever since it was in beta. I love this product – way before I worked for MS.
Anyway – I installed a new VOIP phone at my house about 3 months ago. (I must say that I am very impressed with the service and there is nothing better than getting my voicemail delivered to my inbox.) The phone unit came packaged as a basic router/firewall with a few Ethernet ports and a couple of phone ports…easy enough.
My first notion was to install this box behind the ISA Server on the internal network for the easiest setup…. Just pop it in, open the appropriate ports and I’m off. However, the documentation about exactly what the needed ports are was not evident in the documentation. A quick search of the company’s web site shed some light on the ports that are needed for it to function.
I define the protocols in ISA server as published on the VOIP Provider website and create the needed rules to get those ports to the VOIP box on the internal network – easy.
I plug in the phone, get a dial tone – everything seems ok. However, when I dial a call – nothing happens. I am going to make this part real short – after about 3 hours of troubleshooting with the VOIP Provider I decide to take the easy route and put it outside of the ISA firewall. (It is also is what the technician first recommended) Evidently application layer firewalls pose a problem for the phone…
I didn’t think this a really a big deal as it is a basic firewall after all, and this is how it was intended to be implanted by the manufacturer – hooked directly to the Internet. So ill just pop it out there, it will get an IP address from the cable modem and Ill be on my way. As it turns out – my cable modem will only assign 1 public IP to a DHCP requestor. So it is either the SBS box that gets and IP or it is the VOIP box. (Or me spending more money for another IP Address, and I’m cheap)
So I am determined to save the money I promised my wife we would save with a VOIP phone and only use 1 Internet IP. I decide to let the VOIP get the Internet IP and use the port forwarding features of the firewall to send all unsolicited traffic to the SBS box. I use the 10.2.0.0/16 subnet for the VOIP-SBS traffic as my internal network is 10.1.0.0/16. (Anyone guess what I did yet ;-))
I whipped up a quick diagram here to show what it looks like.
So – everything works perfectly. Or at least I thought it did.
I’ll be the first to admit that I don’t check my ISA logs or alerts as much as I should or I would have caught this problem much sooner…. After a few months, my Internet connection seemed to be getting slower and slower. (Looking back, this all seems so obvious as to what was happening)
One day last week while I was working from home, I established a VPN connection to the Redmond campus. After about 5 minutes it disconnected….again and again and again. Hmmmm….I think. Patch Tuesday was yesterday and WSUS delivered all the patches to my systems as they all rebooted during the night…maybe something went awry during the update? I log onto the SBS server and look through the system and event logs – nothing interesting. I thought I might jiggle the toilet handle on the web proxy service and restart it. When I went into the Web Proxy Service monitoring tab I was astonished to see 90+ anonymous connections using the proxy service – all with Internet Addresses! So I instantly start to panic – maybe there was a 0-day exploit that hit my SBS machine before the patches could be deployed….maybe my daughter or wife installed another ‘Cool Program’ on her laptop. 😉
I quickly run a netstat –a on the SBS computer to see where everyone is coming from. Yup sure enough they are coming from the Internet and the majority of the connections are hitting the 8080 port – used by my proxy service – weird. That explains all the anonymous connections in the web proxy service….. I fire up netmon to get a closer look and sure enough there are hundreds and hundreds of connections to every part of my computer. I immediately unplug the Internet connection from the modem and start to dig.
My first thoughts were to my loving wife and daughter….I turn off their machines and hook back up – the connections are almost instantaneous – dang! I try the same thing until all 10 computers in the house are off with the exception of the SBS computer – connections are still there. So at least I have isolated it. I disconnect again from the Internet and run a full spyware and virus inspection – nothing. I stop all my web sites and services – nothing. Go line by line through the packet filters – nothing. This is getting really fishy.
Epiphanies are a really weird when they hit you – especially when you are so unbelievably frustrated that not even the family pet will make eye contact with you. This epiphany gave me goosebumps – the Local Address Table on the ISA Server!!!!!! (For those who don’t know ISA – that probably didn’t sound exciting)
That was it! When I first installed the ISA server years ago I defined the LAT, or the trusted network, with the IP Addresses of 10.0.0.1 – 10.255.255.255. Why was this? I do a lot of testing on my Internal network and wanted to work with a full Class A subnet Internally for the most flexibility.
This also perfectly explains why all of those Internet connections were able to hit my SBS box with no problem – all the Internet connections were translated (NAT) by the VOIP box to be coming from 10.2.X.X – or a trusted network – so the ISA Firewall did nothing to stop it as it thought it was from a friendly subnet that I defined.
I change dthe LAT to the 10.0.0.0/16 subnet to define the VOIP addressed and untrusted and everything immediately went back to normal. Whew – wish I had thought of that first.
Now the amazing thing to me is that this SBS 2003 computer was sitting exposed thinking the Internet was a safe network for close to 3 months – and it was never compromised. I went through it with a fine-tooth comb. Nothing – nada – zip. I attribute this to few things – but mainly the reason it didn’t get compromised is that it was always up to date with the latest patches.