Check boxes. My enemy of the last two weeks.
The first one, kept me working at a problem for the span of a solid day, and a significant amount of grief for what should have been an almost flawless migration from Exchange 2003 to Exchange 2010.
So why was a check box so painful? When it's not DOCUMENTED anywhere. I had read about 30 different walkthroughs, and 5 very good troubleshooting articles from technet and other sources, that all mentioned the same process, and things to look for. None of them mentioned the problem I was experiencing, and stating everything I did was correct, and should work flawlessly.
What was the problem I encountered? Exchange 2010 to Legacy authentication Proxying. How did this break things? Well it meant that all my users on the old system, had to change their phone configurations. Yes, my current company is small and the users are easy to manage, but this is enterprise grade software. Why the heck was this even happening? The answer? Once you hit the HTTPS of the new OWA/EWS/Active sync, it then use straight up HTTP to pass the information to the old exchange 2003 sub system. This being said, I couldn't find any of this useful information until about half a week later when migrating our CRM mail system to the new Exchange server.
All of the walkthroughs spoke of removing the "Force SSL" on the Active-Sync folder in IIS by unchecking the check mark under security settings. Now, they don't say anywhere else, but when troubleshooting the CRM Mail change (which now syncs with the webmail/EWS interface), I found a nice little tidbit stating the same I had already read a million times (The Remove Force SSL), but to ALSO remove it from the /Exchange folder. Well ROCK my WORLD! It worked for CRM!
Now, at this point, my brain starts to stir, and it dawns on me, this is why my authentication wasn't working, it was forcing SSL. So, I create an e-mail on the old server, and I setup the account on my phone. I point it to the new front end, put the password, and faster than I could hit enter, BING! So I send an e-mail from my mail client, Instantly shows up on phone (this was seriously laggy, at BEST, when I had it before). So I further test, by sending an e-mail. Low and behold, it sent! Yeah, an entire weekend of changing user phones over to legacy.company.com.... Could have been avoided by a check mark. Seriously.
Getting passed this, I now move on to migrating user mailboxes. This, goes fairly well. Till I got to my Manager/VP of IT Services. I move him, and overnight he says nothing sync'ed with his phone. Remove profile, read it (worked for all to date). Research leads to me seeing he doesn't have permissions to create his mailbox on the new server. Double U Tee Eff? He's setup as the only other Domain admin. How is this possible! So, after reviewing his permissions, I see that his AD Security settings no longer have a check mark under "inherit permissions"... So I check it, apply. As I did that, he yells over to me "Hey! It's working! I don't care what you did, make sure it doesn't happen again". Oh joy.
Mothers, Fathers, please have your kids check their verify check boxes. It's embarrassing when they trump you.
Regards,
Adam Melong
Everybody knows me to be a huge dedicated Cisco fanboy. So when I first attempted to upgrade to the latest and greatest Adaptive Security Appliance operating system (8.3) over a year ago, I was scared away when the "upgrade" broke my long worked on and cleverly optimized configuration.
Now, I don't back down from things too easily, but being lazy, it took me about a year to look back at it and attempt a conquering. How did the conquering go? Poorly. Well, not poorly. I figured out what was causing my issues after researching and finding a very nice breakdown on the cisco support forums.
After spending a couple days waiting for a newly purchased ASA5505 with an unlimited users license (I had a 10 user, and user for Cisco means IPs. I have significantly more than 10 IPs on my network...), I spend the time trying to build a new configuration based on the walkthrough for NAT. All of the configuration differences were documented and then put into a text configuration file properly broken down. So, Ready to go!
I fire up my new ASA5505, upgrade the OS to 8.2 (including the ASDM to 6.2), and then a final upgrade to 8.4.1 (ASDM 6.4.1). Off the get-go, most of my default configurations work flawlessly. A few new features and alterations to configurations, but nothing significant, and quickly changed in my text file. Then, came the heartache and disappointment.
First, a bit of background to why I considered my configurations "clever" and "optimized". The ASA has a nifty feature called "object-group", that in my opinion, is just brilliant for making a clean structured configuration for rules and policies. With these object-groups, you can essentially create groupings of networks, ports, port ranges, services, etc... If you're asking yourself why this is important? I answer you: Memory.
If you haven't been following technology over the last decade, let alone the last two decades, you would have noticed all technology is moving towards doing as much as possible in memory. This is due to memory systems being thousands of times faster than conventional storage and processing. Think about it, remember everything you need to do during the day, or simply put it in a calendar you can pull from. Of course the calendar is useful, you just grab the information as the time comes needed to use it, and that frees up your mind to simply process things as it comes by.
Coming back to the object-groups, you can essentially load up a broad range of information into memory, by applying them with access lists. In the networking world, access-lists are used to allow/deny things, but also, to define interesting traffic. A best example for this scenario, will be VPN. VPN relies on handshaking, mirrored access-lists, and pretty much duplication of configurations on both sides, but in reverse. What triggers a VPN, is the ACL (access list) being processed when a packet flows through (yes, PROCESSED). If the packet source IP, port and destination IP, port match the ACL, it then encapsulates the packet and sends to it's destination all bundled like a pig in a blanket.
Where the "optimized" configurations in networking come from, are when you can manage to optimize your routing/processing to the littlest amount possible. This involves refining ACLs to a minimum to stop the router engine from stopping, sending an IRQ to the processor, and then looking at EACH ACL in priority until it hits a match. If no match, it simply forwards out the "default route". In a small office, it's not that heavy, but it's great to stick to best practice and scalability. Who knows, you could be working for the next Microsoft, and you get to help in TAKING OVER THE WORLD! Narf. In this scenario, the firewall will process all packets and you want to keep as much processing for the firewall to do what the firewall does. It encrypts/decrypts packets for secure delivery, it filters packets, intrusion prevention, etc... By adding object groups, you can condense the ACLs based on traffic flows instead of each flow.
Ok, enough shovel following the horse... Why is this all important and why Cisco (in my opinion) screwed up. They took the object groups, and applied it to everything now. Their new NAT is simpler for the newbie admin, and easier to follow, but it's convoluted and a waste of text space. Let's compare the amount of lines of "code" it takes to do a simple Port forward in 8.2, and then 8.3.
8.2 - 10 lines of code, some fluff but to prove a point
object-group service SERVER_Services tcp port-object eq ident port-object range 6100 6199 object-group service NETWORK_SERVICES tcp group-object SERVER_Services group-object SERVER_Services group-object SERVER3_Services access-list outside_access_in extended permit tcp any host OUTSIDE_IP object-group NETWORK_SERVICES ! Removed Code static (inside,outside) tcp interface ident SERVER ident netmask 255.255.255.255 static (inside,outside) tcp interface 6101 SERVER 6101 netmask 255.255.255.255 static (inside,outside) tcp interface 6150 SERVER 6150 netmask 255.255.255.255
8.3 - 12 lines of code, that's just for the ONE server.
object-group service SERVER_Services tcp port-object range 6100 6199 port-object eq ident object network SERVER-ident host 172.32.100.5 nat (inside,outside) static interface service tcp ident ident object network SERVER-6101 host 172.32.100.5 nat (inside,outside) static interface service tcp 6101 6101 object network SERVER-6150 host 172.32.100.5 nat (inside,outside) static interface service tcp 6150 6150 access-list outside_access_in extended permit tcp any host SERVER object-group SERVER_Services
I'm sure you're asking "so what? it's two extra lines..." Well, you're right, but, scale that. The first configuration has 10 lines of code, but 3 of those lines are ADDED to simply combine everything into a single Access list. One access list, for one external IP, which in turn then port redirects to whatever port is triggered. The new method is definitely object orientated, but the old method was a packet derivative. It followed the flow, now, it follows the device. This is where the issue comes up with me when scaling and optimizing.
I now have to create two extra lines for every port forward I need to make, and an access list for every server that will be port forwarded to. The new method uses the internal IP in the access list, while the old method used the global external IP that I used to mask my internal devices with. In essence, you no longer have a hirarchical flow from L4 and down. Your configuration is triggered by the Port and Nat process, and THEN verifies the access list. Instead of simply checking one global ACL, and then converting with a single line of code.
Confusing? Yeah, that was why I was shy'd away from the get-go.
So, after troubleshooting likewise for 5 hours trying to figure out why likewise wasn't starting lwiod, I find out that Debian 6 changed from "update-rc.d" to "innserv". Why is this important? Well, that means all of those wonderful custom scripts I created in the /etc/init.d/ directory now become "broken" due to lack of what I later came to learn as LSB Headers.
LSB Headers are those commented out lines at the start of a script I never realised actually DID something... Go figure, another dimension of comments to work as code... More information: http://wiki.debian.org/LSBInitScripts
From uninstalling the app, to troubleshooting package installation (dpkg, aptitude, apt-get), to ripping logs apart and googling strings, to specific words, to blind clicking, I finally found the LSB header line mentioning insserv. I should have clued in to it originally seeing it was on EVERY line of the error when trying to start the app.
Moral to this story: Linux apps often add the app name to the start of logs, so don't just research errors hoping for specific errors. Research the keyword, then the terms, and THEN the errors. You'll learn more about the programs, and then you can learn about the debugging.
My brain hurts.