140 Village Shopping Center Westminster, MD 21157   (410)- 848-7100

Click here to submit your own IT legend & win prizes!

A little story about Nate (our pseudo Geek wannabe hero) - who learns that Upgrading to the latest and greatest Microsoft Server OS can be a daunting task - to say the very least!

"Behind the Server Room Door"..., - (or) The Saga of Nate!

By: Wade Johansen (2007) ©

Previously ... The employer walked into the IT office one day and asked the IT "lead engineer" , Nate to get the server upgraded to the latest Microsoft server system. His boss promptly asked if he needed to bring in any outside consultants to the table, but Nate stopped him short and said he didn't want any "outside consulting knuckleheads messing about with his servers" that he had so skillfully set up the companies services on....

... now - Nate assured his boss there would "be no issues", and he thought about how nice it was going to be to finally get the server operating system that many other engineers had been talking about in the IM technical engineer chat rooms. This was going to be his own personal gift to himself  for having studied so hard to get all the technical certifications and finally receive the respect of his peers, which had always seemed to elude him. the companies server technology was never really the "latest and greatest", so Nate always felt he was one step behind of his most "idled" peers.

The company had set forth quite a few protocols that had always prevented quick adaptation to newer technologies. There always seemed to be "too much testing" and "paperwork for this and that", that had to be filled out . They weren't a leading and bleeding edge company, but they also weren't too far behind everyone else either.

Still, it always seemed there were too many "trials" that had to be run before anything was ever set up on their "live systems", Nate knew his system well and he was hoping this would be his chance to show the owner that they could be the leader of the pack when it came to adapting to new technology. After all, he had built it with his own hands for the most part, and he had several IT techs that worked under him maintaining this system. Who better to lead the charge, than he?

He began to think about all the requirements though of this new installation..., just to be sure he wouldn't miss anything. Nate already knew his previous configurations of DNS, DHCP, RRAS and were solid. No need to change them now! He also knew the TCP/IP stacks he had configured to run NAT on the single static IP was correctly installed on his edge router firewall server (Microsoft ISA server)... Nates setup was good and solid.

Carefully, and meticulously he had crafted a pro-active filtering TCP/IP scanner, (that he was particularly proud of) that scanned his internal IP scope to protect against most known external attack methods and it even updated his daily tools list to prevent most of the known possible 0-Day attacks and heuristic attack methods. Admittedly he had used the help of one of his IT friends whom he had come to know through his years of IM chatting with other top engineering technical contacts, but the work on his server he considered to be all his own. It was, in his mind - "a thing of beauty".

Nate also knew (by protocol) he had to send filtered data snippets that were non-network compliant, in the form of SNMP packets. He had them use corporate rules of hostile packet engagements, and flagged them and any other extraneous data for later observation - then it quarantine the packets - and saved them to a separate file on an otherwise isolated subnet that was on a separate backup server storage drives.

Nate, being cautious as a network Administrator made sure to password lock this file so only his boss or him would be able to retrieve it in the event of a disaster. Finally.... he made sure that flagged messages were sent an administrative alert (by email and by paging his cell phone) of any possible breach of the perimeter firewall defenses.

It was only possible to view the backup data from a "hidden network share". (Nate prided himself on being a master of security methods, among his numerous other network talents.

After a quick and final thought about his current NAT installation , he considered that he probably wouldn't need any new changes there since he had designed it specifically based in his internal IP network information for his company - and being solidly built and functioning it should not be an issue or require changes.

Nates company only had one Static IP, (by his design). He knew that his LAN Router backup had copied his most recent setup (by default) and that a separate copy of the address table and MAC filtering lists which used his servers internal IP address for Internet use - (so it couldn't be sniffed out by the competitions internet users) was a secure setup.

Plus Nate knew that two of his satellite office locations kept IP tables and copies of all their most important corporate data in their onsite safe storage. Nate was absolutely certain his private circuit was under secure "lock and key".

So, what next then, he thought?

He decided to set up a temporary router, or "proxy pinhole" just in case during the upgrade there would be any chance of outside exposure (being secure minded - Nate took exception to be careful not to forget the obvious risks involved with doing an in-place migration during business hours). His company, after all... ran 24-7-365 - so there really was no time like the present to run a server upgrade.

He decided to use a sliding windows port just to be safe. He knew his competitors had fierce initiatives to steal their valuable data, and that any crack in their perimeter defense could prove to be disastrous. and he made sure that one of the ports still - (as it had always done), securely enabled his separate VLAN functionalities Another feature Nate used was strictly set up for his corporate VPN (Virtual Private Network) and the setup was only for backend encrypted users to get in from their satellite offices.

All traffic still may have used a port forward to port 80 on the WAN side for HTTP requests but the re-directed traffic for web requests went to the (now) dynamic proxy addresses that were being used for internal LAN - To External WAN requests - and so all session traffic was setup using the one time proxy pinholes. After a session was set up, it would tear down the port and begin any new sessions using a uniquely new port. This was "security be design", he thought, and he decided to give it the dynamic traffic a range between ports 12,000 and 20,000 for sending and receiving.

Nate also remembered he needed a DMZ static IP for his local Intra-Net (internal corporate forms and file share location). So, he checked the current DMZ path and IP and decided everything was good and secure. He checked his set-up to be sure his router was till sending his "internal IP traffic requests" to bounce back to his network so no traffic would be sent out across the internet (where the documents or data could be stolen by is competitors) and even ran a tracert to check against  possible "leaks". 

Since everything had checked out just fine, Nate decided it was all a "go for launch"! He sent out an administrative SNMP message to all console users informing them the server would be taken down for 1 hour (during lunch when only a skeleton crew was using the network server), and that they would be able to resume work thereafter.

Then Nate put in the upgrade CD and sat back with a smile. He'd be "done by lunch time", he thought to himself. Maybe - (he considered to himself)... it was time to ask for a raise?

After all... he figured if his boss was "so willing" to pay for an "outside consulting firm" for something so simple as upgrading just one server setup, then he could pay a little extra since it was being done without having to pay for what Nate figured would have been) an expensive consulting fees!

He even knew he could probably make more money as a consultant himself..., but he liked the idea of coming into his job every day and having the benefits of being the "King of the Hill", or so to speak. Plus, the perks weren't bad either, there were a lot of women who worked in the office that were nice to him, which didn't happen outside the office... and he had taken a particular liking to one of the girls who worked in the accounting department. She seemed to like IT, and didn't seem to mind the "technical jargon".

Nate thought to himself of how he would be sure to stop by later and tell her of his particular technical savvy of bringing the company into the latest and greatest operating system (all by himself) and tell her it was because he felt she could make far better use of her time... because - of all the time saving features the new server employed !

He even reasoned that if she could make better use of her time now . Maybe, he would even suggest some of that time be spent having lunch with him later, where he could explain all the details. Nate pondered what his life would be like with her, and he wondered about how good life was going to be as he got to know her better.

While Nate was pondering what he was going to say to the girl from Accounting, he suddenly noticed one of his other server screens blinking... he had began stacking up outgoing HTTP traffic requests, and guessed that someone didn't get the message about server shutting down during lunch. But, when he looked at the amount of requests, he saw they were coming at an incredulous rate... and they weren't coming from any of his known and configured subnets.

He quickly checked his routers traffic requests on the other servers menu screen and realized he had IP requests that were not just bouncing through the routers firewall ... but they were returning true IP values from inside the network and using a VPN IP for the requestor?

Nate wasn't exactly sure what was going on, (maybe this was something to do with the way the new server handled pinhole traffic), but he knew after running his sniffer tool he'd know where the traffic requests were coming from.

But he quickly realized the requests were not coming from inside his own network..., how could that be he thought? Since he was using a pinhole for all traffic, no traffic could be using a static requestor? But, his proxy didn't seem to be using any of the values for port requests that he had set up so carefully, why wasn't it holding up?

Nate started to get that sinking feeling in his stomach... and for the first time in years he began to have doubts about his "superior" IT skills he had just earlier been day-dreaming about. While he was busy trying to bring down the other backend web portals he also was trying to send all forward facing DNS requests to use the routers actual IP tables only. They were "hard wired in", he thought to himself.", and Nate began to run over his SNMP traffic logs in his head.

"Had I checked to block traffic against all external requests"? The scenarios starting playing out in his mind. None of his training at college or the corporate classes he had attended had ever raised the type of questions that began playing out in his mind.  If someone has breached the IP tables. or if someone has managed to slip in a 0-Day... "please, no 0-Days" he thought to himself... "not today of all days, please"! 

He realized his hour of time for bringing the main server back online was rapidly advancing, and he started thinking for the first time ever that the "brain-dump" sites he used to barely squeek out the passing scores he had gotten on his by his MCSE exams may not have been such a great idea after all.

Realizing he was running out of time and options, Nate switched his KVM over to show his Microsoft Operations Manager console. "MOM", Nate thought to himself would save him. It was running on his second rack-mount and was completely isolated from the router traffic. But when he hot-keyed over to the server, all he saw was a blank screen, Nate got nothing... not even a blinking cursor! This, he knew now was not good! It couldn't be an external attack, it could be a virus he thought to himself.

"Maybe it was a logic bomb", he thought to himself? It could have that tripped when he brought the backup server offline and have now tripped starting a system-wide crash. He knew very soon he'd have fellow employees waiting for answers to question that he never thought he'd be asked?

Nate then looked at his only remaining available server screen that was responding, his backup server screen. Quickly he got into a DOS prompt and began a tracert to see if he could ping his router, maybe he'd see a man-in-the-middle attackers IP address or rogue DNS server... anything at all to help him get an idea of where his workstation traffic was going! While no-one had come in (or even knocked on his door yet), he had become aware of some chatter coming from outside the server room door.

He looked back at his screen, and then realized he wasn't receiving ping responses from outside of his ISPs DNS servers. The packets were dropping off into a black hole and timing out... one by one as each server beyond his primary connection painstakingly failed to responded it hit him like a sledge-hammer to his chest.

Nate knew now, he (like his company would know in moments) they were "dead in the water"!

He ran a ping of his only known back door server line to the outside world, a server "DNS survival rope" he thought to himself. Nate had engineered a "last line of defense" in the event of an emergency like this. This certainly qualified as the day to run that defense.

Nate had set up the back door against what he thought had been his better judgment..., but he had done so at the urging of his fellow IM buddies from the engineering Blogs he had frequented and had become so accustomed to getting his answers from. He knew now why they had urged him so badly to do so, he never thought he would have to actually use it..., but he knew he was glad to have it now.

His network having suffered from a tear-drop attack several years ago, he set up the link. He had even used it once before to get in remotely when his remote insight cards had failed. that, luckily had turned out to be a prolonged power outage that had shut his servers down by design when the power failed to come back on, and the batteries had given out. He now was hoping that link could save his career.

While he waited for the ping responses, he quickly grabbed up his phone and sent an "SOS" message out to the single smartest friend he knew simply as "MOU" - or Master of the Universe" as Nate had imagined. Mou was an incredible master of network engineering, and he ran the network Blog that he used most often for his answers whenever he couldn't "get a handle" to a problem on his own.

While he waited for a response, he could feel himself starting to sweat, even though the server room temperature was always at a constant 72◦? He started to run a check of his hand-written MX transfer notes he had performed in the weeks prior. That had gone flawlessly, what could have happened between then and now.

He noticed a minor transfer error and discovered he had missed setting the new mail server up to block un-requested traffic to his backup server IP. But, the web-mail "forwards" to the client desktops that used Outlook accounts were correctly set, and digital signatures were used at every station. Probably of no consequence here he thought..., he was starting to grasp at straws.

He guessed that possibly his secure e-mail certificate that gave the company their digital compliance may have failed, and possibly he had forgotten to re-register the new servers PKI key. he couldn't remember now exactly how he had handled that. But he figured at the time it didn't matter too much, as long as the private key hadn't been compromised. He was sure it hadn't though, so he thought to himself "what else"?

Maybe, he guessed while thinking out loud he needed a different server ID, and possibly even an altogether new certificate. This was actually not his area of expertise, but it hadn't let him stand up and take on the job when the boss had come calling then either. He wished he could just wither away, but knew he couldn't.

In every way he was thinking about what was happening, no matter how hard he tried to come up with an answer he realized he was not going to solve this on his own now. This was not good news for him, this was a "career killer" if he couldn't get a handle on it in the next 5 minutes!

Nate mimicked the words he had said to his fellow IT friends when he touted how easy it would be for him to upgrade the new servers at work, he relished being above his peers technically. He thought about the IM he had sent to MOU just one hour earlier bragging about how he was setting up his "new server installation..., all by himself". Having had talked to his friend on numerous occasions when he had needed advice before, but had never really thought he would need a friend as much as he did now.

Secretly he hated the thought that he would be losing face in the technical savvy arena of being on top of his game by sending the last message to him and it seemed like an eternity since he had sent the SOS.

His gut was telling him to just quit and walk out, maybe even run out and never look back. maybe he could move out of state, he didn't have that many friends and he imagined he could start a new life somewhere and just forget this day had ever happened.

He knew his employee contract stated that any corporate loss due to his negligence, required he make a monetary arrangement as may be required to recover the companies losses.

Nate uttered out loud to himself..... Oh, so it's going to be Soooooo Easy, Right???? he was losing all hope at this point.... thinking back to when he said (just hours ago) how he had jokingly said to his boss when he asked if he had needed any help with the upgrade... "Gah.... yeah right ,"- stating he would have the server down at 11:00 am and then be "like, back up before noon all cleaned and polished for you sir". He had thought to himself then how everyone would know he was the smartest guy in the place!

Suddenly, with what sounded like a thud... he heard a single but distinct little ticking noise on his last remaining backup server. A hard server error BSOD (blue screen of death) appeared. The server froze, for a second and then just died. Nate felt like it was taking the very breath of life with him as it spun down and shut off. .

Suddenly, from behind the server room door Nate could hear for the first time the remaining internal network users started who were talking. "What in the heck is happening with the file servers", someone asked, it was the girl from accounting. She had come down , Nate guessed at the request of the department  head.

"We were in the middle of running bank transfers and just suddenly lost our connection", she said? Nate could hear one of his "junior techs" trying to answer, but his response was simply "I don't know, we'll have to ask Nate". "He was working on one of the servers, so maybe it was something he was doing?"

She sharply bounced back her remark to him, "the accounting department had been told they could stay online during the transfer", and that their accounting package was running on a "different accounting server". It wasn't even "on the same subnet" she said, adding then "while I don't know what that means, we were told we wouldn't be affected". She added, "the boss even said so himself earlier in the memo to all the other office users"..., "accounting and finance were not going to be affected!

Nate became increasingly aware of the silence now emanating from behind he server room door. It was deafening in reality. he knew his link to all of the satellite offices was dead, and he heard the phones begin to ring off the hook! He thought of how it would feel to die from a heart attack, secretly hoping he could feign having had one. But he knew he couldn't fake it, even though it was truly how he was starting to feel now.

Noone but a true master of IT, he thought to himself would ever really be able to go in and explain themselves in the face of what had just happened and get through a forensics investigation. He had already failed to call in the error, or stop the servers when he started to see signs of trouble... or manually disconnect from the router. He hadn't done anything his training had told him to do, he was trying to save face all along from the moment he lost control, until the panic had overtaken him.

As the remaining network users realized they weren't back up yet, they started coming to the server room door and were now knocking. Nate realized his deepest fears of what was going to be the worst possible scenario of events as happening and quickly this was turning into a very ugly situation.

It was, he knew now going to be the single-most worst disaster he had ever had to experience.  As his co-workers that were still gathering outside the locked server room continued knocking he thought about his SOS message to his IM friend even the "miracle of hope from the "Master of The Universe" didn't seem to be coming his way.

He imagined his co-workers outside were starting to gather up piles of  the "lost and found clothes" from the lobby, and were now "stringing them together" to look like an "IT network engineer" as if he were going to be burned "in effigy".

Horribly, Nate now also realized he had forgotten to back up the server files before attempting the upgrade... and the 5 last nights backups hadn't been migrated to any offsite location since the previous Sunday.

How dumb it was for him to propose the idea to his boss for his "suggested design" which was supposed to save the company servers space by emitting a daily full backup, and instead opt for 6 "differential" backups to every 1 "normal" backup. Or, was it 6 "incremental" backups to every 1 "normal" backup.

It had always been a little bit confusing to him before how each one of these made either a "copy of changes", made to files that were created or updated recently..., as opposed to actually recording "new additions of files created recently" all together he now wasn't sure anymore which one he had even chosen.

He realized now, he was in deep trouble, his company dealt with hundreds of clients every week - all  that billing, account management - the whole ordeal would ruin him. The girl from accounting was never going to speak to him again, he thought to himself. She was now out of the question.

The surrounding mob began banging on the door now. And now he also heard his bosses voice outside, coming closer to the room. There was nowhere to hide, he knew he was going to be fired, but worse his reputation among his peers in the local IT community would be ruined. he'd always thought of himself as a a sort of  secret hacker. One his IM friends would aspire to be someday. That was now the farthest thought from his mind. How could this have happened?

He thought to himself, I should have consulted with an outside source. I should have made sure I followed our company protocol. But all the "should have" and "could have" thoughts didn't matter. There was a key in the door, he knew what was coming next..... Nate realized he needed help, and so he looked at his phone one last time.

Incredibly there was a response, in all the commotion he hadn't heard the click of the chirp of the response. Thank God he thought, MOU has replied with help. His friend hadn't deserted him after all.. what words of wisdom he thought might he bring to help him.

When Nate read the screen, his heart actually gave in. When the boss finally managed to open the server room door, he found him Nate sitting at the blank console, dead at the young age of 34. He was clutching his cell phone in his hand and there was simply a single word written on his cell phone... his boss picked up the phone and looked at the screen it simply said "Gotcha"... signed MOU!

Could this really happen...! You better believe it could!

Nate could (if he weren't a fictitious character and was alive and real), have been facing criminal negligence charges, attorneys fees, a lifetime of hardship paying back his employers for loss of earnings, and he would be shunned by the IT community he knows.

Not to mention... he would lose his job, respect of his colleagues and he would have lost the company he works for not only the very real competitive edge it may have in the world of business, nut worse it could have lost control of its own corporate ideas and information... and caused a collapse of the business completely. Ultimately that causes loss of jobs, employees lose benefits, and pensions go unfulfilled. It could have ruined the lives of many because of "the ego of one".

Businesses can fail (and do) when the competition gains control over their information, no matter how it happens.

Nates company and his own life would have been left in ruins, all because of one (slightly poor if not rash) decision to go it alone!

The fact is, this scenario is happening every day to companies, and IT employees all over the world.

While many times, the outcome is different... the feeling quite often is the same for the workers who think too much of their own skills, only to realize that they should have done their research better... or should have consulted with other personal (and private) parties in person to be sure their migrations and upgrades didn't suffer the consequences of failures or theft. They need to have a separate party to hold accountable if something fails..., or someone tangible to go to that can either support or discount a reason for a failed upgrade or installation!

Too many companies are relying on less and less IT professionals to make their businesses run, and those already pressured IT personnel are being asked upon to "make all the decisions" about and for their IT infrastructure. Subsequently many IT professionals are feeling they either can't afford to "lose confidence" by hiring an outside source for help, or in some cases like "Nate" they don't want to!

Either way, the effects on a business can be devastating! As a business owner (or) an IT employee you can't afford not hire a third party when the consuquences are so real and damaging when this type of scenario plays out!

And, this is not the only scenario..., there are thousands just like this.

At Couri Technology, we know we are not the only ones out there with good answers, because we aren't the only ones who work with IT servers or businesses. We also realize that there will always be smarter, better and easier ways of securing against an upgrade failure, or hardening an installation or upgrade to help prevent a "single slip" of security. It doesn't always have to spell disaster, but even a few hours or days of "being down in recovery" can cost a company money.

There are industry "best published standards" and "security breach prevention methods" that are practiced and proven to be better than those that may be used by individuals who (alone) may not have the time or skills required to read through and understand all of the manufacturers specs sheets.

Many of these IT professionals are simply pulling their knowledge from "brain-dump" sites, or worse "hacker infested IT engineering sites" that promote their own ideology. Many websites are solely set up for giving out "false information" in the hopes of someone like a "Nate" being suckered into their scams.

The internet is full of such sites, a quick Google of the word "braindump" reveals nearly a million sites alone, many which are set up solely to lure techs into traps like the one experienced by "Nate". Many sites are often torn down once the target has been compromised, leaving no trace of evidence that could be pointed to by an employee who cries foul after the fact!

There are industry standards that need to be met and employed ... - "every time a server gets "touched", or in particular... goes through an upgrade"!

And, we use them... to the best of our knowledge and ability... every single time we "touch" any server!

We know there are probably others who have already "been there" before us when it comes to your IT infrastructure and maybe even your servers. We also acknowledge , and take advantage of that fact that we probably aren't the first and only technician who may know how your servers should be (and are) set up to work within your environment

We never assume to have more knowledge about your server setup, but we do assume you may have already been attached at one time or another to a possible unsuspecting source. We look for traces of keystroke loggers, back doors, logic bombs, and everything else we can look for prior to (and even after) installations occur .

That's specifically why and how we work "with the customer",  by getting all the knowledge we can about the upgrade..., before it occurs and also is how we help protect you after the changes have been made. We can also help insure you have all necessary required backups, legal documentation and more.... just in case you have ancountered a "Nate" or  MOU" at your business already.

Call us today to get a quote on our consulting services, "before" you make your next server migration or upgrade attempt. It may be the best call you ever placed.

DON'T BE A NATE!

Send mail to admin@couritech.com with questions or comments about this web site.

For Warranty Terms and Conditions information please click here! For Terms and Conditions of All Sales information please click here!
Copyright © 2007 Couri Technology - This website designed by Couri Technology This site last updated: 07/24/2011

Couri Technology (TM) - is completely independent of any business location or ownership!