« ArianeB 6.4 | Main | Rod Harrod remembers The Crom »

20100723

True Story V, Atagong's continuing struggle with helpdesks...

Entry 1706 posted in: 6. Self-Made Monsters


Exceeded quota One of my daily jobs as an IT monkey, next to changing printer toner that for some members of the company I work for seems to be a task of gargantuan proportions, is to take care of the safety of our network.

Although our (rented) mail server has a sophisticated anti-virus and spam removal system (if I may believe the blurb), the firewall has a daily updated spam and anti-virus protection (if I may believe the expensive maintenance contract) and the individual workstations have a top-notch anti-virus suite there is always the odd chance that a Trojan or backdoor (real good old-fashioned viruses are so seldom, these days) slips through the system.

And that is what happened last Friday. As most of the users only have limited rights on their machines usually no harm is done. On top of that these virus-loaded mails can be spotted from a 10 miles distance, because they invariably use old tricks nobody falls for anymore, except when… but that is for later.

Sales car caviar

So when suddenly a machine started spitting out 900 mails in half an hour, using the infamous relay port 25, I had a pretty clear idea where shit had hit the fan. I didn’t need to consult Sherlock Holmes to figure out that this sudden burst of activity could only come from the division that, by definition, is the least active but has the most expensive cars, the most modern laptops, the highest expense accounts but the least discipline. I am of course speaking of the sales department.

For historical reasons sales people in our company must have the nec plus ultra without fully understanding the nec nor the ultra. Sometimes this can be interpreted more literally than you think. I was once present at a sales meeting where the managing director was giving a blurry motivational speech about how sales figures and the company’s future existence where going hand in hand. For about 20 minutes the MD spit out phrases like:
no sales figures ergo no turnover,
no turnover ergo no profit,
no profit ergo no jobs…
At the end he asked if there were any questions, the sales director raised her hand and asked: who is this new client Ergo you have been going on about? (I swear on both my testicles that this really happened.)

The sales division is the one that asked for the latest Microsoft Office Suite, because the 65536 lines limit of older Excel versions was really not enough to make 15 line offers and then, very seriously, phoned me to ask how to they had to calculate a sum. My answer, making my dodgy reputation as a grumpy IT manager only bigger, is invariably the same: you can use exactly the same formulae you used in Excel XP, Excel 2000, Excel 98 or Excel 95 for that matter. Not wanting to probe deeper into my dark cynical brain the person at the other end said ‘thank you’ and is probably still busy typing =A1+A2+A3+A4+A5+A6+A7+A8… Lets hope the columns he needed to add really didn’t hold 65536 plus one cells…

Somewhere last month I got a call from sales asking how they could find out if a certain name (in an Excel list) was present on a long list of names. So I told them, trying not to put an overt sarcastic tone in my voice, that perhaps the function to FIND a certain piece of information inside others pieces of information was the FIND function, just like the function to calculate a SUM in Excel is mysteriously called SUM.

Pie in the sky

And you will probably not believe me if I tell you that one of my side-jobs as an IT-manager is to receive the weekly turnover list of the sales people and turn it into a pie chart. Making a pie chart in Excel for them is what brain surgery looks like for me, apparently. (Don't dare interpreting the last sentence the wrong way!)

Last week I found out to my amusement that not one single person of our sales team was able to calculate a percentage, and after a 30 minutes discussion they decided to send a 10 lines Excel sheet over to me. I changed the wrong total for the good one (I wasn't bluffing when I told you they don't know how to make a sum) and suddenly it all added to 100%. It’s nice to be known as a computer genius.

Google wants you

But the bottom line was that one of our sales laptops was spitting out a backdoor, a Trojan or whatever you call these things nowadays, at an incredible speed. I located the PC in a jiffy and found out that the mail in question had been send (apparently) from Google containing a job offer. Obviously the sender's address had been spoofed and the attachment contained not an application form but a malicious program.

For my own intellectual sanity, and because it is rather hard to believe anyway, let me rephrase the previous paragraph. One of our sales persons saw a mail from the Google headquarters coming in, genuinely believing that, out of the billions of persons on this Earth, she had been chosen personally by Eric E. Schmidt to work for them, unzipped the attached messages and ran the exe file that was hiding inside. If creationists need proof that Darwin's evolutionary theory is humbug they just need to come and visit the company I work for.

It really didn’t take me a lot of time to neutralise the backdoor, although the sales person in question was constantly nagging that she had loads of work to do and that she was missing the sale of the century due to my intervention, but then there was still the matter that my anti-virus providers, all 3 of them, had failed me.

Helpdesk Blues #1

I started with the mail server guys. I sent a mail to the anti-virus mailbox but antivirus@belbone.be replied that it didn’t exist anymore. Fair enough, I don't use this address very often and it might have changed since a couple of years ago.

Time to call the Belgacom helpdesk, Belgium’s biggest telecom operator.

“I would like to point out that your professional spam and anti-virus filter, that I actually pay for, let through a harmful mail last Friday, and that today, on Monday these mails are still slipping through the maze.”

As usual the voice at the other side was very friendly and very professional:
“Please send a mail to abuse@belgacom.be and they will look into the matter.”

I like it when helpdesk people are efficient like that. So I did what they asked. Not five minutes later I got a return message and it read something along these lines:

This mailbox has exceeded its quota.
The exchange server will not attempt to send it again.
Thanks for your comprehension.

It was very reassuring realising that the biggest telecom provider of Belgium hadn't been checking its abuse mailbox for the last couple of weeks.

Helpdesk Blues #2

Time for Plan B. I knew where the original mail, containing the worm, had come from (not from Google, obviously) and I send a mail to abuse@versatel.be, but that mail address also was invalid. A WHOIS lookup showed me that Versatel was now in the hands of KPN.

Time to call the KPN helpdesk. To my amazement the KPN helpdesk lead me to Mobistar, Belgium’s second biggest telecom operator. A very friendly and professional man tried to help me.

"The IP range you gave me is not one of ours.", he said. "We have indeed taken over the professional branch of the KPN business, but the home consumer market has been taken over by Base. I’ll give you the helpdesk number of Base Consumer Market."

Helpdesk Blues #3

Time to call the Base helpdesk, Belgium’s third biggest telecom operator. The phone guy was very professional and very friendly.

"I can see it is one of our routers", he said, "but as the IP addresses are dynamically given whenever someone connects we will need the exact headers of the mail in question."

"I can give you all that.", I replied, "But what are you going to do, as the person probably is not even aware he or she is sending Trojans around?"

"After locating the router in question we will monitor it and if this person is still sending viruses around we will contact him or her and in the worst possible case switch off the router from a distance until the problem has been solved."

"Sounds fine to me, were do you want me to send the mail headers?"

"Support@base.be, sir. We will immediately take care of it, glad to be of assistance to you."

Minutes after I send the information I heard a reassuring ping. It was a message from Base. It read:

We are sorry we can’t deliver your mail, as this mailbox no longer exists.

The world is in safe hands, I can assure you that.


Other helpdesk stories on this blog:
True Story
True Story (the sequel)
True Story (part 3, the horror returns) 
When in shit, call IT (True Story IV)