Oh, sure, NOW you tell me it's dangerous!

This sysadmin has responsibility for a Web-based financial services platform, and his company's customers use it as a mission-critical application for their own customers, reports a pilot fish in the know.

"The sysadmin had no assistance and was overwhelmed by work already in the queue," says fish.

"One of the customers wanted some customization. Sysadmin sent the work documents to this customer to obtain the necessary setup information."

But the customer complains to sysadmin's CEO that he shouldn't have to follow the same process as other customers do. CEO decides to speed things up by hiring a consultant -- without talking to the sysadmin, who only learns of it when he gets a call to set up user access for the consultant.

What's the consultant going to be doing? sysadmin asks. He'll just be doing some customization for a customer, sysadmin is told. But don't worry -- overseeing his work is the CEO's responsibility.

Sysadmin calls the consultant to give him a 15-minute overview of how this particular installation is coded, and lays down some ground rules: The consultant should not, in any way, alter existing IT infrastructure. Then sysadmin sets up the login.

"The consultant implemented the customization, but understood from the CEO that he could respond to other requests from the customer," fish says. "The customer jumped at the opportunity and asked the consultant to alter core infrastructure that is shared with all customers.

"Working late one night, the consultant -- in innocence, with limited knowledge of the current platform infrastructure, contrary to explicit instructions to not touch any existing infrastructure, and with no oversight -- changed many shared constructs."

The result is catastrophic: Every other customer's implementation of the platform is disabled.

Next morning, sysadmin scrambles to get everyone working again. It turns out there are lots of side effects from the consultant's changes, and when sysadmin calls the platform vendor, support says there's no safe way to roll back all those changes. The only option is to restore everything from backups.

That will lose any work done since the backup -- and the vendor won't guarantee that customer data will survive intact. And there will be a hefty charge from the vendor to do the work.

Sighs fish, "The really funny part? The CEO lamented that the platform was 'more dangerous than he thought'!"

Sharky minimizes the danger by filing the identifying marks off every true tale of IT life. Send me yours at sharky@computerworld.com [1]. You'll get a stylish Shark shirt if I use it. Add your comments below, and read some great old tales in the Sharkives [1].

The Best of Shark Tank includes more than 70 tales of IT woe submitted by you, our readers, since 1999. Which all goes to prove, conclusively, that hapless users and idiotic bosses are indeed worldwide phenomena. Free registration is all that's needed to download [2]The Best of Shark Tank (PDF) [3].

via blogs.computerworld.com