The Install from Hell

Date Received: 18 Dec 1996
From: Jon Martin
Subject: cleaning crew

I am not in a happy mood right now - my old place just screwed up two decent working systems that my old customers liked. I got a call asking me if I would consider going back and helping straighten out the mess. Unfortunately, NO! I did get a call from a Navy investigator about a complaint lodged by my old company - you might get a kick out of this email message that I sent around to my crew here and the Naval investigator. At least on the second system they were able to put it back the way it was.....

From: MARTIN J. on Wed, Dec 18, 1996 10:21 AM
Subject: install from hell
To: *EMS Software

I thought that you might like this one - I got a call last night from my old customers (NADEP - Cherry Point). They just had a software install go very badly, and penalized my old company by removing all performance bonuses (I.E. - work was done at cost - no profit). The new head of my old place (whom I have never met) protested (officially) that one install that I had done went exceedingly badly, but my team got praise and the highest performance bonus allowed under the contract. The navy investigator heard about a memo I wrote explaining why the three day install took over two weeks. While I could not find the memo - the following bullet points were sent out email to him.

Needless to say - this was the install from Hell.

  1. Wrong software delivered by the CCC (change control staff) had to wait for "official delivery" of correct software - my personal backups were not allowed to be installed. One day delay. (I was not allowed to verify according to the install rules - all tapes had to be cut by the Change Control people. Who took the entire system for the Charleston Shipyard and loaded it onto tape instead of the Cherry Point automation system).
  2. "Official" backups done by the CCC did not work. (Loaded my personal copies w/ permission of Contract Administrator).
  3. Air Conditioning in computer room went out - computers had to be shut down until AC repaired - one day delay for compressor to be flown in.
  4. Power went out causing systems to be rebuilt/restored after crash.
  5. Flood happened when exaust fan on roof was doused with water by the fire department. Equipment down until DEC field service approved startup.
  6. More flooding happened when water line burst the day after the fire - possibly damaged by the fire. Water line also supplied the AC unit - causing that to fail again. DEC field service too busy to come out that day.
  7. Smoke from new exaust fan on roof caused evacuation for 1/2 day.
  8. Base wide network went down due to router upgrade - 1/2 day.
  9. Customer supplied terminal emulater was no good - had to rebuild our terminal control units.
  10. Network group had their offices and equipment room condemmed. Two days to remove/rebuild equipment and software in new temporary area.
  11. Non-stop fault tolerant Tandem computer went down. Two days repair. We could not run without being fed data from the Tandem!
  12. Primary developer considered a "security risk" due to camera in attache case - was not allowed on base or out from arrest for a day until paperwork was blessed by the base commander.
  13. Harrier crash by the building - building evacuated until wreckage removed.
  14. Visit from Air Force General to see facility--no work allowed during visit.
  15. Base computer repair/incoming inspection team loaded DOS onto our workstations before accepting. They did this without contacting us. Causing us to reload 26 workstations with QNX/FasTECH and source code, as well as recompile 26 different control programs. (12 hour compiles). Only one set of install disks meant a two hour install in sequence.
  16. Cables & cabling equipment FedEx'd into the base were signed for then disposed of by NADEP receiving. All had to be reordered.
  17. Primary system integrator/project lead left to have baby. (this was me - JM)

So we had fire, flood, airplane crash, smoke damage, and a building condemmed. Not to mention a couple of days due to the ^&*(&^ government employees getting a fedex shipment, then sending the shipment out to the disposal group who got rid of the cabling! So equipment we had sent on two weeks ahead of us arriving had to be reordered the day we got there and found out that it had never been delivered to the department. Not bad for one simple install, huh?

I think that this should cover the matter - after all, my team's software worked w/only one hitch once we got installed. And that hitch was due to a system load that we could not duplicate at the office.

This page last updated .