Date Received: 18 Dec 1996
From: Jon Martin
Subject: cleaning crew
I am not in a happy mood right now - my old place just screwed up two decent
working systems that my old customers liked. I got a call asking me if I would
consider going back and helping straighten out the mess. Unfortunately, NO! I
did get a call from a Navy investigator about a complaint lodged by my old
company - you might get a kick out of this email message that I sent around
to my crew here and the Naval investigator. At least on the second system they
were able to put it back the way it was.....
From: MARTIN J. on Wed, Dec 18, 1996 10:21 AM
Subject: install from hell
To: *EMS Software
I thought that you might like this one - I got a call last night from my old
customers (NADEP - Cherry Point). They just had a software install go very
badly, and penalized my old company by removing all performance bonuses (I.E. -
work was done at cost - no profit). The new head of my old place (whom I have
never met) protested (officially) that one install that I had done went
exceedingly badly, but my team got praise and the highest performance bonus
allowed under the contract. The navy investigator heard about a memo I wrote
explaining why the three day install took over two weeks. While I could not find
the memo - the following bullet points were sent out email to him.
Needless to say - this was the install from Hell.
- Wrong software delivered by the CCC (change control staff) had to
wait for "official delivery" of correct software - my personal backups were not
allowed to be installed. One day delay. (I was not allowed to verify according
to the install rules - all tapes had to be cut by the Change Control people.
Who took the entire system for the Charleston Shipyard and loaded it onto tape
instead of the Cherry Point automation system).
- "Official" backups done by the CCC did not work. (Loaded my personal
copies w/ permission of Contract Administrator).
- Air Conditioning in computer room went out - computers had to be shut
down until AC repaired - one day delay for compressor to be flown in.
- Power went out causing systems to be rebuilt/restored after crash.
- Flood happened when exaust fan on roof was doused with water by the
fire department. Equipment down until DEC field service approved startup.
- More flooding happened when water line burst the day after the fire -
possibly damaged by the fire. Water line also supplied the AC unit - causing
that to fail again. DEC field service too busy to come out that day.
- Smoke from new exaust fan on roof caused evacuation for 1/2 day.
- Base wide network went down due to router upgrade - 1/2 day.
- Customer supplied terminal emulater was no good - had to rebuild our
terminal control units.
- Network group had their offices and equipment room condemmed. Two days to
remove/rebuild equipment and software in new temporary area.
- Non-stop fault tolerant Tandem computer went down. Two days repair.
We could not run without being fed data from the Tandem!
- Primary developer considered a "security risk" due to camera in
attache case - was not allowed on base or out from arrest for a day until
paperwork was blessed by the base commander.
- Harrier crash by the building - building evacuated until wreckage
removed.
- Visit from Air Force General to see facility--no work allowed during
visit.
- Base computer repair/incoming inspection team loaded DOS onto our
workstations before accepting. They did this without contacting us. Causing us
to reload 26 workstations with QNX/FasTECH and source code, as well as
recompile 26 different control programs. (12 hour compiles). Only one set of
install disks meant a two hour install in sequence.
- Cables & cabling equipment FedEx'd into the base were signed for then
disposed of by NADEP receiving. All had to be reordered.
- Primary system integrator/project lead left to have baby. (this was
me - JM)
So we had fire, flood, airplane crash, smoke damage, and a building
condemmed. Not to mention a couple of days due to the ^&*(&^ government
employees getting a fedex shipment, then sending the shipment out to the
disposal group who got rid of the cabling! So equipment we had sent on two
weeks ahead of us arriving had to be reordered the day we got there and found
out that it had never been delivered to the department. Not bad for one simple
install, huh?
I think that this should cover the matter - after all, my team's software
worked w/only one hitch once we got installed. And that hitch was due to a
system load that we could not duplicate at the office.