What Happened In San Diego at the Big Bay Boom? Garden State Fireworks Explains
I'm sure by now you've seen this from the Big Bay Boom :
There's a very good article on NJ.com by Louis C. Hochman that outlines the challenges of doing a show of this scale and discusses Garden State Fireworks' explanation.
You should read the entire, detailed press release, which I have here. On a very quick read, it sounds like human/process error (rather than equipment failure), explained in passive voice as, "an unintentional additional procedural step occurred." This apparently occurred as part of an effort to maintain a backup system in case of a failure across the 14 mile wide show site, which is certainly part of good show practice.
Here's an excerpt:
The San Diego Big Bay Boom is one of the most logistically complex displays in the world due to the approximate 14 mile span of the five locations involved in the display. In order to simultaneously coordinate all locations to a radio simulcast, state of the art pyrotechnic computerized systems were utilized.
In an effort to be over-prepared for any disruptions in communications, a secondary back up computer file was created for the display. The technical explanation of the creation and implementation of the back-up file requires a significant understanding of the computer firing systems themselves. Basically, if any of the five locations do not receive the initiation code, the secondary file allows each of the five locations the opportunity to independently 'manually' initiate a new sequence that will allow for seamless synchronization to all locations. Our technicians out on the barges have an independent protocol to activate this sequence if any of the automatic test sequences don't confirm open communication. Both the technical aspect and logic of this failsafe are all sound implementations of the computerized firing systems. This back up method has been executed successfully on countless occasions by our company.
After extensive forensic analysis by the hardware and software manufacturer and our own review of our implementation of the equipment, a full understanding of the simultaneous ignition has been achieved. The primary and secondary files were tested ad nauseam by our technicians in the weeks and hours leading up to the display and everything was perfect. All aspects of the display in the bay were completed and tested two full days in advance of the production and we were completely prepared to execute as normal.
Each of the five locations involved in the display have their own computer controller into which the files are downloaded. Before the two files are loaded into each of the five computer controllers, the primary and the secondary file are merged through the software to create a new file that is then loaded into each of the controllers. During the downloading process, an unintentional additional procedural step occurred in the loading process which allowed the creation of an anomaly that 'doubled' the primary firing sequence. The primary sequence then consisted of a sequence that would fire the entire display simultaneously and then proceed to fire the display in the proper sequence.
The time for the presentation arrived and all systems were ready to go. The design of the code had an additional layer of test measures built in that would provide all operators visual confirmation that all systems were receiving code a full five minutes before the production was to begin. So effectively, the production was designed to start running 'in the dark' five minutes prior to the actual launch time. All clearance was obtained from Police, Fire and Coast Guard to begin the pre-roll command sequence and prior to initiation of same, all zones and personnel were in a safe lockdown. The command code was initiated, and the 'new' file did exactly what it 'thought' it was supposed to do. It executed all sequences simultaneously because the new primary file contained two sets of instructions. It executed the file we designed as well as the file that was created in the back up downloading process
I commend Garden State Fireworks for being so open and honest about the details! I'm sure they feel horrible about this, but I'm glad that they didn't lawyer up and shut all the information down. Failures and mistakes are part of our business; we can all learn from situations like this.