System Troubleshooting
Note: I’m in the process of extracting the Show Control information out of my “big book” Show Networks and Control Systems into a new book. In the process, I’m retiring some of the information from the book and instead posting it here so it’s available.
The extract below was originally on page 114 of the second (2017) edition of my book Show Networks and Control Systems, Chapter 13: System Design Principles.
Troubleshooting is often a daunting challenge, but it needn’t be. In fact, I find it fun (as long as no one’s breathing down my neck). Because every system is different, there are no real standard ways to troubleshoot anything, but I’ve developed a basic approach which seems to work in a variety of situations. I’ve included it here in case you might find it useful.
Don’t Panic!
First, Don’t Panic! In my experience, the vast majority of system problems are simple power, configuration, or connection problems. Entertainment control equipment is mostly very reliable, and connections, cabling, and power are often the cause of many problems.
Before Starting
Before starting to troubleshoot, be sure that:
You have a signal flow diagram.
You understand the signal flow through the system.
The system is capable of working and was working.
You can conceptually break down the system into functional parts.
You have a known test signal source (audio player, signal generator, data source, etc.)
You know how to use test systems, software, and equipment such as oscilloscopes and multi-meters, Ethernet testers, DMX testers, and so on. These systems allow you to look inside complex systems, and without them, there is little you can do to understand what’s really going on in computerized, networked systems.
Cure the Problem and Not the Symptom
Try to cure the problem and not the symptom. There are times when taking a shortcut to cure a symptom rather than a problem is a necessary course of action, such as five minutes before show time. But I’ve found such shortcuts will come back to bite you eventually, so go after the problem whenever possible.
A Troubleshooting Process
After taking care of everything listed above, following is the process I use for troubleshooting, with only one person at a time in charge of troubleshooting:
1) Verify the Tester
Verify the test signal source!
Test your test equipment. (Is your meter or tester in the right mode? Is it broken?)
2) Quantify the Problem
Is no signal at all coming out of the system?
Is the signal coming through somehow changed?
Is the system working as anticipated?
What part of the system is not working as anticipated?
3) Check the Obvious
Is each device in the system turned on and powered up? Ensure that all the power lights and other indicators are showing correct operation.
Are all the connectors connected properly?
Are the output and input indicator lights or meters for each device in the system indicating correct signal function?
Will the system work as designed? Did it ever work? If it’s a new system that is not functioning, it’s possible that there was a design flaw and the system cannot work. If it did work previously, simplify the system as much as possible—bypass any unnecessary equipment or features.
4) Determine “Verified,” “Unverified,” and “Suspect” Components
“Verified” components are those you can determine definitively to be working; “Unverified” are those you have not yet tested. “Suspect” components are those you have tested, but don’t seem to be working as expected.
5) Changing Only One Thing at a Time, Go Through the System Until You Have Verified Every Component
Substitute known good (tested!) parts or components for suspect ones, but change only one thing at a time (if you change two things, you don’t know which one was really bad). Work logically, systematically, and carefully through the system until every device, component, or connection has been verified. This should allow you to find the problem, or at least lead you closer to the answer. Also, it’s often best to work through these steps by yourself, or with a very small group. Other people may have great ideas, but they may add confusion when you are trying to reduce the problem to a single variable.
6) If You’re Stuck, Clear Your Head
If you have the system design in your head and understand it, but you’re still stumped, take a break. Walk around the block, go have lunch. I can’t tell you the number of solutions that have come to me as soon as I walked away from the problem and thought about something else. Clearing your head leaves room for a new solution, or at least a new troubleshooting direction.
7: Consult Others
I started out here saying that only one person should be in charge of a troubleshooting process, and I believe this strongly. Other people—even really smart people—will have their own process and may change something you don’t know about, derailing your process. But when you get stumped, consult others. Often, if you have been looking at something for too long, you can no longer see the obvious. If you’re still stuck, consult an expert.