Beckhoff-Based Network Input Boxes for the Gravesend Inn

We've been doing the Gravesend Inn haunted hotel since 1999, which means 2018 is our 20th year! Some years ago, I started a quest to get all the control systems for the entire attraction onto a network, and I completed that in 2011 (more here). And as the attraction has grown in popularity (we have been getting nearly 5000 attendees a year for the last few), in recent years I've been on a new mission to streamline, simplify and completely document all the control systems to maximize reliability (and so that I don't have to be in the building to babysit the systems every minute that the show is open). Last year, I moved all the video playback to Brightsign players, and that really went great (you can read about it here).  

One of the oldest systems we currently use are our "TCP I/O" boxes, which date back to 2004, and are based on Advantech ADAM 6050 Ethernet I/O units. These units are cheap and, after I navigated a documentation nightmare, the units we built have been working well for 13 years now, with only a couple units failing over that time. But in the last couple years, we've had a few intermittent problems, where the box wouldn't boot up properly or other similar issues.  So, I've been figuring to move onto a new system, and given our educational mission, I wanted to go with what was most widely used in our industry, and these days it seems to me that the most widely used I/O systems are based on Beckhoff technology.  The big theme parks use a ton of Beckhoff stuff, as do other major players like Tait, Hudson, etc etc.  In addition, our original boxes were 12VDC, which worked well with the initial alarm grade motion detectors we used, but my survey of the industry showed that 24VDC is much more commonly used. With all that in mind, four years ago, as funding allowed, I started purchasing some Beckhoff parts, and worked with students through the years, culminating in the completion of three boxes this past spring.  On the summer break I finally got a couple days to finish configuring and documenting the boxes, and as usual, before I put anything into the Gravesend Inn haunted house attraction, I stress test is for typically several days or a week--I want to find any problems now and not in October.

For the system, I worked with Brian Buck, our Beckhoff salesman and selected a BK9050 bus coupler, which connects the I/O bus to the Ethernet network and speaks MODBUS TCP; and a KL1809 "HD Bus Terminal", which has 16 channels of 24VDC digital input.

This Beckhoff system is reasonably priced (by industrial standards) and very flexible. A wide range of I/O is available and can be mixed in the same system; Beckhoff is also very aware of our market and even has a section on their website for stage and show applications

Many years ago, in our I/O systems, we used XLR cables.  But at the time we had a limited stock of XLR, so I eventually moved to Cat 5 for sensor connections when we built our TCP I/O boxes. I chose Cat 5 because it is cheap and could easily carry these small currents over the distances we need.

This worked fine, but inevitably confused the students (students often think that a cable dictates what is carried over it (rather than the other way around), so they often think that the simple contact-closure based sensors we use were speaking Ethernet), and this led to patching problems. Fortunately, since Ethernet is transformer isolated, the systems wouldn't be damaged if incorrectly patched, but it still lead to confusion. So, moving to the new I/O system also gave us an opportunity to change connectors.  After a lot of thought and research, we decided to go with "M12" connectors, which are commonly found on industrial sensors. 

These aren't used for show purposes so cross patching won't be a problem; also they are very robust once connected.  Most factory automation systems, in which these sensors are typically used, are not set up and struck every year, so key to us being able to use this approach was finding field-terminatable connectors like these so we can make extension cables. 

Setting IP Addresses

Beckhoff has a very clever method to change IP addresses using ARP commands (see my book for more information about ARP and associated network topics).  To start this process, you first factory reset the bus coupler by powering off, turning all the DIP switches on and removing any I/O and terminating only with the KL9010 end block:

You then power off the unit again, connect your I/O, turn all the DIP switches off and power up again; this resets the bus coupler to its default IP address of 172.16.17.0.

To change the IP using the ARP method, you first run arp -a command to find out the MAC address of the unit, then you delete the entry, then you create a new entry, and then you do a special ping command to set the address. 

The manual in section 4.4.3 says, "It is, however, only possible to alter addresses within the same network class"; I assume the reason for this is in the final ping command you have to ping outside your subnet, and that won't be possible without a router.  But I figured out that if you have a computer with two ethernet jacks you can set one to a class B address and another to class C, and then connect cables from both interfaces to a switch connected to the Beckhoff (Note: I assume this would also work if you changed the IP address of your machine mid configuration, but I didn't try this).

Update July 9: Jim Janninck learned me on the fact that you can assign multiple IP addresses to the same interface!  Very cool. Use the "Advanced" button on the IPV4 interface screen.

It will show up in IP Config thusly:

Here's a log of the commands which are detailed in the Beckhoff manual (replace your desired target IP with the 192.168.1.xxx address, some MAC and other addressed munged for security reasons)

First, let's run IPCONFIG to check the two IP addresses I configured into my PC:

C:\Windows\System32>ipconfig

Ethernet adapter LAN 2 PCI:

   Connection-specific DNS Suffix  . :
   IPv4 Address. . . . . . . . . . . : 192.168.1.123
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . :

Ethernet adapter LAN 1 Integrated:

   Connection-specific DNS Suffix  . : 
   IPv4 Address. . . . . . . . . . . : 172.16.17.123
   Subnet Mask . . . . . . . . . . . : 255.255.0.0
   Default Gateway . . . . . . . . . :

Now let's ping the newly factory reset bus coupler to get it in the ARP table:

C:\Windows\System32>ping 172.16.17.0

Pinging 172.16.17.0 with 32 bytes of data:
Reply from 172.16.17.0: bytes=32 time=3ms TTL=60 ...

Now let's run arp-a to get the MAC address of the bus coupler:
C:\Windows\System32>arp -a

Interface: 172.16.17.123 --- 0xb
  Internet Address      Physical Address      Type
  172.16.17.0           00-01-05-xx-xx-xx     dynamic

Interface: 192.168.111.123 --- 0xd
  Internet Address      Physical Address      Type

Now, let's delete the ARP table entry for the bus coupler:

C:\Windows\System32>arp -d 172.16.17.0

Now let's create a new ARP table entry

C:\Windows\System32>arp -s 192.168.1.xxx 00-01-05-xx-xx-xx

Let's check to make sure it's there (note that it associated with the 192.168 interface):

C:\Windows\System32>arp -a

Interface: 172.16.17.123 --- 0xb
  Internet Address      Physical Address      Type

 

Interface: 192.168.111.123 --- 0xd
  Internet Address      Physical Address      Type
  192.168.1.xxx        00-01-05-xx-xx-xx     static

Now let's run the special PING command that sets the address.  This is where you need the second interface (or you need to change your IP address to something in the new subnet range before you try this):

C:\Windows\System32>ping -l 123 192.168.1.xxx

Pinging 192.168.1.xxx with 123 bytes of data:
Reply from 192.168.1.xxx: bytes=123 time=622ms TTL=60
Reply from 192.168.1.xxx: bytes=123 time=2ms TTL=60

 

Now let's run ARP -a to check to make sure everything came through OK
C:\Windows\System32>arp -a

Interface: 172.16.17.123 --- 0xb
  Internet Address      Physical Address      Type

Interface: 192.168.111.123 --- 0xd
  Internet Address      Physical Address      Type

  192.168.1.xxx           00-01-05-xx-xx-xx     dynamic

And that's it!

BootP Server

An alternative to the ARP based method is the BootP Method , where a BootP server is used to assign addresses based on MAC addresses.  To allow the BootP method, you turn DIP switches 1-8 and 9 on (10 off) and once you configure the address, if you leave the DIP switches in this setting, “The address assigned by the BootP server is stored, and the BootP service will not be restarted after the next cold start. The address can be cleared again by reactivating the manufacturers' settings ...“ Beckhoff supplies a BootP server called “TCBootP Server”

Watchdog Timer

These Beckhoff modules have a watchdog timer for enhanced safety on outputs.  From Page 53 of the manual:  

The watchdog is active under the factory settings. After the first write telegram the watchdog timer is initiated, and is triggered each time a telegram is received from this device. Other devices have no effect on the watchdog. ...  The watchdog can be deactivated by writing a zero to offset 0x1120. The watchdog register can only be written if the watchdog is not active. The data in this register is retained.

Note it says the watchdog timer is activated by the first write "telegram" (MODBUS TCP write Coil or similar) so if you only ever do read operations (what we do for inputs on these boxes) the watchdog timer will never be started. Once the timer is started by a write command, if no MODBUS TCP command (read or write) is received within the timeout (typically 1 second) the unit will go into a fault state (red watchdog error LED will light) and the outputs will shut off.  You then have to go through a reset procedure to activate the unit again.

To disable the watchdog timer, you have to (as detailed above) write a zero to offset 0x1120.  Through experimentation I found that you have to do this before any write (output) commands are issued, but only after the unit has completely booted up. Watch for the activity LED on the Ethernet port to light up before issuing the command.

Note: If you trigger the watchdog timer (red light on), then issue a reset, and then write the timer to 0, then the unit will function with no watchdog functionality but the red Watchdog error light will remain red.

Connecting to Barco Medialon Manager  

The Gravesend Inn has been run by Barco/Medialon Manager show control software since 2002, and the way I always incorporate any new technology into the network is to first write a small, stand alone Manager program for the new device only, test that, and then import it into the larger Gravesend Inn program. Since this box is input only, and we are not doing any write operations, we disregard the watchdog timer. In Manager you have to configure both the I/O resource and a device pulling from that resource. 

You then assign the resources to a Medialon I/O device:

IOConfig.PNG

Reading and Writing

Note: The Register Numbers in Medialon are "off by one", meaning they are one higher than the Register Numbers in the Beckhoff. For example, Register 4385 (0x1121) in Medialon addresses Register 4384 (0x1120) in the Beckhoff.

Reading Inputs

The inputs on the device are read automatically by the I/O resource if configured as shown above.

Writing Outputs

We're not using this (and if you do, you have to manage the watchdog stuff) but for posterity here's the information: outputs are written using a “Write Coil” command starting at address 1

I ended up with a simple little polling program:

The boxes are undergoing stress testing now, and if we get funding to replace our cables, I plan to start transitioning the Gravesend Inn sensor systems over this year to the new boxes.

Several excellent students (many now alumni) worked with me on this project over the last four years. Michael Sauder, now at Radio City, did the initial testing on and configuration of the Beckhoff units from Medialon manager. Tim Durland, now at Smart Monkeys, expanded on Michael's work and figured out the watchdog timer issues. Woody Woytovich, now in his Local 1 IATSE internship, did the box layout and schematic and constructed the first one. Dominique Hunter, Bailin He, Neil Carman, and Daniel Santamaria all worked on constructing and wiring the last two boxes this spring. Thanks as always to Eric Cantrell of Barco/Medialon for helping us with protocol details on this project and Jim Janninck of Timberspring for advice on box construction and initial testing.

Note: Updated July 9, 2018 with updated ARP-based IP and Medialon Configuration; thanks to Eric and Jim for the updated info.

Controlling Brightsign Players from Medialon Manager for the Gravesend Inn

In the Gravesend Inn, we have several video playback systems. I originally developed the effects on Dataton Watchout, and this is a great and very powerful technology, but kind of overkill for this straightforward playback application. And the bigger problem over the years is that we would occasionally have problems with an HDMI or display port connector getting pulled out accidentally by a technician, or jostled out by overactive audience members (we mounted the computers near the displays, and the walls of the attraction get slammed),  All of this would then cause the display computer to un-recognize the monitor in Windows, and cause it to drop the display, or get into some weird state. This was infrequent but when it did happen it caused a situation that was kind of complicated to reset, with the keyboard on the other side of the scenic wall, the displays sideways, inexperienced student technicians, etc, etc.

So this year, I decided to move all the playback video effects over to Brightsign HD223 players that run on micro SD cards. These amazing little solid state, dedicated players cost about $350 and are made for 24/7 heavy duty digital signage applications; to reset them you just power cycle them. I went back and forth with the sales guy and couldn't really figure out how to control them, but then I ran into Drew Dalzell at IAAPA last year and he told me that they would work; he has used many of them in attractions all over the world. So I took the plunge, ordered them, and worked with my (now former) student Maxime Verdière to get them working over the last semester, and now with summer here got them debugged and tested out.  

What had been confusing me before I talked to Drew was that I was trying to find out what the control protocol was. I was thinking of the units as sort of video player. But in fact It's not really a video player, it's a digital signage system, with software custom developed for each application. So there isn't a standard control protocol--you create it as part of the Brightsign program; using a program called Brightauthor, you create a number of "States" which are then linked together:  You can then publish this code (with the media) to the box over the network (and the box has a very good network implementation, with default gateway, etc, which we need in this application).  We used the "Networked with Local File Networking" mode. 

The code above is for the "Dining Chamber" area of the Gravesend Inn. In this case, we need a still frame that looks like a painting, and then we run a video clip when the painting comes to life. For operational purposes we also needed a screen saver mode between shows. In the figure above, you can see the three states. The top one is the "still" frame (which is actually a one second video because we were having problems with smooth transitions from JPG to video files), the bottom left is the main video, and the bottom right is the screen saver state.

Each of the little connector icons is a UDP Input Event, which triggers the next state, such as:

So there really isn't any overall control protocol; there are events that you can use to transition from one state to another. This also means that (as far as I've figured out) you need to duplicate many of these events, because the UDP commands only act from the currently active state. That might be a bit confusing but that's why you see each state in the program above has three events out the bottom (one to trigger each of the other states and another to retrigger itself). This means, for example, you have to enter the same "play" command above three times into Brightauthor, which felt a little weird but seems rock solid in my testing.

Update June 28: Brandon from Brightsign tech support wrote: Just wanted to throw in a quick tip. For "global" events that should apply to multiple states, if you put all the states into a Super State, you can put the events on the Super State and they will be triggered as long as the playback has on a state in the Super State.

Our second setup in the haunted hotel is a bit more complicated, since it has three monitors that play in sync and two separate video files:

Brightsign offers a "brightwall" feature for making video walls, but in this case we already had three 1080p video files done that needed to run in sync, and we decided against using the brightwall and instead having the left screen mastering the other two. To do this, you put all the players in the same "sync domain" and then issue a "link" command from the master player as part of each UDP event:

Then on the slave players, instead of UDP events, you put Syncrhonization events that follow the master:

Instead of UDP strings, the sync events follow the linked out synchronization keyword:

This worked great, but we did notice one weird glitch.  Even though the master unit was set to hold its last frame at the end of video clips, the master (not the slaves) would loop the media forever in a jittery, weird way. Brightsign tech support was quick to respond with a solution--adding a "Media End" event on the master which triggered the idle (still) state.  Apparently this glitch was due to a tiny discrepancy between the lengths of the master and slave files and the system trying to resolve the difference.

To control all this from Barco/Medialon Manager was pretty straightforward--just creating the UDP commands:

I whipped up a little user screen, which includes a little feedback from the units that I created using the UDP Send commands from each state entry:

Because the fall is so busy, I typically introduce new technologies like this into the Gravesend Inn in the summer when I have time to debug everything and then test the hell out of it.  I made a heavy duty test program in Medialon and pounded all the players repeatedly, let them run over a weekend, and they all were rock solid. Once the code was debugged, the only issue I had was when the monitors were put into power save mode overnight, the monitors would power off.  Disabling some power off settings in the monitor seems to have fixed that; we'll see for sure this fall!