The Circle of HOPE (Hackers on Planet Earth) Conference Wrapup

Chelsea Manning speaking at the 2018 Circle of HOPE Conference

This was my eighth Hackers on Planet Earth (HOPE) Conference; I've presented talks at four conferences, and have assisted the A/V and Lighting teams in various ways over many years (writeups of past conferences and talks here). These remarkable, volunteer-run conferences, held every two years, feature knowledge, sharing, beauty, joy, community, and--always--a dose of chaos. The conferences are always inspiring, and--this year's alt-right trolls aside--I think this was the best HOPE yet, although you wouldn't know that if you only read about the conference on Twitter.

I was there all three days of the conference, went to probably a dozen talks in all three rooms, walked around the exhibits area, talked to many former and current students running the A/V, and other friends who are involved in running the conference. And I wouldn't have even known the level of controversy if I wasn't also active on twitter. What happened? Some alt-right trolls came in and disrupted the conference and intimidated a few attendees. None of that is good, of course, and the volunteer HOPE staff could have dealt with the situation better.  But I agree with the core idea of the conference organizers in that the answer to hate speech is more speech--not banning the wearing of certain hats (or people wearing them) as some people are demanding on twitter. 

Here's a pretty good summary of the situation, a support of HOPE and way forward by @JairusKhan on twitter (the whole thread starts here). Two important key phrases for me from Mr. Khan's thread was that after the incident "...HOPE should be considered under attack by [alt-right] agent provocateurs.", and ending with "But the reason why this all happened isn't because Nazis think #hopeconf is a safe space. It's because HOPE is where the people fighting Nazis are sharpening their swords.": 

 

I'm pretty much in agreement with Mr. Khan, and I don't have a lot to add, but I hope people will understand that this deplorable situation--intentionally caused by few people determined to disrupt-was a pretty small negative part of a conference that was otherwise overwhelmingly positive.

Of course one of the highlights of the conference for me was an inspiring talk from Chelsea Manning (photo above). She clearly, like many of us, was more comfortable as a geek in the shadows than as a public figure.  But she is now embracing her current role and bravely standing up to the many who hate and attack her. I was very inspired listening to her. 

I also enjoyed pretty much every talk I attended, all of which will be online soon (click on any photo to enlarge):

In the closing ceremonies (which I watched online yesterday since I was helping strike the other rooms while it was happening live), HOPE ringleader Emmanuel Goldstein, no stranger to controversy, seemed exhausted by the situation and afterwards even seemed to be questioning whether to even continue onwards.  I've long respected the way Emmanuel handles controversy; I think back to the way he and the HOPE core team handled a remarkable appearance in The Next HOPE in 2010 by Adrian Lamo, who had a major role in Chelsea (then Bradley) Manning going to prison (my writeup and photos of that appearance here). Lamo was widely hated by many at the conference, but in the Q&A part of the talk, Emmanuel, as moderator, demanded respect for Lamo while still allowing challenging questions to be asked.  This was remarkable and inspiring and to me represents the way forward.  Those who are currently attacking HOPE on twitter should keep this spirit in mind. Don't let the trolls divide us, let's rally around our community and overwhelm the hate with love and understanding. If trolls want to attend the conference and act respectfully, let them--they might even learn something. In this environment, we all need HOPE more than ever.

JH-20180722-IMG_20180722_195350.jpg

Beckhoff-Based Network Input Boxes for the Gravesend Inn

We've been doing the Gravesend Inn haunted hotel since 1999, which means 2018 is our 20th year! Some years ago, I started a quest to get all the control systems for the entire attraction onto a network, and I completed that in 2011 (more here). And as the attraction has grown in popularity (we have been getting nearly 5000 attendees a year for the last few), in recent years I've been on a new mission to streamline, simplify and completely document all the control systems to maximize reliability (and so that I don't have to be in the building to babysit the systems every minute that the show is open). Last year, I moved all the video playback to Brightsign players, and that really went great (you can read about it here).  

One of the oldest systems we currently use are our "TCP I/O" boxes, which date back to 2004, and are based on Advantech ADAM 6050 Ethernet I/O units. These units are cheap and, after I navigated a documentation nightmare, the units we built have been working well for 13 years now, with only a couple units failing over that time. But in the last couple years, we've had a few intermittent problems, where the box wouldn't boot up properly or other similar issues.  So, I've been figuring to move onto a new system, and given our educational mission, I wanted to go with what was most widely used in our industry, and these days it seems to me that the most widely used I/O systems are based on Beckhoff technology.  The big theme parks use a ton of Beckhoff stuff, as do other major players like Tait, Hudson, etc etc.  In addition, our original boxes were 12VDC, which worked well with the initial alarm grade motion detectors we used, but my survey of the industry showed that 24VDC is much more commonly used. With all that in mind, four years ago, as funding allowed, I started purchasing some Beckhoff parts, and worked with students through the years, culminating in the completion of three boxes this past spring.  On the summer break I finally got a couple days to finish configuring and documenting the boxes, and as usual, before I put anything into the Gravesend Inn haunted house attraction, I stress test is for typically several days or a week--I want to find any problems now and not in October.

For the system, I worked with Brian Buck, our Beckhoff salesman and selected a BK9050 bus coupler, which connects the I/O bus to the Ethernet network and speaks MODBUS TCP; and a KL1809 "HD Bus Terminal", which has 16 channels of 24VDC digital input.

This Beckhoff system is reasonably priced (by industrial standards) and very flexible. A wide range of I/O is available and can be mixed in the same system; Beckhoff is also very aware of our market and even has a section on their website for stage and show applications

Many years ago, in our I/O systems, we used XLR cables.  But at the time we had a limited stock of XLR, so I eventually moved to Cat 5 for sensor connections when we built our TCP I/O boxes. I chose Cat 5 because it is cheap and could easily carry these small currents over the distances we need.

This worked fine, but inevitably confused the students (students often think that a cable dictates what is carried over it (rather than the other way around), so they often think that the simple contact-closure based sensors we use were speaking Ethernet), and this led to patching problems. Fortunately, since Ethernet is transformer isolated, the systems wouldn't be damaged if incorrectly patched, but it still lead to confusion. So, moving to the new I/O system also gave us an opportunity to change connectors.  After a lot of thought and research, we decided to go with "M12" connectors, which are commonly found on industrial sensors. 

These aren't used for show purposes so cross patching won't be a problem; also they are very robust once connected.  Most factory automation systems, in which these sensors are typically used, are not set up and struck every year, so key to us being able to use this approach was finding field-terminatable connectors like these so we can make extension cables. 

Setting IP Addresses

Beckhoff has a very clever method to change IP addresses using ARP commands (see my book for more information about ARP and associated network topics).  To start this process, you first factory reset the bus coupler by powering off, turning all the DIP switches on and removing any I/O and terminating only with the KL9010 end block:

You then power off the unit again, connect your I/O, turn all the DIP switches off and power up again; this resets the bus coupler to its default IP address of 172.16.17.0.

To change the IP using the ARP method, you first run arp -a command to find out the MAC address of the unit, then you delete the entry, then you create a new entry, and then you do a special ping command to set the address. 

The manual in section 4.4.3 says, "It is, however, only possible to alter addresses within the same network class"; I assume the reason for this is in the final ping command you have to ping outside your subnet, and that won't be possible without a router.  But I figured out that if you have a computer with two ethernet jacks you can set one to a class B address and another to class C, and then connect cables from both interfaces to a switch connected to the Beckhoff (Note: I assume this would also work if you changed the IP address of your machine mid configuration, but I didn't try this).

Update July 9: Jim Janninck learned me on the fact that you can assign multiple IP addresses to the same interface!  Very cool. Use the "Advanced" button on the IPV4 interface screen.

It will show up in IP Config thusly:

Here's a log of the commands which are detailed in the Beckhoff manual (replace your desired target IP with the 192.168.1.xxx address, some MAC and other addressed munged for security reasons)

First, let's run IPCONFIG to check the two IP addresses I configured into my PC:

C:\Windows\System32>ipconfig

Ethernet adapter LAN 2 PCI:

   Connection-specific DNS Suffix  . :
   IPv4 Address. . . . . . . . . . . : 192.168.1.123
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . :

Ethernet adapter LAN 1 Integrated:

   Connection-specific DNS Suffix  . : 
   IPv4 Address. . . . . . . . . . . : 172.16.17.123
   Subnet Mask . . . . . . . . . . . : 255.255.0.0
   Default Gateway . . . . . . . . . :

Now let's ping the newly factory reset bus coupler to get it in the ARP table:

C:\Windows\System32>ping 172.16.17.0

Pinging 172.16.17.0 with 32 bytes of data:
Reply from 172.16.17.0: bytes=32 time=3ms TTL=60 ...

Now let's run arp-a to get the MAC address of the bus coupler:
C:\Windows\System32>arp -a

Interface: 172.16.17.123 --- 0xb
  Internet Address      Physical Address      Type
  172.16.17.0           00-01-05-xx-xx-xx     dynamic

Interface: 192.168.111.123 --- 0xd
  Internet Address      Physical Address      Type

Now, let's delete the ARP table entry for the bus coupler:

C:\Windows\System32>arp -d 172.16.17.0

Now let's create a new ARP table entry

C:\Windows\System32>arp -s 192.168.1.xxx 00-01-05-xx-xx-xx

Let's check to make sure it's there (note that it associated with the 192.168 interface):

C:\Windows\System32>arp -a

Interface: 172.16.17.123 --- 0xb
  Internet Address      Physical Address      Type

 

Interface: 192.168.111.123 --- 0xd
  Internet Address      Physical Address      Type
  192.168.1.xxx        00-01-05-xx-xx-xx     static

Now let's run the special PING command that sets the address.  This is where you need the second interface (or you need to change your IP address to something in the new subnet range before you try this):

C:\Windows\System32>ping -l 123 192.168.1.xxx

Pinging 192.168.1.xxx with 123 bytes of data:
Reply from 192.168.1.xxx: bytes=123 time=622ms TTL=60
Reply from 192.168.1.xxx: bytes=123 time=2ms TTL=60

 

Now let's run ARP -a to check to make sure everything came through OK
C:\Windows\System32>arp -a

Interface: 172.16.17.123 --- 0xb
  Internet Address      Physical Address      Type

Interface: 192.168.111.123 --- 0xd
  Internet Address      Physical Address      Type

  192.168.1.xxx           00-01-05-xx-xx-xx     dynamic

And that's it!

BootP Server

An alternative to the ARP based method is the BootP Method , where a BootP server is used to assign addresses based on MAC addresses.  To allow the BootP method, you turn DIP switches 1-8 and 9 on (10 off) and once you configure the address, if you leave the DIP switches in this setting, “The address assigned by the BootP server is stored, and the BootP service will not be restarted after the next cold start. The address can be cleared again by reactivating the manufacturers' settings ...“ Beckhoff supplies a BootP server called “TCBootP Server”

Watchdog Timer

These Beckhoff modules have a watchdog timer for enhanced safety on outputs.  From Page 53 of the manual:  

The watchdog is active under the factory settings. After the first write telegram the watchdog timer is initiated, and is triggered each time a telegram is received from this device. Other devices have no effect on the watchdog. ...  The watchdog can be deactivated by writing a zero to offset 0x1120. The watchdog register can only be written if the watchdog is not active. The data in this register is retained.

Note it says the watchdog timer is activated by the first write "telegram" (MODBUS TCP write Coil or similar) so if you only ever do read operations (what we do for inputs on these boxes) the watchdog timer will never be started. Once the timer is started by a write command, if no MODBUS TCP command (read or write) is received within the timeout (typically 1 second) the unit will go into a fault state (red watchdog error LED will light) and the outputs will shut off.  You then have to go through a reset procedure to activate the unit again.

To disable the watchdog timer, you have to (as detailed above) write a zero to offset 0x1120.  Through experimentation I found that you have to do this before any write (output) commands are issued, but only after the unit has completely booted up. Watch for the activity LED on the Ethernet port to light up before issuing the command.

Note: If you trigger the watchdog timer (red light on), then issue a reset, and then write the timer to 0, then the unit will function with no watchdog functionality but the red Watchdog error light will remain red.

Connecting to Barco Medialon Manager  

The Gravesend Inn has been run by Barco/Medialon Manager show control software since 2002, and the way I always incorporate any new technology into the network is to first write a small, stand alone Manager program for the new device only, test that, and then import it into the larger Gravesend Inn program. Since this box is input only, and we are not doing any write operations, we disregard the watchdog timer. In Manager you have to configure both the I/O resource and a device pulling from that resource. 

You then assign the resources to a Medialon I/O device:

IOConfig.PNG

Reading and Writing

Note: The Register Numbers in Medialon are "off by one", meaning they are one higher than the Register Numbers in the Beckhoff. For example, Register 4385 (0x1121) in Medialon addresses Register 4384 (0x1120) in the Beckhoff.

Reading Inputs

The inputs on the device are read automatically by the I/O resource if configured as shown above.

Writing Outputs

We're not using this (and if you do, you have to manage the watchdog stuff) but for posterity here's the information: outputs are written using a “Write Coil” command starting at address 1

I ended up with a simple little polling program:

The boxes are undergoing stress testing now, and if we get funding to replace our cables, I plan to start transitioning the Gravesend Inn sensor systems over this year to the new boxes.

Several excellent students (many now alumni) worked with me on this project over the last four years. Michael Sauder, now at Radio City, did the initial testing on and configuration of the Beckhoff units from Medialon manager. Tim Durland, now at Smart Monkeys, expanded on Michael's work and figured out the watchdog timer issues. Woody Woytovich, now in his Local 1 IATSE internship, did the box layout and schematic and constructed the first one. Dominique Hunter, Bailin He, Neil Carman, and Daniel Santamaria all worked on constructing and wiring the last two boxes this spring. Thanks as always to Eric Cantrell of Barco/Medialon for helping us with protocol details on this project and Jim Janninck of Timberspring for advice on box construction and initial testing.

Note: Updated July 9, 2018 with updated ARP-based IP and Medialon Configuration; thanks to Eric and Jim for the updated info.