At INBOUND15, Our Two Worlds Collide

A strange thing happened two weeks ago.

Hubspot, the web product we use to manage our marketing analytics and CRM (customer relationship management), held a convention called INBOUND in Boston, as they do every year.

INBOUND was a large, world class corporate conference. 14,000 people attended. There were dozens of intriguing panels and talks. Headline speakers included Aziz Ansari, Amy Schumer, and Todd Rowe of Google, among many, many others. The scenic design of the show and conference hall was superb, as was the A/V design and services.

What’s strange about that?

Well, once we got to the conference we realized we knew a lot of the people working backstage—literally behind the big “INBOUND” letters perched up on stage in Hall B—the people that make the show go off without a hitch.

For us, INBOUND15 was far more than a conference. It was living proof that our customer outreach and marketing using the Hubspot platform is working, and a great chance to see our products in action. All of the wireless microphones used by the keynote speakers were sent through a pair of our CP Beams, a special type of antenna we manufacture called a circularly polarized helical.

Here is one of those CP Beams backstage, to the right of the display.

And up close…

We thought it would be cool to write an overview for two audiences of the INBOUND15 conference from the perspective of the production of the live event.

The first audience, our audience, contains live sound professionals, A/V integrators, and other audio nerds who use RF Venue wireless antennas and distribution accessories everyday, but might like a glimpse behind the scenes at a corporate event of this caliber and scale.

The second, fellow users of the Hubspot marketing platform who attended the events and keynotes in Hall B (the main, big hall), who were dazzled by the lights and colors and flying cameras, and have marginal curiosity in how all that stuff works.

As we have discussed at great (great) length before, the days of stuffy, boring corporate events are a thing of the past. Gone are the shaky spotlights, the screeching microphones. The popup tables with scorched coffee and stale bagels? Nowhere to be found.

Because of flush stakeholders (in this case, the awesome and successful company Hubspot), and the power of live events as marketing and community building machines, the resources and technical expertise poured into today’s corporate events equal or exceed those of even the glitziest music concert or special event.

INBOUND15 was no exception.

Most corporate events outsource the pre-show design work—scenic design, booths, kiosks, animation and other audiovisual content—many months before the event begins. This design work is accomplished by many different design companies and freelancers. Most of the design work in Hall B was done by CG Creative.

Then, the design work is handed off to production companies, subcontractors, and freelancers, who actually bring in cameras, speakers, microphones, projectors, and lots of other cool things to put the designs to work.

At INBOUND15 there were two main areas where production companies worked: the Expo Floor and “Club Inbound,” and the General Session room (Hall B) where all Keynotes and Spotlight Breakouts were done. Freeman Audio Visual was responsible for the A/V support and equipment on the Expo Floor, and they brought in Alford Media, a Texas based veteran event technology support company, for the General Session room.

Alford’s team arrived at the Boston Convention Center many days before the start of the event to begin the grueling process of “load-in,” which is when all of the equipment, carefully packaged in large, black, rugged flight cases, is unloaded from semi-trucks and “loaded in” to the venue.

Time is often short, so there is what appears to be (but is not) complete chaos, as supervisors and stage managers shout at people to put this case here, that case there. Technicians unfurl miles and miles of fiber optic, power, and coaxial cable where it needs to go.

Then, technicians begin assembling and testing the equipment. Cameras get powered up, cranes start flying, audio fader programmed, wireless microphone frequencies are coordinated and, these days, much of it is intelligently networked together via fiber optic or CAT-5 (ethernet) cable.

The stage manager will then typically begin blocking talent or stand-in talent to focus the lights and do tech rehearsals to make sure everything works.

We, RF Venue, work most closely with the second group, the people who actually put on the production, because our products are used to improve the reliability of wireless audio equipment.

Within that second group there are roughly three divisions: audio, video, and lighting. Each of the divisions sets up a “village” out of flights cases somewhere backstage. At INBOUND15, Audio Village, Video Village, and Lighting Village were behind the cyc (short for cyclorama, the big piece of fabric that was stretched across the stage). Here’s Audio Village:

and Video Village:

There is also a mixing console and control station located in the middle of the audience, called Front of House (FOH for short). The picture leading this article was taken from FOH by Allison McMahan, Communications Administrator at Alford Media.

I spent a number of hours backstage with the Alford crew during and in-between keynote speakers, and with our customer and friend, Justin McClellan of Communication Handled, who was responsible for all of the wired and wireless intercoms, monitoring wireless audio frequencies, as well as some video tasks that I don’t fully understand.

Here is Justin at work:

Pretty cool, right?

Here you can see the wired communications rack where Justin controls audio communications between crew members and security. In this case a Riedel ARTIST Digital Matrix Intercom was in use.

The entire audio team consisted of Audio Engineer Steve Ellis, Communications/RF Technician Justin McClellan, Audio Monitor Engineer Ryan Sartel, and Audio-Video Recording Tech Andrew McIntire. The FOH engineer was Steven Pollema, from CG Creative.

Here’s Steve, Justin, and Ryan.

I often compare live event productions to baseball. There are long stretches of extreme boredom punctuated by intense moments of fear and scrambling when something goes wrong or the crew needs to move fast to stop something bad from happening. The stretches of non-activity are especially grueling on corporate shows which go on, literally, for days.

But don’t mistake the nonchalant demeanor of the techs during these stretches for lack of skill or attention. What they do is really, really hard, and they are always paying attention, even if it doesn’t look like it.

Above is an image of the “video flow” chart put together on a dime by the video department. It was dark back there, so I didn’t get the best photograph, but this gives you a taste of the skill it takes to pull off an event like INBOUND.

Modern show technology skills are a mashup of old school hand-me-down knowledge, electrical engineering, IT networking, fine craftsmanship, and the herding of cats.

There are a lot of people who can do it, but very few who can do it well. Alford, Freeman, and CG are among the small number of elite companies that do it well, and that’s why they get booked again and again for corporate events of this caliber.

Now, let’s get into the nitty gritty tech details of the audio equipment used at the show.

Alford was using a new technology at the proof-of-concept stage from Riedel called MEDIORNET, that allows any type of signal—audio, video, lighting controls, communications—to be converted into and passed through fiber-optic cable, simplifying signal distribution and lowering the quantity and types of cables required for a show.

There were five Martin line arrays (big, tall speakers) each consisting of nine MLAs and one MLD. Additional Martin speakers included 80 MLA compacts, 12 DD6s, 32 DSX Sub-bass boxes, and 16 MLA Minis.

That’s a lot of speaker.

The FOH console was a Digico SD5. The monitor console was a Digico SD11.

There were 24 channels of Shure UHF-R (which were using our CP Beam antennas), a few in-ear monitors from Sennheiser for the opening musical act, Riedel ARTIST wired comms, and two racks of Radio Active Designs UV-1G for wireless comms.

This was a big show with a lot of very cool, expensive toys to drool over. We had a blast both at INBOUND15 and hanging out backstage with Alford Media, Communication Handled, and the other contractors and freelancers involved.

Until next year!

Special thanks to Allison McMahan and Alford Media Services for some of the photographs used in this article, including the leading FOH image.

HeadsetGate Is Not a Conspiracy, and We Can Prove It

We know a thing or two about how wireless headsets used by the NFL work. We also actually know the technicians who are responsible for operating wireless intercoms (as they are more formally known) and other equipment during NFL games.

And we gave these people a call—though none would speak on the record, we got to the bottom of what actually happened last night.

Based on the evidence we’ve gathered, we are almost positive that the radio broadcast signal from the Patriot’s Official radio station, “The Sports Hub” on 98.5 MHz, heard in Steelers’ headsets, was caused by an unintentional electrical malfunction, rather than deliberate sabotage.

[Thoughts? Scroll down all the way to the bottom for Disqus comments.]

Let’s first debunk why it is almost impossible for the Patriots to have intentionally interfered with the Steelers’ on-field intercom system by signal jamming using airborne radio waves.

• The wireless intercoms in use by the NFL use UHF band frequencies, not broadcast band FM. A common misconception that seems to live mostly on twitter is that the Patriots were jamming the Steelers’ coach intercoms by broadcasting The Sports Hub feed over the airwaves through the stadium in some unusual way. The Sports Hub is broadcast on 98.5 MHz. Coach and player intercoms use frequencies between 470-900 MHz. To jam in the traditional and crudest capacity you need to broadcast on the exact same frequency at a higher power than the jammed frequency, not on a frequency many hundreds of megahertz apart. It is technically possible to patch an audio feed into a UHF transmitter and rebroadcast on UHF frequencies for the purposes of jamming, but…
• The intercoms in use by coaches are encrypted. In order for the Steelers to hear an audio feed of the sports broadcast, somebody would have had to have cracked the encryption as well as use the correct frequency in order to spoof the coaches’ beltpack receivers into demodulating the rogue signal as legitimate audio, as well as being transmitted at a higher power than the opponent’s base-station. Furthermore, that audio would replace the coach communications, not appear as another, second, mixed in signal. The Steelers reported hearing The Sports Hub on top of their internal communications, which strongly points to the Sports Hub Signal entering the signal chain far, far “upstream” as we call it, from the beltpack radio receiver, prior to both radio modulation stage (the transformation of audio signal into airborne radio waves carrying audio intelligence) and encryption stage.
• Intentional RF jamming in the absence of the encryption key would manifest as incoherent noise or dropouts. Can we be any clearer on this? The Steelers did not report noise or dropouts, only a broadcast audio stream.
• Any intentional jamming using radio waves would immediately be detected, located, and stopped by the stadium Game Day Coordinator. At every single NFL game, there is a highly trained member of the Society of Broadcast Engineers called a Game Day Coordinator present who controls and closely monitors every single radio frequency signal in the stadium using software and spectrum analyzers. Monitoring team radio signals for sabotage and surveillance is the reason the Game Day Coordination program was created in the first place. We’ve talked to the head of the GDC Program, Karl Voss, at length about the GDC program before (http://blog.rfvenue.com/karl-voss/). A malicious radio jammer could fire up a jamming device, sure, or broadcast on a specific frequency at high power, or even broadcast on a specific frequency with the right encryption key, but all of this activity would never, ever escape the all seeing eye of the Game Day Coordinator, because the jamming would either be broadband (occur across a wide range of frequencies), be frequency agile (occurring intermittently across a wide range of frequencies), or reveal itself through high power, since successful jamming requires the jammer to overpower the received signal strength of the jammed party.

So, what really happened?

The NFL official statement is this:

”In the first quarter of tonight’s game, the Pittsburgh coaches experienced interference in their headsets caused by a stadium power infrastructure issue, which was exacerbated by the inclement weather… The coaches’ communications equipment, including the headsets, is provided by the NFL for both clubs use on game day. Once the power issue was addressed, the equipment functioned properly with no additional issues.”

The NFL’s statement is correct.

Some possible explanations

A poorly grounded audio or power cable in the wired portion of the intercom system that lives in the coaches’ booth, and connects to the wireless cart on the field, acted as an antenna (anything made from metal can be an antenna), and picked up The Sports Hub radio network. The Sports Hub has its own transmitter inside Gillette stadium, so the received power levels are unusually high. The shielding on a poorly grounded cable received the Sports Hub radio wave signal, and introduced it into the audio circuit in one of many different places—we don’t know exactly where (the coaches’ booth is a best guess). It is much more likely that this is the explanation if the Sports Hub broadcast inside Gillette Stadium is AM instead of FM.

The Steelers reported “intermittent” interference most likely because at the time of the game it was raining, and rain easily compounds grounding problems and makes them less predictable.

As to their report that the interference “strangely” disappeared once they walked over to the Patriots’ sideline to confront, that could very well be a coincidence, or the emotions of the person working for the Steeler’s who wrote the report getting in the way of an objective view of the situation.

The wired intercom system is connected to a number of different systems. There are hundreds, possibly thousands of possible roads of ingress for the interfering signal—mixing consoles, jacks, comm controllers—and a plugged or wired cable carrying the Sports Hub signal could have accidentally been introduced at any of these, though tracking down which one is almost impossible.

Back to the NFL’s statement. When we hear the word “infrastructure,” we think of large buildings, high voltage power lines, and lightning rods.

That is not the type of infrastructure the NFL is referring to. What they mean by “infrastructure” is the complex, low-voltage communications system, and its many power supplies, that is set up at each and every NFL stadium to provide coaches and players with secure, reliable communications, both wireless and wired.

Let’s talk about the building blocks of every NFL stadium’s intercom system, which few seem to understand.

Kevin Seifert at NFL Nation said:

The NFL and its corporate sponsors, primarily Bose and Microsoft, provide the hardware and other equipment necessary to maintain communications for both systems.

Wrong.

Bose and Microsoft have almost nothing to do with on and off-field communications. Bose simply supplies the noise-canceling headsets (the microphone and headphone) that plug into the radio transceivers, and Microsoft, as far as we can tell, pays to supply the branded carts that the intercom systems ride inside, and nothing else.

The real wireless intercom system is provided by a small company called Telex, now owned by RTS.

The NFL uses heavily modified, custom built Telex BTR-1 systems for the cart to coach and player wireless links.

These carts sit on the sidelines of both home and away teams and are filled with “intercom base-stations”: the devices that receive and retransmit wireless signals received from radio intercom transceivers worn by coaches and built into player helmets.

These Telex systems are so special they have a special name known only to those who use them: “Lombardi Systems,” and if you have ever watched professional football, you’ve seen them with your own eyes.

We’re not stupid enough to put up a screen grab of NFL network footage showing one of these carts, but luckily, we have in our archives a photograph of a cart taken by Jeff Watson that contained a newer type of intercom system, the RAD UV-1G, that was used for portions of the Super Bowl and Pro Bowl.

These carts are prime real estate and appear in multiple camera shots, so they are covered in the corporate marks of whatever company paid the most to place them there, but the guts (surprise!) aren’t filled with Microsoft Surface tablets. This is not a Lombardi System, but under ordinary circumstances, and at last night’s game at Gillette, it was.

Most of the mainstream media has focused on the wireless link between Lombardi cart and coaches + players. But Lombardi carts are connected (trunked) to wired intercoms running into the stadium and up to the coaches’ booth. This allows coaches to communicate with coaches and players down on the field, and vice versa. They can push a button and talk to one person, or everyone.

The wireless infrastructure is essentially bulletproof to the type of interference the Steelers heard yesterday.

The wired infrastructure? Not so much.

Both coaches’ booths are required to have at least six intercom jacks (where a headset can be plugged into). The booths are also outfitted with wired intercom main-stations (that route signal here and there and let a controller speak to one or many people on command). The jacks and main-stations are connected to dedicated power supplies, and the main-station is also connected to the NFL control booth.

More formally, these types of wired intercoms are known as “two-wire party lines.” [Edit, I’m unclear as to whether the NFL uses party-line only, or party-line/matrix hybrids, probably the latter.]

We have been told by anonymous sources that the electrical grounding procedures on all two-wire comms is dangerously out of date due to something called the Pin 1 Problem.

An audio or high quality power cable is usually shielded—an inner conductor carrying signal is covered in aluminum or copper that protects the conductor and signal from RFI (radio frequency interference).

The shield is always conducting ambient radio waves and turning them into electrical waveforms, but once the cable reaches the device, the RF energy that is in the shield is directed away from the audio circuit: that it is “electrically grounded.”

Pin 1 (the audio cable shielding) should be directly connected to the chassis so any RF signal running through the shielding is diffused through the chassis rather than injected into the audio circuit.

Unfortunately, on most of the wired intercom jacks and main-stations used by the NFL, modern grounding procedures (“infrastructure”) are not used, and occasionally cable shielding can conduct RF and make it into the audio stream.

The RF signal (in this case, The Sports Hub Signal) may have infiltrated the signal chain before the encryption or RF modulation stages in the transceivers on the Lombardi (which is quite secure from jamming).

It should be said the wired intercom system is connected to a number of different systems. There are hundreds, possibly thousands of possible roads of ingress for the interfering signal—mixing consoles, jacks, comm controllers—and a plugged or wired cable carrying the Sports Hub signal could have accidentally been introduced at any of these by a mistake of installation or human operator error—we’re not just talking about accidental ingress and demodulation of an RF signal here anymore, an actual patch into a board or crossed wires are also somewhat possible.

Grounding on two-wire intercoms usually works OK, most of the time, but shabby installation, and rain, can make things worse.

To complicate matters, because the wired intercoms are party-lines, and they are connected to the Lombardis, there is no telling where, exactly, the interference is bleeding through, unless you tear down the system, cable by cable at every jack, main-station, base-station, and power supply.

What about that “almost”?

Earlier we said we were “almost” sure that the interference was accidental.

There are still a lot of people on twitter who think this is a conspiracy, so we’ll concede that it is possible that the Patriots were able to intentionally tamper with the Steelers’ comm system and produce the type of interference observed. The home team is responsible for installing and maintaining the wired comms infrastructure, after all.

We aren’t going to explain how this would be done, because we don’t want to give out instructions on how to sabotage a comm system to all the naughty coaches out there reading this post. NFL style intercoms are used at high school and college stadiums across the country, too.

But we will say that, although theoretically possible, such tampering would require a lot of people, leave behind a lot obvious physical clues, and probably include an inside man or “mole” planted on the comm controllers or consoles of the Steelers.

In other words, even if you were able to get away with it for a game or two, it would never hold up to the scrutiny of a formal NFL review, which the Steelers, Patriots, and NFL all agree is not being pursued.

Less than 24 hours after the beginning of “HeadsetGate,” HeadsetGate is over.

For the time being, football fans will have to look elsewhere for evidence of the foul play that (maybe) explains the Patriots’ success.

Please address editorial and other enquiries to info@rfvenue.com

Leading image courtesy “Art01582”

The Tradeoff: Higher Gain Antenna, Narrower Beam-Width

One of the tradeoffs with high gain antennas is that the higher the gain, the narrower the beam of coverage. This is usually a good thing—but not always.

For example, the CP Beam has a gain of 9 dBd, creating a beam-width of 43°. This narrow beam width is exactly what you want if the antenna is positioned a reasonable distance away from the microphones or belt packs on stage (say, at FOH, or in the wings in monitor world). It provides increased range and reception by focusing more of the RF energy into that area. It also attenuates signals that fall outside of the beam, allowing you to strategically place helical and other high gain antennas and point them away from interference sources, like LED walls, and towards the signal of interest.

However, if a high gain antenna is placed in close proximity to talent using or wearing a wireless microphone or beltpack, the talent could move outside of the coverage area of the antenna, causing a dropout.

The CP Beam, actually, is not that directional, if you include the entire field of antenna design, and the polar plot above shows that it will still receive energy 360° around the element, but in differing amounts. In front of the element the CP Beam will increase the apparent signal strength of a signal, to the sides, it will moderately attenuate the signal.

There are antenna designs that have much sharper—nearly complete—rejection of off-axis signals, but we very rarely see these types of antennas in pro audio. The following polar plots show the extreme rejection of various types of “end-fire array” antennas.

End-fire array antenna patterns, the image of which mysteriously appears on WikiMedia with no author to attribute. The left most point of each of these patterns (where you see the small bouquet of smaller lobes, would be located at the center of the polar plots above. Now THAT’s rejection.*

The point is that antenna coverage patterns and (if specified) beam widths should be considered when setting up a system to ensure the antenna allows sufficient coverage for the movement of talent in a given space, especially if there is poor signal-to-noise ratio—which would, were the talent to wander off of the main axis of an antenna like the CP Beam, increase the chances of dropout.

An Omnidirectional Antenna with High Antenna Gain is Impossible to Construct

Lot’s of people want to have their cake, and eat it too. Which is why we’re writing this post, to clear up the misunderstanding; Many want an antenna that will focus RF energy equally in all directions—i.e. a high gain omnidirectional antenna that will provide higher gain and longer range in all directions.

This simply isn’t possible.

That higher gain antennas have narrower beam-widths is a function of physical laws. Engineers have been able to figure out some impressive and acrobatic designs for antennas that shape radio waves in incredible ways, but so far they have been unable to dupe the simple reality that if an antenna focuses RF more strongly in one direction, it will be less strong in another.

There are a number of omni-directional antennas on the market that claim to improve reception while providing an omni-directional coverage pattern. That is, they promise a single antenna will increase signal strength, and therefore increase range equally in all directions. These antennas are usually dipole antennas mounted in a housing that allows remote placement, but are otherwise no different than stock dipoles that ship with receivers.

Sometimes, these antennas are combined with integrated amplifiers and marketed as “high gain omni-directional antennas.” Such language is misleading because as we’ve learned, omni-directional antennas must, if you are using the correct mathematical definition of antenna gain, have low directional gain.

You can have an exceptionally well constructed and electrically efficient antenna with an omnidirectional pattern that may very well increase range, but it is not increasing range through directional gain, but rather in its ability to collect a greater percentage of available RF energy from the environment. For example, a half-wave dipole will probably collect more energy than a quarter wave whip (monopole) antenna that might be used as a lower cost stock antenna on some receivers.

You can also have a high gain antenna that radiates more strongly across one horizontal or vertical plane in degrees of one axis (elevation angles), but not others.

An example of a radiation pattern of a horizontally omnidirectional antenna
but still not truly (spherically) omnidirectional.

These are sometimes called “high gain omni-directional antennas,” and here the use of that term is more appropriate, but not truly omni-directional; the coverage area is like a horizontal pancake, or wheel, rather than an omni-directional sphere. (which, by the way, does not exist. Even the most evenly radiating dipole antenna still has two nulls located at the top and bottom of the element.)

Still, a lot of people want the best of both worlds. They want an antenna that will increase range while allowing their talent to roam wherever they please. That is in part why the Diversity Fin antenna, although not a single antenna, was designed and is now popular. It is effectively able to provide both directional and omni-directional coverage by combining two antenna elements on the same board, and the diversity receiver votes between which antenna has the best signal.

Want better wireless? Download our eBook on three essential concepts for correctly deploying and maintaining interference-free wireless audio systems.

Leading image courtesy Seth Sawyers.