The problems with audio networks
13/08/13 09:52 Filed in: RHC
We were doing some training on audio networks with a client a few days ago and the subject of security and encryption came up. I was reminded of something I wrote in 2005, which starts on the subject of security but becomes a much wider discussion. I have decided to let it loose again as the points made then are just as relevant today.
I've changed a couple of things for comedy value or just to make it understandable to a 2013 audience, but its almost exactly as written eight years ago…
Hail the Uzbek Ambassador
The Judge entered the courtroom, ready to give his verdict. The public enquiry had gone on for two years and everyone knew that his verdict would change the face of the audio industry forever. They knew that from now on they would be scrutinised and regulated, just like the lighting industry after the blindness riots of 2017. No longer would sound engineering seem like a cool profession, we would be subjected to ridicule and suspicion just like lawyers and lighting designers. Riots were one thing, but how could a war have started because of a sound system?
It had all started four years ago. The installation of the new audio network for the European Union had been a long time coming. The increase in the number of countries joining, the ‘Farage wing’ adding the extra committee rooms and the introduction of voice recognition systems to do all simultaneous translation had made this a mammoth task.
The hackers had tapped into the network right at the very start of the project. They had put an extra DSP in a plant room and labelled it ‘Spare LG1’. The Commissioning Engineers never checked with the Consultant how many devices should be on the network so they implemented the unit into all system configurations. These days, every loudspeaker, microphone, amplifier and playback device was networked so it was impossible to keep track of how many items you had anyway. Thus the DSP was given full access to all audio channels in the system. They could listen in to private meetings between world leaders without suspicion.
As well as getting all the background information on what the Uzbek ambassador was going to say, their final trick was to alter the audio routing, changing the real translation feed with a fake one. Insults were exchanged, diplomacy broke down and the rest, as they say, is history.
The problems with audio networks
This probably won’t happen, but as we move slowly towards complete networked audio systems there are a number of problems and pitfalls that have yet to be addressed, security is just one of them.
I have long been an exponent for computer control of audio systems. I believe that this technology is finally starting to mature. We can now do practically anything we want with an audio signal, though some of our methods still lack elegance; we can send tens of channels of audio along one CAT-5 cable or pair of fibres, the cost of DSP is reducing, network delays are plummeting and more items of audio equipment are sprouting audio network connections, allowing true integration.
Many of these advances are because of developments in the computer industry and the audio industries’ adoption of their protocols. It isn’t just us adopting their hardware, it is also the invention of better software development tools.
Whilst the technology improvements are ticking along nicely, many of the problems we will have in the future have nothing to do with that.
Its fine leaving me
In the old days of XLR connectors you knew where you were. If a signal buzzed it was fairly easy to find where the problem was. In a digital network you might not even be sure what route the signal is taking to get from one location to another.
The issue here is one of support. There are standards adopted by a number of manufacturers that are pretty reliable, but who is going to support you if one item of equipment won’t pass audio to another? In the installation environment, the contractor might be ping-ponged between the two manufacturers each blaming the other for the failure. For live events, the rental company will have to deal with the problems at any time, day or night, they can’t wait until the factory opens in the morning.
Networked audio systems are potentially very complex and the problems can be unusual. What would you do if you got a digital click out of your loudspeakers every few minutes? Normal sound engineers’ logic may not be applicable when trying to find out what is going wrong. The merging of IT and audio introduces a new set of issues that few people understand completely. For example, IT specialists have a completely different idea as to what is acceptable when it comes to latency. They also think that MP3 is good quality.
The support required for networked audio will sometimes result in a conflict of cultures. IT people simply don’t have the same set of priorities as sound engineers. The notion of ‘the show must go on’ is not something they have grown up with. They come from an industry that stops work at five and doesn’t work weekends. Yet these are people who we may rely on for support for some of our problems. This is especially true in large audio installations where the audio system runs on a subnet of a larger IT network.
However, any technical problems, no matter how unlikely, are not the main stumbling block. The main problem is contractual. The purchase of sound equipment often has to go through a committee, accountant or lawyer who will ask the question ‘what if it goes wrong?’ It is not acceptable just to tell them not to worry, you know that you can no longer support this yourself; you have to demonstrate who will take responsibility, if it were to go wrong. The supplier who will give the best support for the whole network will get the sale.
Lets try one of these instead
We’ve all done it. An item of equipment has needed changing over. Maybe swapping over a faulty radio mic, or swapping a two channel amplifier for a four channel one as a new area is added to a system.
In the analogue world this is so easy. It doesn’t matter if you mix your equipment between different manufacturers. You just plug in whatever is available.
Digital networks are different. As soon as you start having to deal with integrating a system as one, the basis on which a sound system is designed and purchased changes dramatically. You have a different set of priorities.
One the design side, you may decide to use self-powered loudspeakers, perhaps with built-in DSP. This currently forces you into choosing just a few products but you get to eliminate most of your racks. You may decide to keep all the system intelligence in the racks to give yourself more choice over what loudspeaker is used. If you are transporting audio digitally you need to decide on which protocol, which will also determine what products you can specify.
For controlling all this you will want to keep it simple. This will mean keeping the number of protocols and thus the number of manufacturers specified to a minimum. This will make setting the system up much easier.
You won’t want to or even be able to have a mixture of different radio mics or amplifiers on a project. Whilst uniformity is often a good thing, you might lose a bit of flexibility in what you can specify.
Another problem with changing your equipment at a later date is in how the software is set up. I discuss this in more detail later. However one of the problems is that all equipment parameters are currently stored in a device-specific format. This means that if you change an item on the network you have to re-program that element from scratch. In the future you should be able to change the equipment to a similar item and for your show to remain the same. Hardware abstraction exists in the computer industry, just look at how your computer can print to any printer and your document looks the same, but this has not yet been addressed for audio.
The fewer options available when swapping items over in an emergency will probably mean its best to build more redundancy into your system from the outset.
I’ll make you a deal
Selling all this new equipment is going to become a more serious business as the deals with a particular supplier are likely to get bigger as you attempt keep the network as simple as possible, consolidating your purchases and thus putting more of your eggs in one basket. They are no longer selling items of equipment, but complete solutions. Salesmen will need more system-wide knowledge in order to do this successfully.
The sale of this equipment also ties into how you will be supported. A distributor will offer a portfolio of products from a number of manufacturers. They may offer the technical support you (or your accountant) require. You may prefer other networkable loudspeakers but they are offering to take responsibility, to make theirs work with the DSP they sell. This is more important than a particular loudspeaker choice.
Also once a system is sold, the relationship with the equipment supplier will continue for longer than before. When the system is upgraded you will more likely be tied to using the same make of processing equipment, or the same amplifiers to keep the same control system and network protocol.
Explaining these product choices to clients will need to be handled with care. They won’t necessarily understand why you need to sell them an expensive networked amplifier just to add two speakers in a small room and then you need to charge them for a bit of reprogramming too! On other occasions you will save them thousands and they won’t notice or appreciate that you have done this.
The need for technical consistency in a system that may change over time means you will have to live with your purchasing decisions for a lot longer than ever before.
I remember when minidisc came along and I thought it was fantastic as I didn’t have to edit tape anymore. The problem was that clients soon found out that you could edit things more quickly and easily on site. This meant that they made decisions late, sometimes minutes before the start of the show, as they knew you had the facilities to make changes at the last minute.
This sloppiness, in making decisions late, has now transferred to sound engineers and designers. This happens at both the design and implementation stage of an audio network.
Often when designing a network you don’t bother looking at the detail, as you know that you have a colossal amount of DSP at your disposal and you will simply decide what to do with it later. You can design a system almost without thinking about how it will sound and merely concerning yourself with network topology and cabling issues. You can get seriously unstuck taking this laissez-faire approach. We are all guilty of this to some extent as the sound itself almost seems incidental.
The Sound Designer is passing many audio design problems onto the Commissioning Engineer, giving him more work to do and moving the risk of discovering that it won’t sound right to a much later stage in the project. So why is the Sound Designer getting the big fat design fee and the posh job title?
When the system has been installed, the sheer volume of processing can overwhelm the engineer who is trying to make it work. It is too easy to add countless processing elements, adding eq in one place, taking it away in another etc. We wouldn’t do this with a conventional sound system but when you are just placing processing objects on a screen it is all too easy to overcomplicate the audio path. What happened to ‘less is more’?
The inability to see clearly what processing you are applying to each audio signal is partly as a result of the poor quality of software that is currently available, and this brings us on to control…
Who is in control?
This is a major issue. Harman have recently introduced HiQNet, a standard across all their brands to let all their networked equipment communicate with each other. It is too early to know whether they will licence this out to become an industry standard or whether a competitor will appear. We might find that we will steal a future technology from the IT industry and that will become the basis of controlling an audio system.
Whatever happens, whilst the actual transport of audio works well, unified control of sound systems is still in its infancy. The reason for this is simple; at present there is no ‘killer app’ for controlling audio networks. The manufacturers are still working on the basics of making the system work. We haven’t yet sorted out an elegant way of controlling all this.
There is even a debate, for those who have got that far, over what metaphor control software should use. I have long talked about a ‘task based’ approach to control but this is still very difficult to describe without going into arcane detail and there are a number of barriers that prevent this approach from happening at the moment. I might also be wrong.
How much information do you show to the user?
This is particularly problematic with sound as there are so many possible parameters that can be displayed just for a single channel of audio, let alone a complete system. The human eye can only take in so much information at once, especially in the heat of a show.
Do you sacrifice speed of setup for flexibility for the user?
Software lets you build control screens where the engineer determines what controls are available to the user, but all this takes time to do. Many of us have spent far too many hours positioning faders and buttons on a computer screen, trying to make it all fit.
It is difficult to write software that will assist the engineer with this task; the uses of a sound system are so diverse that you can’t easily dictate what controls should be displayed. What is appropriate for a nightclub will not work for a conference centre.
It also won’t just be a clever program that will transform our lives. The complexity and unique physical problems associated with audio mean that any software needs to be tightly integrated with the hardware that it is controlling. This is a mammoth task as you are not just dealing with one item of equipment like most computers do (mainly just controlling the computer itself), you are controlling a whole network of equipment as one unified system. Not only that, but we want to do this in real time. Few industries have software challenges that are so demanding.
For the system to be truly unified, the DSP will need to dynamically change what it does without any audible effect to the listener. This is akin to putting a software update into your computer and not having to restart the machine. This will happen, but we are still a way off and this will restrict what we can do with software until then.
Perhaps there will be no single application and you will have software dedicated for each type of use? However this presupposes that there is a common communication protocol for the industry and that there are programmers being paid to develop software for particular markets in what is already small industry. Do we want different software packages fragmenting it even further? It is also an industry that doesn’t have much experience paying for software so who will pay for the development of the ‘killer app’ if there isn’t a good economic model to support its development?
And who is to say that we should use a computer to set up and control a sound system anyway? This is not a natural approach for traditional sound engineer. Perhaps in some circumstances such as live events, the mixing console should be the controller and it should merely display and control all other elements of the sound system. That too involves a lot of code to be written and in that case the console manufacturer will have to stump up the cost.
Lets call security
Sound systems have been hacked into for years, the Linda McCartney split of her singing backing vocals being a famous early example. Manufacturers have taken some steps into making audio networks secure with password protection or the use of dongles.
The protection of intellectual property is becoming increasingly important. Clients will expect their audio to be encrypted and they want to have absolute control over who gets to hear what and how much they will pay for the privilege. They will want evidence that your system will protect them and will be secure. However that doesn’t solve our problem with the Uzbek ambassador.
Most security problems are human. A friend of mine went into the headquarters of a water utility and said that he had come to fix a computer. He was directed to the IT department and he asked which computer was broken. No one knew, so they pointed him towards one they thought didn’t work. After a few minutes, he asked for, and was given the network administrator password. After downloading several thousand customer account details, he went upstairs to give his security report to the Chief Executive.
Boring though it is, the audio industry will have to adopt procedures to prevent similar breaches. Basic information such as what equipment should be on the network needs to be documented and kept track of by everyone who is involved with the implementation of the system.
How lucky do you feel?
For someone who actively promotes the use of network technology these words could be construed as giving out the wrong message. As you read this, shares in companies making analogue multicores are probably spiralling upwards.
However, the fact is, networked audio is the future and for most of us analogue audio will disappear. The price, flexibility and convenience of these systems far out-weighs any of the concerns being raised by others or me. However these concerns should be raised, discussed and solved to make our transition to the networked world as smooth as possible.