RIP ISDN 1992-2013

Unless you've been living under a rock, you've heard that Verizon sent out a notice to many eastern seaboard customers that new ISDN installs are no-go. This set off quite a lot of activity on broadcast mailing lists and blogs, ranging from "OMG--we're doomed" to "Told ya so".

I have to admit to a bit of deja' vu. I wasn't at Comrex at the time, but back in the mid 80's AT&T made a similar announcement, that long-line dedicated loops (that crossed LATAs) were going away. Broadcasting survived the change, and Comrex legend holds that it made a very nice sales bump in the then newly introduced two-line frequency extender.

While I certainly won't complain if our IP codecs sales see an impact on this, my real concern is in making sure customers who find themselves in a pickle get the best possible information about IP codec alternatives. There's a lot of opinion on the Internet <shocker, I know> and it can't always be trusted. The truth is, given the nature of ISDN these days, an IP codec on a suitable wired Internet connection really does rival ISDN in terms of reliability.

We may have been a couple of years early in obsoleting our ISDN products, but we saw the writing on the wall some time back.

Now let's be clear-- those touting "5 9's" reliability in regard to ISDN are kidding themselves. Talk to our support staff if you don't buy it. ISDN has become a legacy technology with legacy support for problems that arise (and they do arise).

So if you're worried about the demise of ISDN, do yourself a favor and find us at NAB or have a call with our knowledgeable sales or support staff. Knowledge is power and we all love talking about this stuff, even if you choose other products in the end.

In Which I Rage Against Digital Rights Management

To promote the Progress of Science and useful Arts, by securing for limited Times to Authors and Inventors the exclusive Right to their respective Writings and Discoveries.

The copyright clause of the US constitution is very important in setting up government's role in granting limited protection to copyright-able works. But it's far from perfect, and has been subject to significant corruption. I'm going to leave aside the foolishness instilled by the Supreme Court in allowing virtually unlimited retroactive extension of copyright, and focus on another really annoying facet of modern copyright law that has an effect on Comrex product development.

This is the ability of rights holders, as a collective, to dictate copyright protection mechanisms in commercial electronics. I think there's no other more outrageous example as the inclusion of HDCP protection in the actual interconnects (like HDMI and DVI) between media devices. 

Now if I wanted to be told by a device manufacturer what I could and could not do with the product I purchase, I'd buy Apple products (cue rim shot). But the copyright industry has managed to require the use of device-hobbling technology on the actual wires we use to interconnect devices.

It achieved this using a two-pronged approach. First, the Consumer Electronics Association was forced to mandate inclusion of copyright protection mechanisms in digital media outputs (and therefore require the corresponding protection on monitors and other inputs). This became a requirement of licensing the technology required for playing out DVD and BluRay discs on digital ports.

Next, the industry lobbied the U.S. congress to pass the Digital Millennium Copyright Act in 1996, which made it illegal to market any device that circumvents copyright protection, even if for non-infringing use. There are some narrow exceptions carved out, but none that would allow disabling of HDCP on an HDMI output. So no such devices exist.

HDCP is a fairly complex security layer which rides on the HDMI interface. The source and sink devices exchange a set of encrypted keys, and then decide whether the sink device is worthy of receiving the copyrighted works existing in the source. If the works are copyrighted, and your monitor doesn't support HDCP, no media for you.

The mechanism is licensed by a controlling body called Digital Content Protection LLC. In order for a manufacturer to support HDCP he must pay the high licensing cost and assume the huge liability of maintaining software keys that open these ports. It's certainly beyond the range of any company not involved in the manufacture of hundreds of thousands of consumer devices to consider.

Let me be blunt. HDCP is stupid. Here's just a few of the many reasons:
1) It does nothing to stop piracy. As anyone even slightly knowledgeable about digital media knows, the way to extract data is not through an uncompressed, real-time interface.  Real pirates have already cracked digital media protection and use it at will.
2) It annoys the consumer. Besides incompatibility with sink devices manufactured before the mandate, HDCP is the cause of a raft of consumer headaches. User support forums for consumer devices are chock full of reports of users getting "blue screens" even when properly connecting HDMI devices.
3) Manufacturers of media playout devices are stupid, lazy, or both. Instead of properly detecting whether media has copyright protection enabled, they'll take the short cut of simply enabling HDCP on all outputs, all the time. Many consumer video cameras implement permanent HDCP on live video outputs. 

That's right, the copyright industry restricts what you can do with video you are taping live! This type of "protection" is just a huge power grab by copyright holders that has become completely embedded in our industry and culture.

Why do I care? For the past three years I've been working diligently on our LiveShot video codec product. One of the important features of the encoder is the ability to interwork with cameras and other sources that output video and audio via HDMI. But if you go to Amazon and buy yourself an HD Handi-cam, there is at best a fifty percent chance it will work with the HDMI port on LiveShot. Or if you try to get live HDMI video off your phone, it will almost always fail. And it's because of HDCP.

Now when this happens to a LiveShot customer and I get the call, I can go into details on all the reasons this is so, but the end result is an unhappy customer, and me with no options to make him happy. And I really hate that. Hence the rage.

The HDCP ship has sailed, and no amount of rage-blogging will change things. I think the responsibility of the tech professional now lies in keeping an eye on the copyright industry and lobby to prevent further abuse. Since the DMCA was enacted in '96, it's become much easier for groups opposed to copyright interests to organize and publicize abuses through social media and other on-line resources. I also think it's a good idea to be aware of what your political rep's opinions on excessive copyright outreach is. It's hardly a topic that will make or break a political campaign, but it's a good topic (along with patent reform) to ask questions about to find out whether your congressman or senator is working for you, or "big copyright".

NAB is coming

The Comrex technical department is in the midst of the annual NAB shipping date crash! This year the pressure is a little less, since all our big product intros are now shipping and have been to NAB before. So we're actually bringing production-ready hardware to the show, and it's relatively bug-free.

Last year we had two booths to staff--here's our guy Henrik at the Verizon Booth in the south hall

And here's our own Kelly modeling what's been coined the "Kelly Cam". This is an adapter that converts handheld "pro-sumer" style camera into a shoulder-mount arrangement-- much friendlier to be used with our LiveShot video codec.

This will be my 26th NAB, always exhibiting for Comrex. It's something I look forward to every year, as nothing makes me happier than chatting about our products. I'm really proud of them.

Now I'm aware we are there are the seller, and the customer should be given ultimate deference in most matters, but I'll take this opportunity to list a couple of "don'ts" at NAB I hope will sink in to some of the more "challenging" visitors. If not, it still makes me feel better having the chance to whine about them.

1) Phone etiquette- Sure I'm here to borrow some of your time, but I'm not always wearing my sales hat (I run the tech side of things here at Comrex on most days) so how about a little respect and not answering your phone while I'm mid-way through a deep tech answer? I'd like to wait for you to end your call, but the three people patiently waiting to talk to me won't feel quite as generous with their time. Our company spends a small fortune to come to NAB, and I really enjoy tech discussions even when they don't lead to sales. But it's pretty dispiriting to work your brain that hard to try to help someone, only to realize your thoughts are the least important thing in their world. So let it go to VM while we're talking OK?

2) Product Support-  As mentioned, NAB is really expensive for us to do, and the peak times (Monday and Tuesday) are extremely limited and busy. I am truly interested in helping diagnose that thing that happened last month with your ten year old product, but could you do me the favor of coming by on Wednesday or Thursday when things are slow so I can give you my full attention? Keep in mind, we also have a crew of friendly and sharp techies to handle this stuff back home.

3) Dealer Inquiries- Don't be offended if I ask you a few details about your company before launching into a summary of our entire product line. I do it because there are a host of guys on the floor just looking to represent our products in their territory. This is a huge time waster, as most of them don't understand what we do, and are just "cold calling" to drum up business. If you're a potential dealer, wait until slow times at NAB. If you interrupt me with a customer, I may have to be less than polite. Also true of those trying to sell me their services at my NAB booth. Actually, I don't want to talk to those guys anytime :)

4) Breaking the Ice- Please don't be shy about approaching us at our stand. Lots of folks like to come and read the signs and get a feel for the products so they have better questions to ask. That's cool-- and my personal tactic is to hang back and let you do that until I sense you have need of help. It can be a tough call, and I don't always get it right. When I sense you're ready I'll then ask if you have any questions. At that point the attendee will either turn on his heel, or say what's on his mind. So if I'm unoccupied (or chatting with Comrex folks) but not giving you attention, it's because I'm giving you space. Feel free to approach. You are the most important thing at NAB to us.

5) The no-brainers--Please don't raid our stock of swag. Please don't manhandle the live demos (or at least ask first). The seating in our booth is in short supply--it's perfectly OK to rest your feet for a few minutes but please don't make it your personal workspace. The ladies in our booth are employees and are really sharp and shouldn't be patronized or called "honey".

'nuff said. The vast majority of our visitors are there for all the right reasons and we have a great experience meeting them, discussing the issues, and showing off our stuff. Hope you can come by and at least say hello!

The Reluctant Troubleshooter

This post strays from Comrex specific stuff into general engineering philosophy for a bit.  My son Andrew, who enters high school next year, was offered a course called Introduction to Engineering as an elective (kudos to whoever instituted this option btw). It got me thinking about how my engineering skills developed over time, and what my education was lacking and what I had to learn "the hard way".

One of those major skills was troubleshooting philosophy and discipline. When trying to tackle a tough engineering problem, there are several pitfalls I caught myself falling into in the early days. Luckily I had a good mentor, and between consultation and perseverance was able to "mold" my approach to tough problems in a way that serves me well. But a difficult problem (particularly one with a tight timeline) can be a stressful thing, and there's a certain degree of self-discipline involved in stepping back and evaluating whether you've followed your own rules.

All technical issues, as random as they seem, follow the rules of physics and logic on some level. Even with the best engineers, sometimes one simply doesn't have the time, resources or experience to dive deeply enough to solve the problem using the proper engineering philosophy. We all have our limits. On some level, the quality of an engineer can be defined by the percentage of problems solved using sane logical reason vs. the amount of "hack it til it works".

It's not always easy staying on the path. I've had to tackle a couple of tough ones lately that manifest themselves in what I call the "paradox of doom" (POD). Typically, this involves trying to fix an intermittent failure of hardware. It's universally easier to fix something binary in its failure mode. For the most part, when you fix a software bug it's fixed and done.

But the POD shows itself in hardware much too often. A prime example-- a system presents itself with a failure mode (to put it in Comrex terms, an example would be a 4G modem not being recognized by an ACCESS codec). So you use your logic to narrow things down to the USB interface hardware, and make a list of the usual suspects like EMI, power supply noise, routing, stray capacitance, etc. You figure out a way to simulate an improvement in each of these factors on the bench.

Now, Murphy's law dictates that the last item on your list will resolve the problem, and invariably that's what happens (I'm not by nature a superstitious type, but I must admit to a habit of tackling debug task lists backwards in an effort to foil ol' Murphy). But that's not the source of the POD.

Lo and Behold, the POD rears it's head when you reverse the change. I swear, this is true---that damn circuit will continue to work just fine n' dandy 95% of the time once your fix is removed.

The Internet has come up with a meme to express exactly this emotion

Even the most grounded and logical person is not immune from the despair that comes with spending several hours or days on a toughie, having the thrill of finding something that appears to fix things, and having that hope come crashing to earth under the POD.  Depending on the depth of brainpower and time involved, mitigation techniques can consist of a long walk outdoors, a long hot shower, and eventually working up to a bottle of gin.

I always get annoyed by punditry like this that defines the problem without any solution. So here's my short list of factors that have made a difference to me in the past. Maybe this is useful wisdom to pass down to the next generation, or maybe it's all hot air. You decide.

1) Step away--It might not seem so when you're in the grip of the POD, but to paraphrase that otherwise annoying song, things will look better in the morning. Get busy with something else (like updating your blog) then go home and get some rest.

2) Consult with others-- A half hour with respected colleagues and a white board talking things through will really change your perspective quickly. Even if they can't offer much, simply the act of talking it out and writing it down can reveal hidden truths.

3) Get back to basics--Go back to the list of possible "root cause" of the problem, and evaluate whether you have the tools to properly run them down. Sometimes, an equipment purchase or rental not previously considered will start to look pretty good.

4) Swallow your pride-- Engineering is such a vast science it's not reasonable for everyone to know it all. And it's often not possible to learn (or re-learn) all the factors of something like EMC or transmission lines within the scope of the project. So find a list of competent consultants to look up when you've reached the limit of your experience. Industry contacts, colleagues and social networks like LinkedIn can help.

4G modem situation becomes clearer

One of the biggest challenges Comrex has met in the years since we've been shipping our ACCESS audio codecs is support for third-party 3G, WiMAX and 4G modems. In the early days, these were delivered in a variety of form factors: Cardbus, Express Card, and USB. Mercifully, the industry settled into a single interface (USB) but there are were still a variety of USB "data cards" from each carrier, and there are hundreds of carriers worldwide.

In the US and Canada, the situation remains sticky, primarily because each carrier has a unique combination of operating frequencies and modulation standard (e.g. UMTS, CDMA, LTE). So one data card is not portable to another carrier (at least in a way that it will still deliver peak performance).

Luckily the field is finally thinning, and we can suggest a single USB data card for each US and Canadian network, and two relatively universal LTE capable cards that should work in the rest of the world. These modems are tested to work with the latest ACCESS and LiveShot firmware.

From Left- Verizon/Pantech UML295, Sprint/Franklin USB Tri-mode, AT&T 313U, T-Mobile Rocket 3.0

United States
Verizon Wireless
By far the only modem to buy today is the Pantech UML295. See our results in NYC to find out why. This modem seems to resolve all the ongoing issues we've seen with previous Novatel and Pantech offerings.
AT&T
The "Momentum" (also known as the Sierra 313U) is the only choice these days and does a fine job. One note--if upgrading to this device from a 3G modem (or AT&T's confusing non-LTE 4G) you'll need a new LTE-capable SIM card to experience LTE service.
Sprint
The name of the device is the "Tri-Mode USB". This is because Sprint's data offerings are spread across a range of frequencies and technologies. But the Tri Mode will handle them all, including the new LTE deployment.
T-Mobile
Despite what the commercials imply, T-mobile offers no LTE service. Their HSPA+ service (branded 4G and still quite speedy where available, but not not nearly as deployed as competitors) works fine with the "Rocket 3.0" (also known ad the ZTE MF683)

Canada
Rogers
All current Rogers offerings (including LTE, where available) are supported by the Sierra 313U (the same device promoted by AT&T in the states).
Telus
The Hauwei E397 is the modem of choice for Telus customers, covering all services including LTE
Bell
The U679 is promoted by Bell, but this is currently untested by Comrex. The twin device available for Verizon in the US (Novatel 551L) had some trouble synchronizing with ACCESS's USB ports, so we can't yet recommend this modem.

From Left- Hauwei E392, Option Beemo

Other Countries
Hauwei E392- This modem seems to be available from many international carriers that support LTE. It's a good choice, but if you buy this modem from places other than your carrier, be sure it supports the right frequencies, as there are several varieties.
Option Beemo- This is becoming our preferred international modem, supporting the widest range of commonly used frequencies and modulation styles.

VZW LTE looking good again in NYC

We at Comrex have been excited about LTE since we first heard about it in the early aughts. When it first hit, we were blown away at codec performance. One of my first real experiences was travelling to midtown Manhattan, a place where 3G had become congested to the point of unusability. At that time, performance was exceptional. But over the last two years we've heard frustration from ACCESS users that the Verizon LTE network in NYC (and a couple of other major markets) was increasingly quirky.
In conjunction with our friends at WCBS and Verizon we ran some extensive tests last year, and the results pointed to our preferred USB modem, the Pantech UML290. While this modem seems quite well behaved in most markets, it seems in a highly congested environment like NYC it has a tendency to get into a state of total data stoppage. This seems to occur at almost predicable cadence, with good data flowing for 1-3 minutes, followed by a data stoppage of 5-15 seconds.
Our working theory is that some kind of handoff is happening within these high-traffic networks that the Pantech takes some time to recover from. The only alternative modem, the Novatel 551L, has its own set of behavior problems that prevents us from recommending it. Up until now, the only real solution was use the provided Wi-fi dongle in conjunction with a Mi-fi hotspot-style device, which seemed to mitigate the issue nicely. But adding the extra radio layer isn't really a good engineering practice to this type of problem, especially in an environment with as much Wi-fi present as NYC (Note VZW does not currently supply Mi-fi hotspots with 5GHz capability).

Enter the Pantech UML295. This modem is now available from VZW (off contract price is $200) and is well worth the investment for an upgrade. We've completed an initial round of testing with some cooperative Comrex fans in NYC and the results are super. Money quote: "The card worked flawless!!!". It's currently supported in all LiveShot firmware, and test firmware for ACCESS portable is available from our support guys.

Welcome to the Comrex Technical Blog

The idea here is to provide some insight into various technical issues we encounter that don't quite rise to the level of everyday support, and don't fit well into the sales channel. Hopefully readers will appreciate a deeper dive into some of our technical findings and concerns, and sometimes we'll "leave the reservation" and post about geeky concepts that don't have much to do with our product line.

I'll lead with a quick shot of a "test studio" we've recently begun building in the luxurious Comrex headquarters conference room. It's all very preliminary (note the folding table furniture) but the idea is to provide a station where some of our newer products, (in particular our STAC-VIP phone system and LiveShot video codec), can be put through its paces by the support, sales, and even administration staff. We hope to invest in some real furniture and make this more of a permanent fixture soon.

Look for more fascinating posts soon!