Tuesday, May 29, 2012

Lessons in Software Development from Dish Network's Hopper DVR

How many times has this happened to you:

You develop and release software to production. The users report some issue with how the system works. Not a bug, mind you, but a bona fide 'it causes me all manner of headaches when the system does xyz'. Knowing why the system behaves the way it does, you explain all the esoteric edge cases that require the current implementation. In a nice way, you explain 'works as designed'. The users are educated and all is well, right?


 I recently had an all too human experience. You know the one I'm talking about. It's the one where, in an instant, your perspective shifts 180 degrees and you suddenly understand another perspective. It is a wonderfully clarifying moment.

What happened? I changed satellite television providers from DirecTV to Dish Network.

The Proposition

The motivation for change was alluring: new customer incentives, an HD signal, and a really whiz-bang three-tuner DVR called 'The Hopper'. This Hopper tech was sexy. It has 3 tuners and uses MoCA to network with tertiary thin-receivers and serve up content off the main DVR. Networking over my coax. Pretty cool, I thought.

But wait, there's more! It has this 'Prime Time Anytime' feature that, during prime time every night, it uses a single tuner to record the big 4 broadcast networks. Even better, they've raised the ire of the broadcasters by inventing tech to auto-skip all the commercials.

 I was seduced by the siren call.

The Installation

The first disappointment came during the installation. I didn't have line of site to the satellite that would provide my local channels in HD. I had to use another satellite. I was assured, though, that everything else was in HD. No big deal, right? I didn't think so, either, until I found out that the touted 'Prime Time Anytime' feature only works with HD locals.

Lesson 1

To meet deadlines, engineers make decisions to cut backlog items. It really hurts to be on the losing side of that decision. 

I'll admit, I'm guessing that the feature was cut and wasn't due to some eccentricity of non-HD satellite signals. Nonetheless, the impact is clear: the technical decisions you make have real impacts on your users. Here's to hoping Dish runs an Agile shop and plans to iterate into this feature.

DVR Usage

We started to notice that many of our recorded shows were being cut off a minute or two early. I did some research and was surprised at the explanation to this problem. I was also surprised that the problem has been reported since back in 2006. The explanation was thus:

Broadcasters, fed up with commercial skipping DVR technology, have been running their broadcasts past the hour by 1 to 3 minutes despite sending electronic program guide information to the contrary. 

This didn't make any sense to me. My DirecTV DVR didn't have this problem. Other people on the internet stated that they had Comcast, DirecTV and Dish installed.  They reported only Dish Network having the problem. Further, others reported that the satellite was in lock step with internationally accepted atomic clocks. The party line from Dish Network seemed to be:

'Our clock is right and aligns to the programming guide sent by the broadcasters. There is nothing wrong with the DVR. Its the broadcaster's fault'.

To confirm this, I called tech support and received the same explanation from the representative.

Lesson 2

Just because your software implementation is technically right doesn't mean it isn't totally broken from your user's perspective. 

The gist for me, the user, is that my shows were getting cut off with Dish whereas DirecTV had somehow figured out a way around this issue. Perhaps DirecTV's implementation was as simple as 'we know broadcasters do this, so if there's nothing scheduled to record after the show, then record a few extra minutes'. Or maybe DirecTV received better guide data. Irrespective, they figured it out.

Other Problems

Adding to this were a litany of miscellaneous issues. For instance, occasionally the Hopper would become intolerably slow after using it for a few days. Restarting it fixed it (memory or thread leak, anybody?). Also, the Joey units occasionally lost their MoCA connection to the Hopper thereby requiring a full restart of the unit. Which leads to:

 Lesson 3

When your users have lost confidence in your ability to address real (and perceived) issues, even the occasional minor issue can cause extreme consternation.

I don't mean to come off as overly harsh on Dish. As a whole, I've got a better signal, I can watch DVR programming in any room and I'm saving money. That's great. These few issues, though, served to remind me of these several important points of Agile software development.


  1. It IS frustrating getting a piece of equipment before the software is completely cohesive on all fronts. When did you get your Hopper? I know they released the Hopper before it had all its expected functions, and are doing rolling software pushes with new system updates 10,000 or so at a time. I haven't had any problems with mine yet, and with the latest push I have Auto Hop, but haven't had any DVR cut-offs. A Dish coworker was telling me about the On Demand manual timers, and you can actually set recordings to go an extra 5 minutes or so to cover that early cut-off you're getting. Sorry about the issues, that’s a bummer; can I get you a Cust. Service number or a link for Online support?

    1. I got it installed about a month ago. They have pushed several updates since I've got it installed. I wish they were more transparent about what was being fixed, though.

      The tech support agent suggested the record extra x minutes approach. Unfortunately, it will cause my subsequent shows to be skipped if all the tuners are busy. I'd really like it to automatically do it for me if there's nothing else scheduled.

      Thanks for the offer of online support, but it looks like I'm stuck. All in, it's an upgrade from my 6 year old DirecTV equipment. I just wish it worked better.