OK Labs Story (5): Qualcomm

Our first customer was Qualcomm, and they were the most important one for a long time. During its lifetime, OK sourced the bulk of its revenues from Qualcomm. They were a critical partner.

The relationship was fantastic for a long time and, I’m told, quite unusual. Qualcomm doesn’t have a reputation of being an easy partner, particularly for small players. They also don’t have a track record of licensing technology, their usual approach is to by outright (technology or company) or do it themselves.

Throughout we maintained an excellent relationship at the engineering level. The technical people at Qualcomm held our engineers in high regard, I never heard anything other than compliments.

Steve’s original achievement was the re-negotiation of our Qualcomm contract, providing secure (and significantly increased) funding that was the stepping stone for creating the company. And he managed to increase the volume of the contract a few more times.

By that time, Qualcomm had long decided to base their future firmware versions on our L4, and, in fact, end-user products were close to shipping. Getting from there to deployment on a billion devices was now mere engineering, and we knew we wouldn’t fail on that.

However, over time, Steve’s style created problems with Qualcomm. He ended up seriously annoying a sequence of division heads (i.e. the people controlling the money), mostly through his negotiation style. As a company, we gained a reputation of arrogance, which really appalled everyone (who knew about it) in Sydney.

Negotiations, in true Steve style, always went to the brink, and all deals were last-minute. Essentially we were betting the company every year when the contract was due.

I vividly remember one particular Steve moment: The Qualcomm manager was hospitalised. As Steve reported at the Board meeting, he called him up “to wish him well” but at the same time remind him of our negotiations. Steve seemed mightily proud. But, as I soon heard through the engineering back-channel, the Qualcomm manager was hugely unimpressed.

This was on the back of an earlier disastrous misjudgment in dealing with Qualcomm. Google was moving into the mobile space with Android, and the first Android phone was the Nexus One, or HTC Dream. We knew it would have a Qualcomm chipset, with OKL4 the OS on the baseband processor. Steve thought it a smart idea to have a press release timed for the release, titled something like “Google had a Dream, and we’re in it”.

To say that Qualcomm was unimpressed about us preempting their PR that way is the understatement of the century. They were livid. They were foaming out of their mouth. Steve got a call saying “take that down immediately!” As Steve proudly reported at the board, they fumbled around with “communication issues” due to flights etc for a few hours to give it more exposure, until finally taking it down after about half a day. Fantastic way to treat your friends, and clearly a good way to make the relationship last, right?

And it was so annoyingly short-sighted and unnecessary. We could have simply waited for a few days, until Qualcomm had made their move. There was a lot of hype around Android, and it wasn’t going to die down in a day. We might even have got more attention by staying away from the peak of the hype, and in any case, by timing it less aggressively, Qualcomm wouldn’t have objected, the release would have stayed up, it would have come up in Google searches, etc. But long-term thinking isn’t Steve’s strength.

Needless to say, an increasing number of people inside Qualcomm had their knifes out on OK, and were waiting on the opportunity to stick them in. And the opportunities were bound to arrive. The main thing that kept the relationship going was the excellent work done by our engineers, which clearly provided value to Qualcomm. I talked more than once to senior people on the Qualcomm side who were torn between enjoying working with OK engineering, and pissed off with what they saw as arrogance and lack of trustworthiness of OK management. The net effect that influence inside Qualcomm shifted from our friends to our opponents.

Early in the life of the company we were introduced to the Qualcomm team which was working on their new DSP architecture, what later became known as the Hexagon. We met the chief architect and a number of other staff involved. Hexagon was to become Qualcomm’s proprietary replacement for ARM cores as their modem processors, and L4 was to be ported to that architecture.

On the engineering side the work made good progress. However, problems arose from multiple directions. One was Qualcomm-internal politics, where someone started developing their own (unprotected) RTOS, and started to sabotage us. But he was greatly helped by Steve’s waning popularity with Qualcomm management. After a long and painful period of politics and negotiations, Qualcomm finally decided to use the home-grown RTOS for the Hexagon.

Technically, this was a step backwards for Qualcomm, from a protected system to a traditional flat address-space model. But for us it meant that our days as the OS underneath Qualcomm’s modem firmware were numbered. Today (by all I know) L4 still ships on Qualcomm chips (so total L4 deployment should now be well over 3 billion), but on one or more of the auxiliary cores, no longer the main game.

While clearly not good, this wasn’t a disaster yet. Qualcomm had, throughout, received our code under the original open-source license (and could have just released it in source form). We didn’t earn royalties, only (lucrative) consulting revenue. This was one (but by far not only) reason for a complete re-design and -implementation of our kernel, which resulted in the OKL4 Microvisor, the basis of our virtualisation product. Essentially we now had two code bases, the “microkernel” for Qualcomm and the “Microvisor” for everyone else.

Our big chance to change the Qualcomm relationship arose when Qualcomm looked at supporting virtualisation on the application processor, and negotiated for a (non-exclusive) buy-out license of the Microvisor, to ship on their apps processors. Things seemed to go well, until Steve put an insanely huge asking prize on the table. Qualcomm was seriously pissed off, and the relationship never recovered. Internally our aim was to get just over 1/6th of the price asked. Most people involved believe that, had we asked what we really wanted, Qualcomm would have just accepted it without much negotiation. But with the price Steve asked, it was too bloody obvious that they could have done it themselves much cheaper. In the end we got nothing, except a terminally poisoned relationship.

The final straw came later, in the form of the AMSS disaster, which I’ll discuss in a future blog.