OK Labs Story (9): The End Game

My October ’09 report to the Board was basically ignored. The more critical one of the investors was initially interested in following up, but in the end nothing happened. Steve obviously worked on him, but we also got, at a perfectly inopportune time, two deals which seemed to prove me wrong.

The AMSS disaster

One had started earlier, an Asian phone manufacturer wanting to build a “mass-market smartphone”, i.e. Android and modem stack on the same low-end (ARM9) processor core. I never understood why, and I was kept away from the customer or any real information from them, so I can only suspect that they were after an iPhone lookalike on the cheap (where looks would be more important than functionality).

So, we actually build a new product specifically for outdated hardware (the successor core ARM11, an ARMv6 implementation, was already in widespread use then). Technically it was a fun challenge. The project, code-named Marvin (the paranoid Android, paranoid because he’s never run on an ARM9 before) back-ported Android to ARMv5. The interesting bit was to get performance on the processor, which features a virtually-indexed, virtually-tagged cache. Windows as well as Linux deal with this cache by flushing it on every context switch, a performance killer. However, we had many years ago invented an approach to fast context switching on ARMv4/5, which made Linux faster by virtualising it; we used the same approach on the original virtualised phone, the Motorola Evoke. So we could run Android on that processor pretty much as fast as it could possibly be, and our engineering team made it work, of course.

The catch was that the customer was using Qualcomm chips, and was therefore running Qualcomm’s AMSS modem stack. So we had to port AMSS to the Microvisor. Technically this was not a big problem for our engineering team (and they did it just fine), but there was a legal issue: we did not have a source license to AMSS. The customer claimed that they had the rights to sub-license for the purpose. I voiced concerns at the board, resulting in a supposedly thorough analysis of the legal situation, which supposedly showed it was allright.

Obviously, the clean thing would have been to simply talk to Qualcomm, as they were bound to find out eventually. But for some reason Steve thought he could get away without telling them. (This is consistent with Steve’s 90-day horizon, he routinely goes for short-term gains even if this seriously damages long-term prospects, while my attitude is exactly the opposite.) It beats me why the rest of the Board followed.

Of course, Qualcomm eventually found out, and sent us a cease-and-desist letter, with which we complied. That (which happened after the presentation of my market analysis) was the end of this deal, but also of our once-great relationship with Qualcomm.

The poisonous Dash deal

There was another project which seemed to undermine my analysis, the only one that ever came through one of our investors. One of the directors had long-running high-level links with a major US network operator, let’s call them “Dash Mobile”. He got them to contract us for a prototype “mass-market smartphone”, running Android on an ARM9-based, single-core phone provided by another Asian manufacturer (not involving Qualcomm IP). Again, I never got enough information to understand why Dash was interested in this and whether their reasons added up. But, given the small size of the contract, it might be them just using some spare cash to investigate something which could be seen as having the potential for some market disruption, which Dash urgently needed (they weren’t going so great at the time).

Needless to say, the project went no-where (although that didn’t become obvious until after my departure). OK engineering delivered, as usual, and our overheads were low enough to be in the noise margin. But running a smartphone OS on the low-end processor did simply not result in a satisfying user experience (even with zero virtualisation overhead). But for a long time it seemed like a great project totally in line with Steve’s PR, contrary to what I was saying, and thus undermined my credibility. By the time I was proven right, it was too late. The timing of the Dash deal was disastrous.

In the end, the Marvin project was damaging in several ways. On the one hand we wasted a year or so, and millions of dollars, on developing a product for which there was no market, it was a complete waste. On the other hand it was a serious hit on morale in the engineering team: people had worked their asses off for many months, including regular work on week-ends, to meet the schedule. And in the end they saw that, while the work was technically good, the resulting product was useless. Several of the top people left after that.

Blood in the Board Room

Eventually I realised that I was powerless to do anything constructive in the company. Furthermore, 90% of frustration in my live originated from OK – in the mornings I dreaded looking at my email, which was full of shit having come in from Chicago overnight. At the same time, most of my fun came from research and working with bright students, and I was doing less and less of it. I decided I was too old to play such a masochist game and had to get out.

However, I thought I owed it to the many great OK engineers to at least make one more attempt to change things, unlikely as it was to succeed. So, in May 2010 I forced a bloodbath in the boardroom, and, predictably, the blood was mine, as the investors backed the CEO. Note that to this date we had a single quarter where company revenue and booking performance had matched the goals, and that was only because a large Motorola deal had come in late and spilled into a quarter that had low expectations; the sum of the two quarters was still well below expectations. But, of course, salvation was just around the corner, and we would soon be profitable. I have no idea why the VCs kept buying this…

The exit

I was given a face-saving 12-month consulting deal for leaving at the end of June ’10, and remained on the Board.

Incidentally, Benno left at the same time, voluntarily. The fundamental reason was the same: he couldn’t stand the bullshit and daily frustration any longer. (Benno and I are great friends again, I was invited to his wedding the following year, and his new startup Breakaway Consulting is a close partner of NICTA. We’re jointly developing eChronos, a verified unprotected RTOS which is already being licensed for medical implants. And he founded Cog Systems, the moral successor of OK Labs.)

The internal announcement of the departures was telling. I was asked to draft my own “to ensure it’s correct”. I kept it brief and focussed on simple, public facts, and, of course, didn’t praise myself, that would be the job of others. Steve mailed it out as is, with no attempt to mention my contributions to the company. In contrast, what Steve wrote about Benno was full of praise.

Nevertheless, were also both asked to pack up our desks after hours to avoid “unsettling” people. Steve has always under-estimated the intelligence of our engineers, and generally has no clue how they tick. People were already fairly unsettled, and the way I was farewelled clearly looked like a sacking to everyone. Given that the majority was there because of me, this obviously didn’t help morale at all.

While following Steve’s instructions to the letter, I did my best to counteract the effects on morale in engineering. I met up with them for lunch, putting up a cheerful face and taking positively about the whole development. Not sure how much this helped in the end, but it would certainly have been better than just vanishing the way Steve wanted me to.

The sale

With Benno and myself gone, the company had no-one left with a technical vision. All they could do from now on was implementing the designs Benno had made before leaving, and do whatever Chicago dreamt up. Needless to say, the company’s performance didn’t improve. The investors lost patience and wanted an exit: OK Labs was put up for sale.

The search for a buyer dragged on and on. Hardly surprising, of course, given that the “business plan” was a joke that didn’t hold up to scrutiny by anyone with technical insight. The “business plan” was built around our “enterprise strategy”, which basically meant “we’ll sell to enterprises” without any clue why they would buy from us, there was no value-add anyone could formulate in a way I could understand. Any real opportunities were outside that “strategy”.

Ironically, although maybe not surprisingly, by the time the company was finally sold, the only royalty deals we ever made were exactly in the two areas I had championed: automotive (where we may be in cars by now) and security (where there is at least one product in use by the military of a NATO country, probably more). But never a cent in royalties in consumer/enterprise mobile!

The strategy for the sale could only be to find someone interested in buying the engineering team (no longer the A team, a few individuals notwithstanding, but still very good) or the existing customer relationships.

What amazed me (despite all I had seen in the past) was the slide deck which was used to shop the company around. As part of the second investment round, OK Labs had acquired all rights to seL4 and its verification, and NICTA had delivered it sometime in 2010. Yet the sales deck didn’t mention seL4 a single time! The only obscure reference was the highly-misleading bullet point “World’s only fully verified Microvisor” (it’s seL4 that was verified, not the OKL4 “Microvisor”).

This is truly stunning! In 2011, MIT Technology Review listed seL4 in its “TR10”, what it judges the ten most important breakthrough technologies. So, OK sat on this truly unique technology, and the CEO had no clue what to do with it! Steve simply never understood seL4 and its potential. Which must make OK Labs the only company ever that owned a TR-10 technology but didn’t know what to do with it!

The way he came to his assessment that seL4 wasn’t of any use was a “market validation” exercise we had done earlier. It used the process Rob had used in the past at Wind River: You’ve got some ideas on how to evolve your product, so you go to all your customers and ask them what they thought about it. Which is a perfectly fine approach for a company with an established products it wants to improve, and an established, large customer base. In our case (where we had no relevant customers) we went to potential customers, all established players in their field and, effectively, asked them “hey, we’ve got this great technology that will completely disrupt your business, are you interested?” Can anyone guess the answer? But this is how strategy was decided at OK Labs.

The spoils

In the end, the company was sold to General Dynamics in August 2012, for a price that gave our investors more or less their money back.  None of the common shareholders saw a cent. In the dying days of the company, the investors had doubled as loan sharks, bridging the company at incredible interest rates, making double sure that there was nothing left for anyone else. (Actually, I was offered $1000 for signing away some irrelevant rights, which I declined, as being below my dignity.)

The only person who made a profit was the one who comprehensively fucked it up. Steve got about $900k in a combination of completion and retention bonuses (and on top of that may have retained the full-year separation pay he had written into his own employment contract). Isn’t the world of business wonderful?