A behind-the-scenes look at LinkedIn’s mobile engineering

Reference: http://arstechnica.com/information-technology/2012/10/a-behind-the-scenes-look-at-linkedins-mobile-engineering/

LinkedIn is the career-oriented social network that prides itself on professional excellence. But the company's original mobile offering was anything but—it left much to be desired. There was an iPhone application, but no support for Android or tablets. The backend was a rickety Ruby on Rails contraption; afflicted with seemingly insurmountable scalability problems. And despite serving only seven or eight percent of the LinkedIn population, the company's original mobile build required approximately 30 servers in order to operate. This was clearly not made to sustain a growing mobile user base.

Now, a little over a year has passed since LinkedIn relaunched its mobile applications and website. And the company recently marked the anniversary by debuting a number of new mobile features, including real-time notifications and support for accessing company pages from mobile apps.

Mobile is gradually becoming a central part of the LinkedIn landscape. The company says roughly 23 percent of its users access the site through one of its mobile applications, up from ten percent last year. As our friends at Wired reported last month, the underlying design language and development philosophy behind the company's mobile experience is playing an influential role as the company works to revamp its website.

No one knows more about this paradigm shift than Kiran Prasad, LinkedIn's director of mobile engineering. Prasad was one of the key players in LinkedIn's efforts to resurrect its mobile offerings, and he was kind enough to walk Ars through the process. LinkedIn's mobile man provided a behind-the-scenes look at how the company built its mobile applications and the associated backend infrastructure, while also describing the design strategy the company used to build its mobile interfaces. As you'd probably expect from the brand, it was all a professional level effort.

Initiating an overhaul

LinkedIn decided it was time for a massive overhaul as the company began recognizing the increasingly important role that mobile access would play for the social network's users. Jet-setting professionals rely on their smartphones to stay in touch while they travel, right?

Mobile soon became a major strategic focus for LinkedIn. In August of 2011, roughly five months after the mobile overhaul began, the company launched an all-new set of mobile apps powered by a completely new backend.

When the effort began, the mobile engineers at LinkedIn had several major goals. They wanted cross-platform compatibility, with support at launch for Android, iOS, and the mobile Web. They also wanted to simplify the user interface, taking the number of icons on the main screen down from 12 to three or four. Finally, they wanted a holistic rewrite—rebuilding both the frontend and backend together with an eye for boosting scalability.

Prasad previously worked on WebOS at Palm before joining LinkedIn. He brought a wealth of knowledge about building native-like user interfaces with modern Web standards. He regards the mobile Web as a platform in its own right, one that LinkedIn had to support well alongside Android and iOS. So Prasad decided that making HTML5 a central part of the company's mobile strategy would make it easier to reach all of those environments.

HTML offers a useful way to reach more screens, but Prasad believes there are still many places where native user interfaces and native code are needed in order to deliver the best possible user experience. LinkedIn set out to use a hybrid model; blending the two to theoretically offer advantages of both.

This approach made it easier for a small team to support multiple platforms. And crucially, it allowed other LinkedIn developers outside of the mobile team to contribute to the effort using their existing skills.

Everything must be simple

Prasad told me that simplicity is at the heart of LinkedIn's mobile vision. The company's internal definition of simplicity, he said, holds that a good solution must be fast, easy, and reliable—in that specific order. Each of those characteristics is considered ten times more important than the next, making performance the chief concern. That philosophy motivated his team's decisions in almost every area, ranging from visual design to backend engineering.

The thing people value most, according to Prasad, is their time. When users encounter flaws in software, they tend to be more forgiving if the software is extremely fast. A crash is less disruptive, for example, if the application is quick to restart and offers a quick path back to where the user was.

Speed is also especially important for mobile experiences in his eyes, because smartphone users tend to have shorter sessions (often less than three minutes long). Users expect applications to deliver relevant information as quickly as possible.

Ease of use was his next major priority, and Prasad said LinkedIn measures this by counting the number of clicks that it takes for a user to complete a given operation. If it takes more than three, he said, users are going to lose patience and easily be drawn away by push notifications or other things happening on their phone.

Reliability is the final of the three priorities. Basic robustness and stability are important, Prasad told me, but reliability also encompasses other ideas like consistency and predictability. To him, a reliable application is one the user can depend on to behave the same way every time. When the user taps a button and it takes them to another screen, for example, going back and tapping on the same button should take them there again. A surprising number of applications ignore that seemingly obvious design principle, Prasad said.

During the design process, LinkedIn used a simple metaphor to encourage predictability and ease of use. The idea is that an application is like a house—there are rooms with specific functions, and there are corridors that connect those rooms in a practical way. When you are putting together a room, you don't fill it with an incongruous range of functions. You may end up with a living room, a bedroom, a kitchen, and a bathroom. But you don't cook in the living room and you don't sleep in the bathroom.

Continuing this house analogy, when you begin designing an application, you start by defining the structure. You determine what rooms you want, what their purposes will be, and how those rooms will be connected. You don't start with the visuals (in the house analogy, these are like the carpeting or the paintings on the walls). When you introduce a feature, you think about the room in which it belongs.

For the mobile application, LinkedIn decided that it didn't want more than four "rooms" of functionality. You can clearly see the house metaphor in action when you open the LinkedIn iPhone application. The main screen limits itself to four icons: your updates, profile, messages, and groups. It serves as the hallway, allowing the user to tap an item in order to enter a given room.

Building a hybrid app with HTML5 and native code

LinkedIn uses a combination of HTML5 and native code in its user interface. Prasad told me which technologies they use for various parts of the app and gave me a detailed explanation of how LinkedIn has made the two approaches interoperate.

Prasad says that HTML is very effective at rendering what he calls "detail" views, large blocks of informational content consisting largely of rich text and graphical media. The flexible layouts offered by HTML make it useful for such usage scenarios, but there are also major areas where LinkedIn chose to use native controls. Why? Prasad said Web technologies weren't entirely up to the task.

Prasad dismisses the idea that you have to choose between native controls and HTML, saying that it represents a false dichotomy. Ultimately, LinkedIn's iPhone application consists of 70 percent HTML and 30 percent native.

The biggest example he cited of an area where HTML5 still falls short as a user interface layer for data-driven mobile applications is in displaying long lists of content. He said that native widgets are needed in order to achieve smooth and seamless scrolling for list displays with hundreds or thousands of items. Attempting to display such lists with HTML and JavaScript proved impractical. Prasad saw this as especially true for "infinite" lists, where new content is fetched dynamically and appended to the bottom as the user scrolls.

When implementing an infinite list, you can't just keep adding items to the bottom. Memory overhead quickly becomes unacceptable unless you also simultaneously pull items from the top after the user has scrolled past. Prasad explained that manipulating the HTML DOM (Document Object Model) during scrolling caused some stuttering.

It's hard to measure exactly what is going on when that happens, but LinkedIn guessed that page layout computations or JavaScript garbage collection were sapping too much of the device's limited resources. On top of the performance issues, Prasad felt it was exceptionally difficult to implement kinetic scrolling in HTML that felt truly native across platforms.

It's worth noting that rival social network Facebook raised very similar concerns about list performance and scrolling in a recent message to a W3C mailing list. Facebook largely ended up swapping HTML for native user interface controls in the latest update of its own mobile application.

For Prasad, however, the limitations of current Web rendering engines don't demand a full return to native code—it's a matter of using the right tool for the job. He dismisses the idea that you have to choose between native controls and HTML, saying that it represents a false dichotomy.

Ultimately, LinkedIn's iPhone application consists of 70 percent HTML and 30 percent native. The Android application is roughly 40 percent native and 60 percent HTML. The Android application relies more extensively on native elements because the platform's HTML rendering engine isn't quite as capable as the one on iOS.

LinkedIn had started with an 80-percent-native Android application, but has been able to gradually increase the amount of HTML over time due to incremental improvements in the platform's HTML renderer. Prasad is hopeful that the greatly enhanced HTML renderer that comes with Chrome for Android will eventually be available in the embeddable HTML display component that Android supplies for third-party applications. He described Chrome for Android as "awesome" and said that its support for hardware-accelerated rendering is great.

Aside from handling some performance-critical user interface elements like lists, the native part of LinkedIn's application also serves a vital role in trapping and responding to errors uncovered in the embedded HTML views. In cases where an embedded HTML pane encounters a fatal error, the native part of the application can cleanly bring it down and then repopulate it, often without the user even knowing.

Prasad briefly discussed his views on Web runtime solutions, such as PhoneGap, that aim to simplify HTML-based mobile application development and provide such applications with native shims to underlying platform functionality. While PhoneGap and similar frameworks are useful for rapid prototyping and for companies with limited resources, he said, it's better to implement the native parts of hybrid mobile software in a way that meets the specific requirements of the individual application. Ideally, he told me, developers should choose the right balance between native and Web for each application and then build their own bridge between the two environments.

I asked him to describe the specific mechanism he uses on each platform to enable communication between native code and the HTML user interface elements. On Android, the LinkedIn application largely relies on the platform's built-in support for exposing specific Java functions into the JavaScript runtime of an embedded HTML view. That feature made it relatively straightforward to bridge the gap.

On iOS, the matter is a bit more complicated. He said the company tried several different methods before settling on one they felt was most effective. Their first approach involved using a platform API to "eval" JavaScript expressions in the embedded WebView. This proved to be too computationally expensive, sometimes introducing undesirable jerkiness.

The second approach that LinkedIn tested was one using WebSockets to establish a connection between the HTML components and native code. This worked exceptionally well from a performance perspective, but wasn't stable enough for practical use. LinkedIn used this method in an actual release, but later replaced it.

Finally LinkedIn tested, and ultimately chose to adopt, something surprising. The company embedded an extremely lightweight HTTP server in the application itself. The HTTP server exposes native functionality through a very simple REST API that can be consumed in the embedded HTML controls through standard JavaScript HTTP requests. Among other things, this server also provides access to the user's contacts, calendar, and other underlying platform functionality.

When I asked Prasad if this approach raised any security issues, he said that the company reviewed it internally and found it to be acceptable. There are a number of precautions that are taken in the application to prevent it from being abused. The server is bound only to localhost and can't be accessed by other devices on the network. It also suspended immediately whenever the LinkedIn application is sent to the background.

Building a more scalable backend

When LinkedIn was deciding how to rebuild its backend, the company used a similar philosophy to the one that had directed the application design decisions. Like everything else, they wanted their backend technology stack to be fast, easy to work with, and reliable.

In a large-scale backend system, Prasad said, companies typically adapt the traditional model-view-controller (MVC) pattern into a three-tier system. The bottom tier consists of the database storage layer, the middle tier handles caching and some business logic, and the top tier serves as the presentation layer, generating your HTML views.

When you are building a mobile application, he said, this structure is no longer applicable. Your presentation layer is on the device itself, often consisting of native user interface elements. LinkedIn wanted a new kind of middle tier that would facilitate more effective communication with a mobile frontend.

In order to minimize the latency introduced by establishing new connections, the LinkedIn developers decided the application should establish as few connections as possible with the server. This led them to a model where the application is essentially piping all of its data through a single connection that is held open for as long as it is needed.

The client application connects to a system on the backend that functions as an aggregator, pulling together all of the data that it needs from various components of the backend stack and combining it all into a unified stream of data that can be piped down to the client application through a single open connection.

To build this streamlined middle tier, the LinkedIn developers wanted to use a lightweight event-driven framework. They also wanted an environment that would be well-suited to handle the aggregation and string interpolation capabilities required for the service. They tested several candidates, including Ruby with EventMachine, Python with Twisted, and the JavaScript-based Node.js framework.

They found that Node.js, which is based on Google's V8 JavaScript engine, offered substantially better performance and lower memory overhead than the other options being considered. Prasad said that Node.js "blew away" the performance of the alternatives, running as much as 20 times faster in some scenarios.

Another advantage of adopting Node.js for the project is that it allowed LinkedIn's JavaScript engineers to put their existing expertise to use on the backend. Prasad said the company was able to combine its frontend and backend mobile teams into a single unit. The event-driven nature of frontend development made it easier for the user interface programmers to understand the way that Node.js works.

Prasad was practically giddy when he told me just how much the transition from Rails to Node.js improves the scalability of LinkedIn's mobile backend infrastructure. Impressively, the company was able to move from 30 servers down to three, while still having enough headroom to handle roughly ten times their current level of resource utilization.

Firefighting scalability problems on the old infrastructure had been a major distraction, one that forced the company's engineers to spend a lot of time just keeping the system running. With that problem defeated, the engineers were free to spend more time focusing on user-facing product improvements.

Perfection is elusive

The mobile engineers at LinkedIn have done some impressive work to make the application fast, responsive, easy to use, and aesthetically pleasing. But reliability, the third item in LinkedIn's taxonomy of mobile priorities, remains an issue for some users. LinkedIn's iOS application currently has an average rating of two-and-a-half stars out of five, with an average of three stars across all versions. The Android application fares a bit better, with 3.7 stars out of five. LinkedIn rolls out routine updates as it continues to improve its software.

LinkedIn's mobile application has also been the subject of privacy concerns. It faced scrutiny earlier this year when security researchers discovered that it was programmed to send user calendars back to the LinkedIn mothership. The company responded by tweaking this behavior and attempting to make it more transparent to the end-user.

But ultimately, building a mobile experience for a popular social network is not an easy task. Whatever shortcomings currently exist are being monitored, and could potentially be addressed in the same way this redesign built on past feedback. Prasad's description of the underlying technical details not only offers a glimpse into the challenges and complexity of this problem space, but he provides unique insight. When you experience LinkedIn mobile now, you're experiencing the company's engineering culture and the philosophy that guided its design and development process.