Clay Allsopp

Mobile At Scale

Oct 26, 2013

Facebook hosted a little one-day conference called Mobile @ Scale, featuring talks by developers from Facebook, LinkedIn, Twitter, Pinterest, Dropbox, and Mailbox.

The big trend across all companies: traffic is quickly becoming or already is a majority of product usage. Mobile is no longer just a "version" of the product - it is the product that users experience most often.

Each company felt growing pains as their web-driven development processes transitioned to mobile, and these were the recurring themes I noticed.


There was a common story in how every company's mobile product evolved over time:

  1. Started with just a few people, independently making "the mobile version" of the product.
  2. That small mobile team grew quickly as they were inundated with feature requests from every part of the company
  3. As team size grew, so did release cycles - folks wanted to get their new feature shipped at the last minute, delaying new releases by days or weeks
  4. Lots of pain
  5. Mobile team gets re-organized from "the mobile team" to having each product group retain mobile engineers (either full- or part- time), plus a core mobile infrastructure team.
  6. Switch to the train model for releasing apps - twelve big releases a year, with hot-fix releases when necessary.

Automated Testing

The only way to scale mobile app releases is to use continuous and automated testing as the definitive measurement of quality. If your mobile workflow sounds like "Let's run the automated tests...and then play with it for a day or two," then your tests aren't doing their job.

Writing unit tests is straightforward on iOS and Android, but the conclusion is those alone really don't answer "Is the app ready to ship?" In response, Facebook came up with a screenshot diffing test to increase automated confidence, and LinkedIn switched from focusing on raw code coverage to just reliably reproducing 90% of user-facing workflows.

Most companies now have their test suite running in continuous integration, and can reliably ship a green build without human interaction.

Feature Gating

Features are developed behind server-controlled gates. New code can ship in each release, but if it's not quite ready then it can shipped to a small percentage of users or outright disabled. This also opens the door for A/B testing and experimentation, which is still a very recent development for all the big companies on mobile.


Among the presenters, the current opinion is that HTML5 is a good choice for developers but bad choice for users. Current in-app browser implementations by Apple and Google are too prone to memory bloat, battery drainage, and general performance degradation to rely on as the primary vehicle for the app's experience.

Android Beta Program

Facebook and Twitter espoused the benefits of the Android beta program, which allows them to continuously test new code on hundreds of devices and environments. This means hardware- and OS-dependent bugs get spotted faster, without a huge investment in time or a device testing lab.

I think this only works really well for companies of their scale, where 0.1% of users still amounts to tens-of-thousands. Facebook says they have nearly one million users signed up for their Android beta, which is pretty wild.

Sharing Code

Dropbox and Mailbox independently arrived at the conclusion that they needed a shared library for their mobile apps. The most accepted and performant way of doing this is to build a C++ library, bridged by via JNI on Android and Objective-C++ on iOS.

These libraries concern themselves with model structure, server syncing behavior, and SQLite storage. This allows a split of mobile engineers into one infrastructure group and multiple platform-native UI groups. The UI engineers can safely interact with bridged C++ APIs without any knowledge of the underlying requests or storage mechanisms.

I think this is a really great idea - if you have the bandwidth to write a C++ library and bindings of each platform. In a perfect world, this would be in a language a little more accessible than C++. Hm...

One aside: Dropbox's libDropbox calls out to the native networking libraries (i.e. NSURLConnection I presume), instead of writing them from scratch in C++. However, Mailbox writes their WebSocket implementation at the libMailbox level, so their networking code is exactly the same on all platforms. Neat ideas.


To me, the whole event made it even more clear that mobile is eating the world. This is even more apparent in the developing world, where smartphones (albeit slow, small, Android 2.x devices) will be the first and primary way most people interact with software.

Good thing I'm working on hard problems in mobile - you should, too.

Enjoy this? Follow for more on Twitter