Using Android Helper Apps to Put Purple Robot on a Diet

By June 26, 2015Tech Blog

Since Purple Robot’s inception in the fall of 2012, it’s become the largest Android application I’ve ever written with over 50 different sensors, a built-in web server, and two complete embedded scripting environments. Lately, I’ve had to temper my ambitions a bit given that the app has begun to hit fundamental system limitations that are causing major issues moving forward.

The first limit that I hit was on older phones related to how Android installs APK files. The high-level summary is that a limitation on a fixed-size buffer on pre-3.0 Android devices doesn’t allow an Android app to install because the installer exhausts an internal fixed-size buffer and fails with a cryptic error message to the user. If you have had issues installing recent Purple Robot on 2.x devices, this is the culprit. This issue was addressed in Android Honeycomb by expanding the hardcoded limit, and then expanded further in the Android 4.x series. For a fuller description of the issue, Facebook has a great blog post on the topic. While I am not planning on using a native JNI extension to solve this issue, Facebook’s description was extremely valuable in helping me make sense of a cryptic error.

A similar issue that I’m currently experiencing on newer Android devices is also related to how Android parses Dalvik bytecode. When I compile Purple Robot against newer versions of the Google Play library, this happens:

[2014-05-09 14:51:53 - Dex Loader] Unable to execute dex: method ID not in [0, 0xffff]: 65536
[2014-05-09 14:51:53 - Purple Robot] Conversion to Dalvik format failed: Unable to execute dex: method ID not in [0, 0xffff]: 65536

This occurs because only 2^16 (65536) methods can be present in an application APK. Purple Robot isn’t that large on its own, but once I link against the Facebook SDK, Google Play services, Rhino JavaScript library, Apache Commons Math library, and more, the app grows and exceeds that built-in limitation.

Contributing to the problem is the growth of the Google Play services bundle, which continues to expand its own method count with each new release. 2,500 more methods were added between the 4.2 and 4.3 releases and those new methods count against Purple Robot’s method budget. (Recent investigations revealed that linking against the Play Services library will automatically add 14,000 methods in total.) A lot of this functionality are things that I don’t need, such as game center integration and mobile ad serving. However, I don’t expect Google to split these off into separate modules, so I’m stuck with the current situation for the foreseeable future.

One solution proposed is to split the application into separate APKs and load classes dynamically. Given that I’m trying to deliver as close to a checkout-and-compile experience as possible, I’ve ruled out this approach as it makes the build process non-trivial and introduces more potential for things to break at a basic level. I’ve come up with an alternative solution that I’ll be implementing moving forward.

Late in March, I was working on getting Purple Robot to communicate with the open-source Shion home automation platform that I’d built a couple of years ago. The Shion home-to-client protocol uses XMPP as a messaging layer and pulling house device events (lights on, temperature changes sensed by the thermostat, motion sensors, and more) into Purple Robot really only needed an XMPP client that spoke Shion’s XMPP dialect. In the past, I’ve used the Smack XMPP library on Java and that was my plan for this integration. However, I’d already encountered the methods limit described above, so I worked around it by creating a helper app that would have its own process and memory space and it would communicate with Shion on Purple Robot’s behalf. This was a pretty simple app to write and consisted of an activity to let the user enter their Shion authentication credentials and a persistent service that managed the XMPP communication. When the service received a new event from my smarthome, it would package the information into an Android bundle and use the built-in broadcasting features to send the information to Purple Robot. On the Purple Robot side, I constructed a very minimal probe that would intercept the broadcast and transmit the bundled data to the data pipeline. From there, the data could be processed by triggers (“alert me when someone turns on my desk lamp”), transmitted to the analysis server, or any of the other nifty things that Purple Robot does with probe data.

Since the initial Shion integration, I’ve successfully replicated this process by integrating with some (currently) closed-source research platforms with Purple Robot. Moving forward, I’m evaluating the current collection of Purple Robot probes and looking for opportunities to “slim down” the core offering by pulling out collections of plugins and offering them as add-ons. A couple of potential add-ons:

  • Social presence: Probes that capture your social activity on platforms like Facebook, Twitter, and Instagram.
  • Fitness: Probes that can communicate with the growing population of fitness trackers: FitBit, Jawbone, Shine, etc.
  • External devices: Shion devices, Nest appliances, Chromecast devices, etc.
  • Location data: Information from Google Places, Foursquare, etc.
  • Research: Integration with research platforms like Open mHealth and Cornell’s SAINT system.
  • Automotive: Integration with automotive data buses and smart consoles: Ford/Microsoft Sync, CarPlay

I could go on, but the path forward should be clear. Now that we have a proven mechanism for expanding Purple Robot’s sensing features without hitting the limitations described earlier, this allows me the latitude to push Purple Robot beyond the personal sensing space into a much more interconnected world.

Keep an eye open for these changes in the Purple Robot 1.6 release cycle.