There is a flaw in thousands of apps (including top apps like HBO GO) that will cause them to crash on Android M, which is expected in September. We’ve scanned the Google Play Store and identified which apps need to be fixed, as well as which libraries most commonly are affected by this issue. It’s particularly hard to know if you have this problem because the apps work fine right now and the issue is usually in how their third-party libraries were built.

To see if your apps are affected, search for them with this simple tool we created. Our Searchlight users have been notified of this issue and have been updating their apps.

The problem is that some native code libraries link against a private Android API that is not part of the NDK. Google recently changed the OpenSSL API by moving to BoringSSL. The BoringSSL library provides a much-needed cleanup of OpenSSL, removing a lot of complicated functionality that led to numerous recent security flaws like Heartbleed.

The OpenSSL library is not included in the official Android NDK, so theoretically Google could make this change without affecting any apps. But developers aren’t so predictable as to follow the rules all the time, and there are numerous guides instructing you to retrieve libraries from a random device and link against them (note: this is an example of what not to do).

What will happen is that apps that link against the phone’s libcrypto.so or libssl.so libraries will crash on Android M devices. It’s unlikely these crashes will be logged by crash reporters like Crashlytics or ACRA because this problem happens at the dynamic linker level, which is probably before the reporter can initialize. The only clue that something is wrong may be a flood of negative user reviews.

This flaw is so subtle that even Google’s own YouTube app had this issue until they fixed it a few weeks ago. Other apps that need to be fixed include heavy hitters with a cumulative 1 billion installs like UC Browser, Waze, Modern War, and Zalo.

We scanned a million apps and found there are almost 3,000 that have this flaw. But what are they doing? We dug into their native libraries to find the pattern.

There are 5,689 copies of 393 unique libraries in these apps. (The replication is because you need one build for each CPU architecture.) Here are the libraries that most frequently have this issue.

Library % of affected apps
libdodo.so Web/RPC network library 19.2%
libpjsipjni.so SIP and NAT traversal library 10.1%
libopvpnutil.so OpenVPN wrapper library 7.0%
libcasdkjni.so MM billing payment SDK 5.6%
libsqlcipher_android.so SQLCipher database encryption SDK 4.7%
libbdpush_V2_2.so Baidu push messaging SDK 4.3%
libnexalfactory.so NexStreaming video SDK 3.2%
libopenvpn.so OpenVPN wrapper library 1.6%
libvoPreHTTP_OSMP.so OMXPlayer media player 1.5%
libMozzoMZV.so Mozzo MZV document format 1.4%
libodyssey.so PDF rendering library 1.1%
libkonyjsvm.so Kony cross-platform JS native extension 1.0%
libmedia_jb.so Media common library for Jellybean 0.9%

We were surprised that a C++ networking library is the most common, being present in almost 20% of affected apps. The libdodo library is not a dependency of any other library in our affected set, so one guess is that it is just popular with developers. The other top categories include various multimedia and payment SDKs that frequently link against crypto libraries.

Because OpenSSL and its underlying crypto library are so common, there’s quite a long tail of miscellaneous libraries with this issue, from curl to AirTunes, UpLynk streaming to AllJoyn device discovery.

There are three ways to fix this:

  1. Add the OpenSSL source code or other crypto library to your native code project. This is the best way and is guaranteed to remain compatible with builds for future devices.
  2. Include the libssl.so and/or libcrypto.so libraries in your APK. If you don’t have source code to an affected third-party library, this may be your only option. It’s certainly a good quick fix to make while you investigate other options or wait for a patch. The only downside is that there’s still a possibility the libssl.so file you’re using will not be binary-compatible with future versions of Android.
  3. Use JNI from your native code to call into the Java crypto API. This is a more complicated approach, so be careful to take the appropriate precautions when calling into Java crypto, just as you would when calling it from Java.

The more we learn about apps, the more we’re surprised by how subtle issues like this are. If you write a lot of apps, it’s difficult to remember what libraries you used and especially what versions. Look up your apps and drop us a line to let us know what you’d like to know about them.