Shipping Two Apps in One on Android

The initial prototype of the new Snapchat Android app (covered in Part 1 and Part 2) was written in a completely separate repo. As we put together the plan for taking the initial prototype to production, we were faced with a very important choice to make: How will we release and test the new app once it’s ready? We knew early on that rebuilding the app is only half the battle - releasing a stable app that doesn’t regress any of our business metrics is as big of a challenge (if not bigger). Therefore, we needed to start testing the app with some of our users very early on in the process. We came up with following set of requirement for our release plan:

• We need to be able to AB test the release

• We need to be able push updates to the original (OG) Snapchat app

• We need to be able to scale up / scale down the roll out of the new app as we see fit.

We considered launching a separate app in the Play Store, but that didn't allow us to gradually ramp up the app roll out. We also considered using the Play Store staged rollout and slowly increasing the rollout percentage as the app became more stable, but this prevented us from making updates to the original app. Moreover, both options made it impossible to AB test the new app. Eventually, we realized we had to ship both apps together in a single APK (under original package name) in order to have the flexibility we wanted. While this option would come at a cost of increased app size and complexity (as we will discuss later), it was the only way to fulfill the requirements of our release plan.

Complexity of Having Two Apps in One

The project to figure out how to ship both apps in one APK was appropriately named “Turducken”, after a famous American Thanksgiving dish, which consists of a chicken stuffed inside a duck stuffed inside a turkey. A couple of technical requirements of Turducken were:

• Turducken shouldn't cause the apps to depend on each other. You should be able to build a standalone original app, a standalone new app, or the combined Turducken app by choosing the right build variant

• When in Turducken mode, only 1 app could be active at a time and the active app needed to work in complete isolation (ie: it should be no different than running a standalone app)

Unfortunately, shipping two apps in one APK and keeping them isolated is not something that is natively supported by the Android operating system: Android does not directly launch Applications, it launches manifest components (Activities, Services, Broadcast Receivers) which are bound to a specific Application in the AndroidManifest file. When a Component is launched (via the Launcher, intent receiver, etc) the OS will create the parent Application and then the needed Component, but there are no hooks available in this process. Additionally, an Android APK is only allowed to have a single Application configuration in the AndroidManifest. All of the manifest components are child nodes of this Application. When packaging an APK, the Android build tools merge the manifests of the libraries that the application depends on into a single top-level manifest. As a result, the manifest components for all applications are included in the single <Application> tag, regardless if they are relevant for the active application. To work around these limitations of the Android OS and build a Turducken app, we developed a small library called Stuffing, which allowed us to have multiple Android Applications stuffed into a single APK. We are open sourcing Stuffing as a part of this blog post, and if you are eager to take a look at the code, you can head straight to Github: https://github.com/Snapchat/stuffing

How Stuffing Works

The Structure of a Stuffed APK

In a stuffed APK, each child application is called an app family. An app family includes all AndroidManifest components that correspond to a given child application. Given that the APK is only allowed to have a single Application class, the Stuffing library uses a common top-level Application class or AppShell , which delegates to multiple ApplicationLike classes that represent individual app families. This concept is inspired by the Exopackage functionality from Facebook’s buck build tool.

An ApplicationLike is type that adheres to roughly the same interface as the Android framework Application type. It acts as a delegate for any application-related functionality in the application lifecycle and manages the global state for an app family. DynamicLaunchActivity serves as the primary entry point for the stuffed application. Given that the Android system doesn’t know which app family is active when launching the app, Stuffing provides a layer of indirection/routing via this special activity. When this activity starts, it uses the DynamicAppManager to determine which child Activity should be launched

Managing The App Families

DynamicAppManager class is the heart of Stuffing as it’s responsible for managing the multiple children applications stuffed into the APK:

• It maintains the state of which app family is currently active (via shared preferences in
  DynamicAppManagerPrefs)

• It provides the hooks for switching between different app families.

• It enables/disables AndroidManifest components depending on which app family is active, thus, providing the required level of isolation between each app family

There are two discrete implementations of this interface:

• MultiDynamicAppMananger - manages multiple applications

• SingleDynamicAppManager - manages a single application

The primary reason for having two implementation is provide an easy way to build multiple variants of the app. For instance, while the primary build variant of the app could include both apps, it is useful to have an additional build variant for testing that only include your new app. Swapping different implementations of DynamicAppManager makes this easy.

Switching between app families

When DynamicAppManager.switchToAppFamily is called, the class will iterate through all manifest components using PackageManager and disable/enable the right set of components depending on the target app family. Switching between app families can take a while if your app has a lot of components. While the switch is in progress, a special AppSwitchActivity is displayed to the user. This activity runs in a separate process, and it waits to receive a Intent.ACTION_PACKAGE_CHANGED] before allowing the transition to continue. This Intent signals that the package manager has finished updating with the app switch changes. If we don't wait for this signal to be received before switching to the new application, the OS might close that application once it receives that signal since it thinks the app has changed. AppSwitchActivity works around that by waiting for this signal, and then kicking off the launch of the new intended `Activity` once it has been processed.

Shipping Apps with Stuffing in Production

A more detailed walkthrough of how to integrate Stuffing into an existing app can be found in the documentation on Github. While these steps are relatively straightforward, there are many tricky edge cases that we had to deal with when using it to ship the new Snapchat app. While this list is far from being an exhaustive summary of a months-long road to releasing the new Snapchat app in production, it should give you a rough idea of the tricky problems that we had to solve:

Maintaining Launcher Icons

One of the first issues that we ran into was that the launcher icons for the app would disappear once the app family was switched. This happened because the launcher Activity of the old app was disabled along with other manifest components. In order to work around this issue, we used an Activity Alias. If your old activity class is com.company.app.FirstActivity, you can rename it to something else (ex: com.company.app.OldFirstActivity), and create an activity alias to link any existing launchers to the DynamicLaunchActivity. If you are doing this, make sure to remove the intent-filter from DynamicLaunchActivity.

Cost of Stuffing

Using Stuffing had a non-zero cost on the performance & app size of the application, but it was much lower than we originally anticipated:

• We optimized the DynamicLaunchActivity to do as little work as possible and it only added around 20 milliseconds to the app start up time

• We ended up sharing some of the framework level code between both apps, so the 

  stuffed

   version of the app was only 

  25mb

   bigger than the standalone new app.

AB Testing

The primary benefit of using Stuffing in Snapchat was that we could now properly AB test the new application in production. We checked for user’s eligibility for the AB test immediately after startup, and triggered app family switch via DynamicAppManager.

• Given that initial switch of app families could take 10+ seconds on older devices , we had to built an interstitial screen that let users know that they were being upgraded into Alpha test of the new app.

• After the app family switch was complete, the app had to be restarted. The latency of switching apps and the mandatory app restart caused some accuracy issues in our tests (especially for new users) that we had to work around and account for in our analysis.

• When we started testing the new app, it lacked various features that were still in development. To account for this, we provided a way for users to opt out from the test via an option Settings, but we eventually removed it as we got closer to our release candidate.

• Shortly after we kicked off our initial production test, the existence of the Snapchat Alpha app in the Android community online. The instructions for enabling the app required rooting your Android device. We didn't want to encourage our users to root their phones, so we decided to put an easter egg for a way to enable a new app for the curious hackers out there. We put a ghost on Snap Map near Alpha island in Bermuda that triggered an opt in screen. It only took a few days for this easter egg to be discovered.

Data Migration

When switching the app to the new family, we had to build special logic for carrying over some of the data from the old application:

1. The user session information

    - to avoid logging users out

2. In-progress background operations - 

   such as Memories backups to avoid losing user data.

3. Cached downloaded content - 

   to avoid re-downloading large amounts of data immediately after switching apps

While (1) and (2) were strong product requirements, (3) proved essential through AB testing.

Supporting All Entry Points

Launcher icon is not the only entry point into a typical Android app. An app could be launched via other Intents such as as ACTION_SEND for sharing or PendingIntent from a notification. In Snapchat, we had to ensure that all entry points into the app properly handle app switching to a new family, and that critical Intent data is carried over across the app switch boundary when necessary.

Removing the Old App

Once AB test was rolled out to 100%, we wanted to remove the old code as quickly as possible and reduce the performance & app size impact of the stuffed APK. Unfortunately, we couldn’t simply remove Stuffing completely from the codebase. While an APK without Stuffing would work well for any new installs, we had to support a number of backwards compatibility cases:

• App update. Logged out user state.

• App update from a recent version. Logged in user state. New app already enabled.

• App update from a recent version. Logged in user state. New app already disabled.

• App update from an old version (pre 

  Stuffing

  ). Logged in user state. New app never enabled.

The first 2 cases were relatively easy to support. The last two proved particularly tricky, and Stuffing was essential to making it work. While the old code was removed, we maintained a thin version of Stuffing in the codebase, which was a responsible for the following:

• If the new app was disabled (user opted out), we had to re-enable all of the manifest components.

• When switching from an old version, we still had to migrate any of the critical data (as discussed above).

• The activity alias to the dynamic launch activity had to be maintained to ensure that the launcher icon is not lost.

Conclusion

If there is one thing that we would like for you to take away from this post, it’s that, while it’s technically possible to ship an Android app with two virtually different apps stuffed in one, it’s not a path that one should choose without evaluating the tradeoffs. Shipping two apps in one empowers you to do proper AB testing, which is critical for large apps such as Snapchat, but it also introduces a large number of edge cases that you need to account for. The Stuffing library played a critical role in helping us rollout the new Snapchat Android app with no regressions in any of our critical metrics. We are very excited to open source it as a part of this post, and we are looking forward to your contributions! If you’re interested in solving hard technical problems that Android engineers at Snap face every day, we are hiring!