engineering

The (re)making of a Download Manager

Novoda has a reputation of building the most desirable apps for Android and iOS. We believe living and sharing a hack-and-tell culture is one way to maintain top-shelf quality.

Android allows developers to add download functionality to their apps, but if you want to extend that behaviour, or customise it to your liking — that’s not easy. That’s why we created our improved version of the Download Manager.

We started working on a fork of the AOSP Download Manager, but there were a number of factors that led us to create our brand new version of the download-manager. First of all, at the time of creation the standard AOSP library did not allow writing to private storage. Second, the nature of AOSP code can at times be confusing to maintain and add features to, which is particularly troublesome to new developers contributing to the project. The original download-manager from AOSP did not allow clients to implement batching of downloads, meaning that each downloading file would have had its independent notification. The lack of batching required hacks in client applications, that led to flickering notifications when quickly showing and hiding individual ones after each file was downloaded.

The new version of Download Manager hinges on the concept of downloading batches of files; now, multiple file requests can be grouped into a single download batch. With this approach, client apps can make it opaque to the end user that a download is composed of multiple separate items, instead having a single update notification displayed for the whole batch. Notifications are now also built into the library: Download Manager v2 intelligently shows notifications that are persistent, dismissible (stackable), or hidden, based on the download state.

Notifications can be fully customised, through the NotificationCustomizer interface. Client applications can customise what information to show in each notification, or hide entirely the notification for some specific status. An application, for example, might be interested in showing a notification only while a download is in progress, hiding all other notifications, while another might want to notify the user also when the download has failed, or is completed.

The original version of the download-manager made use of ContentResolvers and ContentProviders to provide access to their data, a sign of it’s time, this design pattern has fallen out of use, in favor of newer, more flexible, less boilerplate solutions. In the new version, this pattern has been replaced by a persistence layer based on Room. Now, all data is treated as private, except for what is part of the public API. In doing this, we improved how client applications can query the status of current and queued downloads: while the original download-manager relied on the BroadcastManager mechanism to dispatch updates, now there is a public API to fetch the status of all downloads, and to register a callback for updates to a DownloadBatchStatus. The frequency of these updates can be throttled through the use of the FileCallbackThrottle interface; time- and progress-based throttles are already built into the library, but client apps can define custom ones as well.

The client app can also specify constraints regarding the network connection to be used: downloads will start only when the device is connected to the desired network type (metered or unmetered), pausing and resuming automatically when the network status changes.

The old, AOSP-based, download-manager is used by several published applications and we thought about that when working on the new version. A migration interface is currently in development to ease the transition between the first version and the new, completely rewritten v2. In the meantime, it’s already possible to migrate the downloads handled by the old Download Manager into the v2, if you upgrade your app. To perform a migration a client should inform the library about completed and partial downloads. It’s possible to do so by providing a CompletedDownloadBatch and passing it to downloadManager.addCompletedBatch() or a VersionOnePartialDownloadBatch passing it to downloadManager.download().

It is possible to see the full migration flow in the demo application.

We originally had planned an automatic migration mechanism, but client applications had complete freedom to use the database columns in any way they wished, and this makes a "one size fits all" solution impossible to provide.

download_demo_2_framed-both-1

The demo app, showing how to use most of the features of the library

Architectural Design

While working on our fork of the AOSP Download Manager we faced several difficulties when we tried to fix bugs, extend its functionalities, add tests and in general with the readability and maintainability of the code. Based on that experience, we wanted to achieve the following:

  • Clear and good separation of concerns so that we have small classes with very a narrow focus
  • Prime performance by avoiding unnecessary disc operations as much as possible
  • Allow the library to be extended by the client apps
  • Use the tell don’t ask principle, so that classes expose the minimum amount of information and encapsulate all the logic in them
  • Use interface wherever possible to avoid leaking or depending on implementations
  • Provide common functionality out-of-the-box including notifications, recovery on network failure and migrations from the previous version of the library
  • Provide a demo application with good examples on how to use the library

download-manager-1

Hard lessons we learnt

A task like downloading a file looks very simple, but it is actually a very complex one to achieve properly. There are many things that can go wrong and many statuses to consider. Besides that, our requirement of downloading batches of files in bulk adds lots of complexity.

Writing to disk too fast might block your device, or make background tasks too slow. Too frequent callbacks might completely collapse the system. We have implemented a throttle for our download callback because otherwise having a callback that will end up with a UI notification every frame will completely grind the system and the host app to a halt. Updates on downloads progress are impacting the UI, and can happen very fast, very frequently, so it’s important to be careful.

Backward compatibility and data migrations. As a download library we wanted to provide some sort of continuity between the previous version of the library and the new one. It wasn’t acceptable to offer a new library that ignored any already downloaded (or queued) content by a previous version. Thus, even though we wrote this new version from scratch, we spent a considerable amount of time thinking, implementing and testing a migration process that means updating an app to the new version would not mean losing all the previously downloaded data.

Threading is complex, and robustness is important. It's easy to run a task in a thread and wait for its result. But it gets complex when you need to interact with a thread immediately and move on to the next task in a queue of tasks that are batched. For example, a file might be downloading in a background thread but the user might want to pause or delete it. Changes on statuses from background threads to the main thread and back can happen extremely fast. You have to be very careful with the flow of your states and how those statuses might change in the middle of an execution. Storing statuses in memory is faster than using disk persistence, but we had to consider what happens if the client app is suddenly shutting down. How are you going to recover the pre-crash state, and what’s considered an acceptable loss of data?

Testing is hard and the real world is harsh. We spent a considerable amount of time smashing buttons in the screen trying to make the demo application to crash and we succeeded way more often than we’d hoped. By now we are true pros at it! Even with the most rigorous development and testing regime, there will be many scenarios you didn’t think of that you need to go and fix. Some of them might have a considerable impact on your implementation. If you find that something is fundamentally wrong, you might have to go back to the drawing board, sketch all your system again and think a better way out.

Recovery is tricky. There will be many different reasons for which a download might fail. It could be a network issue, a server error, or your device has run out of space, or even more obscure errors such as out of memory and concurrency errors. Think of a clean way to recover your system when the library starts after a crash or a bug. We made the decision of not writing every single status change in the persistence layer, but we write the important events that will allow us to find out what went wrong and how to recover from that status.


If we managed to intrigue you, head to the official repository and give our new Download Manager a try!

Enjoyed this article? There's more...

We send out a small, valuable newsletter with the best stories, app design & development resources every month.

No spam, no giving your data away, unsubscribe anytime.

About Novoda

We plan, design, and develop the world’s most desirable software products. Our team’s expertise helps brands like Sony, Motorola, Tesco, Channel4, BBC, and News Corp build fully customized Android devices or simply make their mobile experiences the best on the market. Since 2008, our full in-house teams work from London, Liverpool, Berlin, Barcelona, and NYC.

Let’s get in contact