We’ve all been there: In the midst of using an app a dialogue pops up asking us to stop everything we are doing and to leave a review on the Play Store.
This interruption is not only annoying but also not very effective as most people will simply dismiss the dialogue and continue what they are doing.

In this post we’ll explain how we implemented a prompt for feedback process for The Times & The Sunday Times Android app that helped boosting the average user rating from 2.8 stars to 4.3 stars, increased the average number of ratings per day by tenfold, and helped create a dialogue with unhappy users.

The importance of app ratings

Before we dive into how it was done, let’s talk a bit about the why.
Simply put, ratings are important because potential users view them as a way to indicate if an app is good or bad. Users are more likely to consider downloading a higher rated app than one with bad ratings.[1]
That aside, app ratings also influence the discoverability of an app. The most common way to discover apps - after general browsing through the store or referrals by friends - is by browsing the “top rated” and “most popular” sections of app stores.[2] Thus an app with 1 or 2 star-ratings is potentially losing out on new users.

The Times & The Sunday Times Android App

The problem

We worked with News UK to re-launch the new The Times & The Sunday Times app as an update instead of as a new, standalone app. This meant upon release we had an existing user base but also inherited a play store rating of 2.79 stars based on ~1,500 ratings. So our first challenge was to overcome this historic rating and improve it. With over 30,000 monthly active users and an average of six read articles per session, we knew that customers used the app regularly. Surveys and interviews also highlighted that the customers enjoyed using the app. Despite this positive feedback, we hadn’t received the ratings we hoped for. The average rating remained stable with only a slow upwards trend.

Our goals

We wanted to achieve these three overarching goals:

  • Improve The Times & The Sunday Times overall rating
  • Boost the average number of daily/monthly ratings
  • Gather customer feedback and start a dialogue with dissatisfied customers

A two-step prompt

The minimal changes of the average rating of the app was consistent with the general notion that ratings are usually left by a few vocal critics and a few vocal fans. This state leaves out the ‘silent majority’ of customers who use the app but don’t rate or give feedback. We set ourselves the goal of targeting active customers to rate the app or give us feedback for improvement. At the same time we wanted to learn how our frequently active users feel about using the app. After some consideration, we opted to introduce a two-step prompt:
In the first step we ask the customer if they enjoy using the app. If they do, in the next step we ask if they're happy to leave a review in the Play Store.
If the customer is not enjoying using the app, we instead ask they can give us some feedback through our in-app customer service tool.

Ask customers if they enjoy using the app before asking for feedback

A dark pattern?

It can be argued that directing users who are having negative experience to a funnel other than the Play Store or AppStore constitutes a 'dark pattern'. The reasoning behind this is that it deflects only potential negative ratings, leading to an uptick in ratings by default and at the cost of an 'honest' reflection of user feedback.
It's a solid argument, but we believe that this is not the case if the app is directing you to a better customer support avenue than the Play or App Store can provide.

To give an example: if a user is having a negative app experience because they believe the app is missing a feature they've simply not noticed, they may leave a 1 star review simply for lack of other options. While it's possible to respond to reviews on the Play Store, these often aren't seen by users nor is there a guarantee they'll update their rating once they realise their mistake. With our pattern, this user could be directed to a customer support representative able to quickly point out the feature or to an FAQ covering it. This, in our opinion, is a faster, better support experience for the user and an easier, more manageable support experience for the maintainers of the app. With the rise of direct, in-app customer support tools such as Intercom, it can also be very easy to implement. And, of course, users are still able of leaving negative reviews in the Play/App Store at any time.

What we did

Instead of integrating a dialogue which interrupts the customer while using the app, we thought about how customers are using the app, what constitutes an engaged user, and how we can communicate with them without being interrupting their experience.

The Times & The Sunday Times app is used by subscribers to consume news, magazine articles, and videos. Knowing what our customers use the app for, we set out to define what “engaged” means for us in this context. This would help us know from whom to solicit feedback. We decided to use two triggers that would classify a user as engaged.

Trigger 1 is activated after the customer opens the app for a certain number of days in a row.
Trigger 2 requires the customer to save 5 articles in the app.

After we'd defined these triggers, we had to pick a good moment to ask users who'd triggered them for their feedback.

Our assumption was that customers are more likely to leave feedback when they’ve completed the primary action for which they use the app. In this moment they should be in the best possible mental state to supply a rating: happy that they’ve been able to complete their task, uninterrupted, and open to new engagement with the app.

It was also vital that the prompt for feedback should not interrupt the user. No-one has ever enjoyed an unsolicited pop-up - most people will just dismiss them.

Our hypothesis was that a styled inline prompt asking whether the user was enjoying the app would lead happy users toward responding and ultimately toward leaving a review.
We also decided that asking the user whether they were enjoying the app, rather than directly asking them to give a rating, would soften the request. This would also give them an opportunity to reach out to us with our in-app customer service tool if they weren’t enjoying their experience. With the ‘silent majority’ in mindset we mentioned above, we believed they would feel relaxed and willing to leave us their otherwise unheard feedback.

The results

Impact of the prompt feedback dialogue

The impact of this prompt was fantastic. Five days after we'd release (when our first trigger started to be activated) we saw a huge spike in ratings. Even after this initial spike, the average number of ratings we received remained at around 7x-10x higher than previously.
The average rating we received also increased from 2.9 stars to 4.4 stars, while the number of 1 & 2 stars ratings dropped to an all-time low.

If you'd like to read a little more about our work on the Times & Sunday Times app, check out our case-study!

Quick facts: App feedback prompt

  • 10x more ratings per day
  • Boosted average rating by 1.5 stars
  • Halved 1 and 2 ratings**

  1. The Mobile Marketer's Guide to App Store Ratings & Reviews:
    http://cdn2.hubspot.net/hubfs/232559/The_Mobile_Marketers_Guide_To_App_Store_Ratings_and_Reviews.pdf?t=1470268162234 ↩︎

  2. Ibid. ↩︎