Testing ideas for mobile apps

  • Updated
This topic describes how to:

Visitors may have measurably distinct preferences and use cases when it comes to mobile experiences. By testing on your app, you can optimize your app to drive engagement, retention, and other key metrics. Read on to learn more about key strategic themes to include in your mobile a/b testing program!

Focus on primary CTA (call to action)

What is the principle? In mobile apps, less is more. With limited screen real estate, focus user attention on a simple, clear, primary CTA.

How to shape your hypothesis: Simplifying the experience makes it easy for users to identify and engage with the primary task on the screen, driving adoption and loyalty. You can validate this in a few ways, for example:

  • Removing navigation links, search fields, and hero images —while appealing and functional, they may actually be distracting.
  • Affixing a CTA to the top/bottom of the screen as a floating element keeps the most important feature in view at all times.

Examples: Electronic Arts validated this principle with a landing page test that it ran on Optimizely Experimentation as part of its SimCity launch. The elimination of a promotional banner refocused user attention on the primary “Buy Now” CTA and product details, resulting in a 43.4% lift in game purchases.

Let us look at how the Eventbrite and Lyft mobile apps engage users to test this hypothesis.

Eventbrite features a prominent Connect with Facebook CTA in its events feed that enables users to find events that their friends are attending, providing social proof to users.

Eventbrite is more valuable to users who look to their social network to validate the value of events or who prefer attending events with friends. Affixing this CTA to the top of the screen when users scroll could drive social adoption for users who have not yet registered via Facebook.

 

 

Lyft has only two buttons affixed to the top of the screen: links to the menu and to the driver application. Lyft offers a promotion that awards $25 in ride credit for each referral who registers. This promotion is located in the menu, which many users may not explore during their first few uses of the app.

If Lyft’s primary success metric is to drive app use and encourage users to share their experiences, they could test replacing the steering wheel button in the header with a CTA for the $25 credit offerfor users who have not yet clicked on the referral link in the menu.

 

Process steps and guiders

What is the principle? Providing clear context along the user path makes reaching goals more simple and predictable. Guiders and progress bars provide instructional indicators that teach a user how to interact with app features and the steps involved to achieve a goal. 

How to shape your hypothesis: Even a task as straightforward as a form completion can be improved by including guiders to inform a user what information is required and if their information was recognized as valid inputs. Users often respond well to clearly marked steps because such a format instills a sense of achievement. Guiding users toward goals can drive retention and conversions.

Examples: On its lead generation landing page, Iron Mountain revised its form to more accurately communicate required inputs and next steps, resulting in a 140% improvement in lead quality.

QuizUp encourages repeat plays by consistently displaying user metrics. Other apps can benefit from showing markers to inform users how many steps they must complete before achieving their goal.

 

Fitmob users must consider a date, time, location, and type of workout before reserving their spot in a group class. The current interface achieves this in one screen view. Try testing a process flow to encourage users to use filtering options early in the funnel, which lets you serve a more personalized experience.

 

Hipmunk requires multiple steps before achieving the end goal. Users select departure and arrival flights, clearly numbered steps 1 and 2, but also have options to filter/save searches and enable fare alerts. Hipmunk may want to experiment with showing less functionality in one view, as users may miss one or more features that may be better highlighted in a process flow. 

Condensed vs. long scroll

What is the principle? Long scroll layouts have become a norm on websites and within apps, but that does not necessarily make them the optimal experience for every task. 

How to shape your hypothesis: Mobile users have adapted to longer scroll experiences for two key reasons:

  • Scrolling does not require users to locate a specific button or link to tap, which can be challenging on a small screen.
  • Users stay engaged on long scroll pages because they are not waiting for new page loads. It is a more fluid experience that enables users to easily discover new content and review that which they have already seen.

Consider how much information is appropriate to feature on the first page, using anchor links that scroll a user to appropriate sections below the fold. Try experimenting with gestures vs. scrolling and see which approach increases engagement.

Examples: The Hotel Tonight and Airbnb apps target similar users, yet present listing detail in very different ways. 

Hotel Tonight condenses listing detail to one page, focusing the experience on the image carousel and map link.

Airbnb opts for long scroll, revealing a story about the place, the host, and validating trust through guest reviews. 

Discovery functionality

What is the principle? Ecommerce apps have two fundamental use cases: product discovery and purchase. If your discovery phase is too long or cumbersome, you may be discouraging purchases.

How to shape your hypothesis: Purchase is dependent on discovery, so test shortening the discovery process. Enable users to browse multiple views of the same product without tapping. This should shorten the funnel and reduce friction in the purchase process.

Examples: Two examples of mobile apps that could take advantage of this test opportunity are Etsy and Zara. Both present visually appealing product layouts at the category level; however, users must tap through to the product page to view additional images. Etsy and Zara could try replacing taps with swiping or other gestures. 

 

Gilt offers more discovery functionality at the category level with an image slider. This feature is not immediately apparent and an appropriate test to run would be to add guiders in the form of arrows and/or markers to communicate to users that more than one image is accessible from this view.

Login timing and app tour carousels

What is the principle? Gating app content with logins or registration flows can potentially create a barrier to entry. Think through the optimal time to have a user log in.

How to shape your hypothesis: Too many upfront user inputs, even if optional, may add friction to the app adoption and feature exploration processes. Letting users interact with the app before entering personal information may increase engagement, because they will have had time to see your app's functionality.

Examples: Vine and Vimeo are examples of video sharing apps that gate all functionality to new users until they have registered. Both apps could make a preview video available to give users a "test drive" of the app before they commit to providing personal information. They could then measure registration rates as a result.

MixBit and YouTube target similar users and do not require upfront registration to encourage app exploration. MixBit is an example of a platform that educates new users about features through a carousel that is loaded the first time the user loads the app and can be revisited via the menu. MixBit users can begin exploring content immediately by accessing the main menu denoted by the marker in the top left, while YouTube users view the What to Watch screen immediately after opting to proceed through the app as a guest.

Another consideration in the above examples: Notice that both Vine and Vimeo use social login prompts. In addition to testing how much content is available before login or registration, you can also test using social login through Facebook, Google, or other services to remove the barriers to entry that come with manual logins. You can also test reducing the length of forms in your registration flow for the same effects. Both types of test can affect your engagement and retention metrics.