quality

Working with Legacy Testing Frameworks

Bringing twenty some odd years of software quality assurance experience and leading test automation teams to the role Head of QA at Novoda.

Automated_test_suite

As projects mature, they pick up a lot of history.
This can lead to working with legacy integration test frameworks, which can over time become slow and flaky. After seeing a few of these we've have noticed the same anti-patterns emerge. Let’s have a look at those and how we can correct them by using "good practice" pattern.


Sullied checks

One antipattern to investigate is putting everything in the checks (with maybe some helpers around). Specifically the antipattern, where testdata, locators and flows can be found within the checks.

@Rule
public ActivityTestRule mActivityRule = new ActivityTestRule<>(
       MainActivity.class);

@Test
public void signIn_ValidCredentials_loginPossible() {
   onView(withId(R.id.main_activity_sign_in_button)).perform(click());
   onView(withId(R.id.sign_in_activity_username_field)).perform(typeText("Username"));
   onView(withId(R.id.sign_in_activity_password_field)).perform(typeText("Password"));
   onView(withId(R.id.sign_in_activity_submit_button)).perform(click());
   onView(withId(R.id.main_activity_sign_in_button)).check(matches(isDisplayed()));
}

Setting up checks this way leads to the problem that it will be hard to scale those. When you think about maintainability and extensibility, it is a worrisome trend that engineers are building their checks in this manner. Because of the code duplication maintaining will eventually become a tough task which means that you'll end in the maintenance hell.

Depending on the complexity of the software that you're testing it also can be complicated to understand what the check is doing in detail. The reality is that, while you do have checks running, they become either flaky or take ages.

Eventually, those checks get disabled or ignored, which makes all the effort you put in not to have been worth it.

What to do?

First off - you'll need to identify the problematic point:

  • Test data generation is mixed with the execution.
  • Description of long flows.
  • Locators are put into the check file and often duplicated.
  • Magic Numbers can be found in the code.
  • Unclear arrange, act and assert.

Once you've identified specific problems in the legacy test code, try to isolate those and refactor your Framework. What follows are some examples and some approaches for rectifying.

Extract Magic code

One possibility would be to create constants representing the test data. By doing this, you could already remove some duplication and make it more transparent for other developers to read and understand.

@Test
public void signIn_ValidCredentials_loginPossible() {
   private String username = “Username”;
   private String password = “Password”;

   onView(withId(R.id.main_activity_sign_in_button)).perform(click());
   onView(withId(R.id.sign_in_activity_username_field)).perform(typeText(username));
   onView(withId(R.id.sign_in_activity_password_field)).perform(typeText(password));
   onView(withId(R.id.sign_in_activity_submit_button)).perform(click());
   onView(withId(R.id.main_activity_sign_in_button)).check(matches(isDisplayed()));
}

As we're already working with the test data, a next good step could be creating a non-primitive data type and fixtures which could help you to create a domain specific language in your framework. However keep in mind that it easy to set up, but it can get very messy if there is a lot of data needed to execute the checks.

public class User {
   private String username;
   private String password;

   public User(String username, String password) {
       this.username = username;
       this.password = password;
   }

   public String getUsername() {
       return username;
   }

   public String getPassword() {
       return password;
   }
}


@Test
public void signIn_ValidCredentials_loginPossible() {
   User user = new User(“Username”, “Password”);

   onView(withId(R.id.main_activity_sign_in_button)).perform(click());
   onView(withId(R.id.sign_in_activity_username_field)).perform(typeText(user.getUsername()));
   onView(withId(R.id.sign_in_activity_password_field)).perform(typeText(user.getPassword));
   onView(withId(R.id.sign_in_activity_submit_button)).perform(click());
   onView(withId(R.id.main_activity_sign_in_button)).check(matches(isDisplayed()));
}

A different approach could be creating data provider for the data, which you could reuse in other contexts. This method is a bit more complicated to set up but is better if you have a lot of data.

@Test
public void signIn_ValidCredentials_loginPossible() {
   UserRepository repository = new UserRepository();
   User user = repository.getDefaultUser();
  
   onView(withId(R.id.main_activity_sign_in_button)).perform(click());
   onView(withId(R.id.sign_in_activity_username_field)).perform(typeText(user.getUsername()));
   onView(withId(R.id.sign_in_activity_password_field)).perform(typeText(user.getPassword()));
   onView(withId(R.id.sign_in_activity_submit_button)).perform(click());
   onView(withId(R.id.main_activity_sign_in_button)).check(matches(isDisplayed()));
}

public class UserRepository {
   private static String DEFAULT_USER_NAME = "Username";
   private static String DEFAULT_PASSWORD = "Password";
   private static String WRONG_PASSWORD = "12"

   public User getDefaultUser() {
       return new User(DEFAULT_USER_NAME, DEFAULT_PASSWORD);
   }

   public User getUserWithWrongPassword() {
       return new User(DEFAULT_USER_NAME, WRONG_PASSWORD);
   }
}

You should decide which one of these methods is the best approach depending on your needs. A good rule of thumb will be to use fixtures if you have maximum two or three sets of data. If it becomes more or if you think it might grow you should opt-in for the second option.

Page Objects

A next step could be creating page objects which represent your application and extract those into classes. This method helps you again to reduce duplicated code. It is one of the most used and sometimes unfortunately misused patterns in the industry. The Page Objects should only contain the locators and methods to access them. Otherwise, they become bloated, and you may create an undesirable god class.

public class MainPage {

   private ViewInteraction SIGN_IN_BUTTON = onView(withId(R.id.main_activity_sign_in_button));

   public void openSignInPage() {
       SIGN_IN_BUTTON.perform(click());
   }

   public void validateLoggedInStatus() {
       SIGN_IN_BUTTON.check(matches(isDisplayed()));
       SIGN_IN_BUTTON.check(matches(withText("Sign out")));
   }
  
}

By not putting the flows in the page objects - it creates a better maintainable piece of code, but it also pushes the responsibility of handling flows to the checks. By letting the checks describe all the flows, they can become quite large, which again leads to files which are hard to understand. One possible solution for this would be either to use the screenplay pattern or to create flows which are working as an interface between the checks and the page objects.

Flows

Flows are a form to organise activities performed in your checks into repeatable methods. They are combining different Page Objects and their respective parts into an understandable and human readable DSL which less experienced automation engineers can plug together. They can help you keeping the checks small and straightforward as well as having the flows to reach specific points of your application at one position in your code that changing this will be much easier.

public class LoginFlow {

   private MainPage mainView;
   private SignInPage signInPage;

   public LoginFlow() {
       mainView = new MainPage();
       signInPage = new SignInPage();
   }

   public void doLogin(Credentials credentials) {
       mainView.openSignInPage();
       signInPage.doLogin(credentials);
   }

   public boolean userIsLoggedIn() {
       mainView.validateLoggedInStatus();
       return true;
   }


   public boolean correctErrorDialogIsShown(String expectedError) {
       signInPage.validateErrorDialog(expectedError);
       return true;
   }
}

Conclusion

Setting up an automation solution is a simple task. However it can get quite messy in a short while when you do not consider extensibility and maintainability. You should always think about architecture and the dependencies of your classes. In the end automation is like every other engineering task easy to learn but hard to master and you should always keep improving.

To be fair, the vast majority of the input for this article came from Sven Kroell. Thanks Sven.

Enjoyed this article? There's more...

We send out a small, valuable newsletter with the best stories, app design & development resources every month.

No spam, no giving your data away, unsubscribe anytime.

About Novoda

We plan, design, and develop the world’s most desirable software products. Our team’s expertise helps brands like Sony, Motorola, Tesco, Channel4, BBC, and News Corp build fully customized Android devices or simply make their mobile experiences the best on the market. Since 2008, our full in-house teams work from London, Liverpool, Berlin, Barcelona, and NYC.

Let’s get in contact