Test Automation with Feature Flags

A basic introduction to using Feature Flags with Test Automation and FeatureHub.io— open source feature flag management and experimentation platform

Feature Flags are becoming more and more popular as they have significant advantages to the business and to the software development teams as a whole. Feature flags help to facilitate the following:

• Allow teams to decouple release from deployment. Product owners can decide when they want new features to be turned on with one click of a button
• Feature flags is a key enabler for trunk based development. They can be used to avoid large risky code merges and long lived branches
• Allow large numbers of developers to work on monolithic and service based applications across the business continually deploying yet not impacting each other

But in adding considerable value to the development teams, what are the trade offs?

One of the most obvious problems that teams encounter with feature flags is test automation. Feature toggles introduce more complexity for testing because each additional feature is a new possible combination of tests that need to be written.

Let’s imagine we have a simple “Todo” app and a feature flag that sets todo items to “upper case” if the toggle is turned on.

If you look at the example below — there are 2 possible paths that you will need to test. First one is when the flag is on, it will “upper case” the list of Todos. And second one is when it is off, it will preserve the case they are entered in.

So in a simple case we would have the following 2 scenarios to cover the above paths. Let’s use Cucumber BDD format to describe them:

As you may have noticed there are couple of things that are missing:

• You don’t actually know which feature flag you are testing here
• We haven’t indicated what the feature flag state needs to be for each scenario

To solve this problem, let’s add an extra step. FeatureHub comes with a Test API in each of the SDKs that allows you to set the values of features as long as your SDK API Key has permission to do so.

To make this work, go to the FeatureHub Admin Console and ensure you have an SDK API Key that has “Change Value” permissions for your test environment.

One extremely valuable feature of FeatureHub is that you can set a flag into “Lock” state — which adds extra security to your flags and prevents anyone from changing its status. Thus, don’t forget to set your Service account to have “Lock” and “Unlock” permission so it can ensure the feature is unlocked before trying to change it.

Once you have the permissions sorted, got to Service Accounts page and copy the Client eval API Key for an environment you are going to test on:

Given you have a feature flag setup (and implemented in your app), for example:

Next let’s add a new step to the test scenarios that will control feature state from the tests:

And the feature FEATURE_TITLE_UPPERCASE is off

And

And the feature FEATURE_TITLE_UPPERCASE is on

This will tell our scenario to ensure the feature is on or off depending on which one we are using.

Next we need to implement this step. In this example we will use Java, and it also depends on which Java HTTP client library you are using, so we will give a generic solution for it. If you look at our full example you can see this step in detail. The essence of the step is:

• Decode the feature and the on/off state you want
• Ask the Test SDK API to request the server set your feature to the desired state
• Wait for the local repository of features to reflect the change. This means the server accepted the change, stored it in the database and then rebroadcast the update to everything that listens to it.

And that is it!

For the full example please refer to this repo .

Don’t forget that with FeatureHub you can also use non-boolean flags, e.g strings, numbers or JSON.

Summary

You can see it is pretty simple to get up and running with test automation and feature flags using an open source platform like FeatureHub. The demonstrated approach is quite common, however things may get more complicated when you need to run tests in parallel that require certain feature toggle states. We will cover this approach in our next article and demonstrate the usage of feature overrides in microservice architecture.

If you would like to learn more about FeatureHub platform and how to set it up from scratch, please read this article: Manage feature flags for free with FeatureHub. Or find us on GitHub here.