---
title: Rollout, feature flag, experiment, operational toggle
date: 2020-10-10
layout: slides
lang: en
ref: rollout-feature-flag-experiment-operational-toggle
---
# Rollout, feature flag, experiment, operational toggle
Different use cases for **backend**, **frontend** and **mobile**
---
"Feature flags" tend to come up when talking about **continuous deployment**
???
I'm using "quotes" because I'm mixing up different meanings of "rollout"
---
# CI
continuous integration
# CD
continuous delivery
# CD
**continuous deployment**
???
Background: build vocabulary, why are feature flags related to CD
CI solves: manual integration of long-lived branches
CD solves: automation of deployment process
CD solves: releases as frequent as possible
That's where the "GoCD" name comes from
---
# Types:
1. rollout
2. feature flag
3. experiment
4. operational toggle
---
# rollout
## For *rolling out* a new version of software
**Short-lived** using **percentages**
- a [new deployment of k8s][k8s]
- new [APK released to the Play Store][apk]
[k8s]: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#creating-a-deployment
[apk]: https://support.google.com/googleplay/android-developer/answer/6346149?hl=en
???
Relevant as long as the new code is deployed
---
# feature flag
## For turning a feature *on* or *off*
**Medium-lived** using **allow list**, **A/B test**, **percentage**,
**app version**, *etc*.
- `:new-chargeback-flow`
- `:new-debit-card-activation-screen`
???
Relevant as long as the new code is being developed
---
# experiment
## For analyzing behaviour
**Medium-lived** using **allow list** and **A/B test**
- `:debit-withdrawal-test`
---
# operational toggle
## For disabling features in `#crash`-like situations
**Long-lived** using **percentage**
- `:bank-barcode-payment`
- `:savings-bank-barcode-query-provider`
???
Lives for as long as the code is in production.
---
We now know about the types
## But they have different relevance for **backend**, **frontend** and **mobile**
---
# backend
1. **rollout**: k8s, canary pipeline and `common-rollout`
2. **feature flag**: `common-rollout` and datasets
3. **operational toggle**: `common-rollout`
4. **experiment**: `common-xp`
???
rollout: k8s, common-rollout
This is a bit why common-rollout isn't called *common-feature-flag*: it was
initially designed with backend usage of *rollouts* in mind.
feature flag:
---
# frontend
1. **rollout**: CDN and page refreshes
2. **feature flag**: percentages and maybe IPs (no `:customer/id` on [www.nubank.com.br](www.nubank.com.br))
3. **operational toggle**: via dynamic backend control
4. **experiment**: via dynamic backend control
---
# mobile
1. **rollout**: app stores
2. **feature flag**: via dynamic backend control
3. **operational toggle**: via dynamic backend control
4. **experiment**: via dynamic backend control
---
Key differentiator is
## How much **control** we have over the **environment**
---
## **backend**
# Full control
🎉
???
Can edit, update and even delete rollouts as desired.
Mix and match at will!
---
## **frontend**
# Partial control
When choose when to make a new version available
???
We can control when a new version is available, partially when someone will
upgrade it.
But it is easy to fallback to "reload the page and try again".
---
## **mobile**
# Very limited control
- app stores can restrict updates (worse for iOS)
- customers still have to download new versions
---
# Costs
- more complex code
- nested flags combine exponentially
---
# Benefits
- dynamically choose code paths for each customer
---
# Best practices
---
## Dynamic content > feature flag
Always true for **mobile**, almost always for **frontend**
---
## Use `:include-list` for named groups
Always true for **backend**, **frontend** and **mobile**
{% raw %}
```clojure [2-3]
{:rules
#{{:type :include-list
:content {:filename "debit-team-members.txt"}}}}
```
{% endraw %}
---
## Always use `:app-version`
only for **mobile**
{% raw %}
```clojure [2]
{:rules
#{{:type :app-version
:content {:min-version #{{:platform :android
:code 1000000}
{:platform :ios
:code 2000000}}}}}}
```
{% endraw %}
---
## Extend `common-rollout` if required
That's how `:include-list`, `:app-version`, *etc.* were born
---
## Beware of many nested feature flags
True for **backend**, **frontend** and **mobile**
???
Exponential growth of combinations
---
## Don't delete app-facing feature flags
True for **mobile**
???
This could break old app versions, only do this intentionally
We don't have (yet) a strategy for dealing with LTS of the app, and we just say:
"we'll support every app version out there".
---
## Include a feature flag on the whiteboarding phase
---
## Include deleting/retiring the feature flag at the end
---
## Avoid renaming a feature flag
Use `:app-version` with `:min-version` instead
---
# And most importantly...
---
# ***Always*** rely on a feature flag on the app
Hotfixes and expedited releases is a thing of the past
???
The app is where we have less control, so the feature flag is how we get some of
that control back.
This doesn't mean you'll need 1 feature flag per PR
There's not such thing as:
"This is such a small thing, it doesn't need a feature flag"
You should ask yourself:
"It this crashes the app, am I OK with waiting for the next release train?"
---
## Thank you!
References:
1. these slides: https://euandre.org/slides.html
2. [prose version of this presentation]({% link _articles/2020-10-10-feature-flags-differences-between-backend-frontent-and-mobile.md %})
3. ["Feature Toggles (aka Feature Flags)"](https://martinfowler.com/articles/feature-toggles.html),
by Pete Hodgson
4. [Continuous integration vs. continuous delivery vs. continuous deployment](https://www.atlassian.com/continuous-delivery/principles/continuous-integration-vs-delivery-vs-deployment),
by Sten Pittet
5. [Accelerate](https://itrevolution.com/book/accelerate/),
by N. Forsgren, J. Humble and G. Kim
