path: root/_slides/2020-10-19-rollout-feature-flag-experiment-operational-toggle.slides

---
title: Rollout, feature flag, experiment, operational toggle
date: 2020-10-19
layout: slides
lang: en
ref: rollout-feature-flag-experiment-operational-toggle
---

# Rollout, feature flag, experiment, operational toggle
Different use cases for **backend**, **frontend** and **mobile**

---

"Feature flags" tend to come up when talking about **continuous deployment**

???

I'm using "quotes" because I'm mixing up different meanings of "rollout"

---

# CI
continuous integration

# CD
continuous delivery

# CD
**continuous deployment**

???

Background: build vocabulary, why are feature flags related to CD

CI solves: manual integration of long-lived branches

CD solves: automation of deployment process

CD solves: releases as frequent as possible

That's where the "GoCD" name comes from

---

# Types:
1. rollout
2. feature flag
3. experiment
4. operational toggle

---

# rollout
## For *rolling out* a new version of software

**Short-lived** using **percentages**

- a [new deployment of k8s][k8s]
- new [APK released to the Play Store][apk]

[k8s]: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#creating-a-deployment
[apk]: https://support.google.com/googleplay/android-developer/answer/6346149?hl=en

???

Relevant as long as the new code is deployed

---

# feature flag
## For turning a feature *on* or *off*

**Medium-lived** using **allow list**, **A/B test**, **percentage**,
**app version**, *etc*.

- `:new-chargeback-flow`
- `:new-debit-card-activation-screen`

???

Relevant as long as the new code is being developed

---

# experiment
## For analyzing behaviour

**Medium-lived** using **allow list** and **A/B test**

- `:debit-withdrawal-test`

---

# operational toggle
## For disabling features in `#crash`-like situations

**Long-lived** using **percentage**

- `:bank-barcode-payment`
- `:savings-bank-barcode-query-provider`

???

Lives for as long as the code is in production.

It feels like a system-level circuit breaker.

---

We now know about the types

## But they have different relevance for **backend**, **frontend** and **mobile**

---

# backend

1. **rollout**: k8s blue/green, canary and `common-rollout`
2. **feature flag**: `common-rollout` and datasets
3. **experiment**: `common-xp`
4. **operational toggle**: `common-rollout`

???

This is a bit why common-rollout isn't called *common-feature-flag*: it was
initially designed with backend usage of mostly *rollouts* in mind, and just a
bit *feature flags*.

Avoid using configuration for doing operational toggles: it is less dynamic, so
it defeats the purpose.

---

# frontend

1. **rollout**: CDN and page refreshes
2. **feature flag**: percentages and maybe IPs (no `:customer/id` on the website)
3. **experiment**: via dynamic backend control
4. **operational toggle**: via dynamic backend control

---

# mobile

1. **rollout**: app stores
2. **feature flag**: via dynamic backend control
3. **experiment**: via dynamic backend control
4. **operational toggle**: via dynamic backend control

---

Key differentiator is
## How much **control** we have over the **environment**

---

## **backend**

# Full control
🎉

???

Can edit, update and even delete rollouts as desired.

Mix and match at will!

---

## **frontend**

# Partial control

When choose when to make a new version available

???

We can control when a new version is available, partially when someone will
upgrade it.

But it is easy to fallback to "reload the page and try again".

---

## **mobile**

# Very limited control

- app stores can restrict updates (worse for iOS)
- customers still have to download new versions

---

# Costs

- more complex code
- compatibility with old app versions
- nesting is exponential

---

# Benefits

- dynamicity

---

## Weighting costs × benefits

The less control we have, the more we value dynamicity

---

## Weighting costs × benefits

- backend: sometimes worth the cost
- frontend: almost always worth cost
- mobile: **always** worth cost

---

# Best practices

---

## Dynamic content > feature flag

Always true for **mobile**, almost always for **frontend**

---

## Use `:include-list` for named groups

Always true for **backend**, **frontend** and **mobile**

{% raw %}
```clojure [2-3]
{:rules
 #{{:type    :include-list
    :content {:filename "debit-team-members.txt"}}}}
```
{% endraw %}

---

## Always use `:app-version`

only for **mobile**

{% raw %}
```clojure [2]
{:rules
 #{{:type    :app-version
    :content {:min-version #{{:platform :android
                              :code     1000000}
                             {:platform :ios
                              :code     2000000}}}}}}
```
{% endraw %}

---

## Extend `common-rollout` if required

That's how `:include-list`, `:app-version`, *etc.* were born

---

## Beware of many nested feature flags

True for **backend**, **frontend** and **mobile**

???

Exponential growth of combinations

---

## Don't delete app-facing feature flags

True for **mobile**

???

This could break old app versions, only do this intentionally

We don't have (yet) a strategy for dealing with LTS of the app, and we just say:
"we'll support every app version out there".

---

## Include a feature flag on the whiteboarding phase

---

## Include deleting/retiring the feature flag at the end

---

## Avoid renaming a feature flag

Use `:app-version` with `:min-version` instead

---

# And most importantly...

---

# ***Always*** rely on a feature flag on the app

Never do a hot fix, avoid expedited releases at all costs

???

The app is where we have less control, so the feature flag is how we get some of
that control back.

This doesn't mean you'll need 1 feature flag per PR

There's not such thing as:
"This is such a small thing, it doesn't need a feature flag"

You should ask yourself:
"It this crashes the app, am I OK with waiting for the next release train?"

---

## Thank you!

References:

1. these slides: [{{ site.tld }}/slides.html]({% link slides.md %})
2. [prose version of this presentation]({% link _articles/2020-10-19-feature-flags-differences-between-backend-frontend-and-mobile.md %})
3. "[Feature Toggles (aka Feature Flags)](https://martinfowler.com/articles/feature-toggles.html)",
 by Pete Hodgson
4. "[Continuous integration vs. continuous delivery vs. continuous deployment](https://www.atlassian.com/continuous-delivery/principles/continuous-integration-vs-delivery-vs-deployment)",
 by Sten Pittet
5. [Accelerate](https://itrevolution.com/book/accelerate/),
 by N. Forsgren, J. Humble and G. Kim