# Rollout, feature flag, experiment, operational toggle Different use cases for **backend**, **frontend** and **mobile** --- "Feature flags" tend to come up when talking about **continuous deployment** ??? I'm using "quotes" because I'm mixing up different meanings of "rollout" --- # CI continuous integration # CD continuous delivery # CD **continuous deployment** ??? Background: build vocabulary, why are feature flags related to CD CI solves: manual integration of long-lived branches CD solves: automation of deployment process CD solves: releases as frequent as possible That's where the "GoCD" name comes from --- # Types: 1. rollout 2. feature flag 3. experiment 4. operational toggle --- # rollout ## For *rolling out* a new version of software **Short-lived** using **percentages** - a [new deployment of k8s][k8s] - new [APK released to the Play Store][apk] [k8s]: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#creating-a-deployment [apk]: https://support.google.com/googleplay/android-developer/answer/6346149?hl=en ??? Relevant as long as the new code is deployed --- # feature flag ## For turning a feature *on* or *off* **Medium-lived** using **allow list**, **A/B test**, **percentage**, **app version**, *etc*. - `:new-chargeback-flow` - `:new-debit-card-activation-screen` ??? Relevant as long as the new code is being developed --- # experiment ## For analyzing behaviour **Medium-lived** using **allow list** and **A/B test** - `:debit-withdrawal-test` --- # operational toggle ## For disabling features in `#crash`-like situations **Long-lived** using **percentage** - `:bank-barcode-payment` - `:savings-bank-barcode-query-provider` ??? Lives for as long as the code is in production. It feels like a system-level circuit breaker. --- We now know about the types ## But they have different relevance for **backend**, **frontend** and **mobile** --- # backend 1. **rollout**: k8s blue/green, canary and `common-rollout` 2. **feature flag**: `common-rollout` and datasets 3. **experiment**: `common-xp` 4. **operational toggle**: `common-rollout` ??? This is a bit why common-rollout isn't called *common-feature-flag*: it was initially designed with backend usage of mostly *rollouts* in mind, and just a bit *feature flags*. Avoid using configuration for doing operational toggles: it is less dynamic, so it defeats the purpose. --- # frontend 1. **rollout**: CDN and page refreshes 2. **feature flag**: percentages and maybe IPs (no `:customer/id` on the website) 3. **experiment**: via dynamic backend control 4. **operational toggle**: via dynamic backend control --- # mobile 1. **rollout**: app stores 2. **feature flag**: via dynamic backend control 3. **experiment**: via dynamic backend control 4. **operational toggle**: via dynamic backend control --- Key differentiator is ## How much **control** we have over the **environment** --- ## **backend** # Full control 🎉 ??? Can edit, update and even delete rollouts as desired. Mix and match at will! --- ## **frontend** # Partial control When choose when to make a new version available ??? We can control when a new version is available, partially when someone will upgrade it. But it is easy to fallback to "reload the page and try again". --- ## **mobile** # Very limited control - app stores can restrict updates (worse for iOS) - customers still have to download new versions --- # Costs - more complex code - compatibility with old app versions - nesting is exponential --- # Benefits - dynamicity --- ## Weighting costs × benefits The less control we have, the more we value dynamicity --- ## Weighting costs × benefits - backend: sometimes worth the cost - frontend: almost always worth cost - mobile: **always** worth cost --- # Best practices --- ## Dynamic content > feature flag Always true for **mobile**, almost always for **frontend** --- ## Use `:include-list` for named groups Always true for **backend**, **frontend** and **mobile** ```clojure [2-3] {:rules #{{:type :include-list :content {:filename "debit-team-members.txt"}}}} ``` --- ## Always use `:app-version` only for **mobile** ```clojure [2] {:rules #{{:type :app-version :content {:min-version #{{:platform :android :code 1000000} {:platform :ios :code 2000000}}}}}} ``` --- ## Extend `common-rollout` if required That's how `:include-list`, `:app-version`, *etc.* were born --- ## Beware of many nested feature flags True for **backend**, **frontend** and **mobile** ??? Exponential growth of combinations --- ## Don't delete app-facing feature flags True for **mobile** ??? This could break old app versions, only do this intentionally We don't have (yet) a strategy for dealing with LTS of the app, and we just say: "we'll support every app version out there". --- ## Include a feature flag on the whiteboarding phase --- ## Include deleting/retiring the feature flag at the end --- ## Avoid renaming a feature flag Use `:app-version` with `:min-version` instead --- # And most importantly... --- # ***Always*** rely on a feature flag on the app Never do a hot fix, avoid expedited releases at all costs ??? The app is where we have less control, so the feature flag is how we get some of that control back. This doesn't mean you'll need 1 feature flag per PR There's not such thing as: "This is such a small thing, it doesn't need a feature flag" You should ask yourself: "It this crashes the app, am I OK with waiting for the next release train?" --- ## Thank you! References: 1. these slides: [euandre.org/slides.html](/slides.html) 2. [prose version of this presentation](/2020/10/19/feature-flags-differences-between-backend-frontend-and-mobile.html) 3. "[Feature Toggles (aka Feature Flags)](https://martinfowler.com/articles/feature-toggles.html)", by Pete Hodgson 4. "[Continuous integration vs. continuous delivery vs. continuous deployment](https://www.atlassian.com/continuous-delivery/principles/continuous-integration-vs-delivery-vs-deployment)", by Sten Pittet 5. [Accelerate](https://itrevolution.com/book/accelerate/), by N. Forsgren, J. Humble and G. Kim