A/B Experiments and Version Control for Learning

A/B experiments and version control are the two approaches often used in software development and in marketing to test out a new product. The methodology behind these approaches can be very useful for learning and testing new patterns of behavior.

The basic idea is that before you deploy a new product or commit to a new program, it needs to be tested first. If something doesn’t work right or as expected you can always “roll back” to the previous version that worked good enough. We call this approach “backup branching” and consider it highly beneficial in the context of learning and physical practice, especially where overall stability and sustainability are very important.

This approach is used both by the giants of software industry like Google as well as small marketing firms who want to test a new sales funnel. It allows them to have their service constantly running but to also deliver frequent updates and to evolve, without breaking it apart.

For example, if Google wanted to change the color of its search results, they could make the necessary modifications in code and put it online. But if they made a simple mistake in code (of course, this is a very hypothetical situation) the whole service would go down and it would take ages to find out what happened. Instead, they can simply revert to the previous version (or “commit”) and while the stable version is running they can test what was wrong with the new one.

In the same way supposed somebody wants to try a better way of selling their stuff online. They make a new homepage for their website, adding some modifications to the previous one. But what if that change is actually worse? It would take time to roll back to the previous version and to find out what exactly went wrong. That’s where the A/B experiments come in. The basic approach is very simple: when implementing a change (let’s call it B) you are still delivering the previous version (let’s call it A) in 50% of cases. Using the feedback you get (data tracking, etc) it is possible to estimate, after some time, which approach works better for you. You can then settle for one of the options and continue developing through experimentation.

In the context of learning or physical practice it functions the same way:

1. Understand (“commit”) your current pattern of behavior (A) and make a backup;
2. Think of a variation (B) — a new pattern of behavior;
3. Keep the original intact and operational while trying the variation (create a “branch” of B) — implementation;
4. Get the feedback from both A and B, compare the results;
5. Deploy A or B (or “merge” the changes introduced by B into A);
6. Reiterate (experiment again — back to Step 1);

Artwork by Timm Ulrichs (2005) — “Wolf in a Sheepskin — Sheep in a Wolfskin” / “Wolf im Schafspelz – Schaf im Wolfspelz: Ein Verwandlungskunststück”