Why are you running A / B testing?
It's simple: see which version gets the most audience response.
Until recently, I thought there was only one kind A / B test. Then I met another version of it. This method is called Bayesian A / B testing, and if you want a more specific tactical approach to ad testing, this might be the answer.
It still includes variation checking to identify audience preferences, but it takes more computation and trial and error. But first, let's talk about how Bayesian A / B testing differs from traditional A / B testing. split tests.
Traditional A / B testing.
Frequentist is a standard split testing method that only uses data from the current experiment.
Everyone A / B test has the same number of components. They use data based on a metric that identifies options A and B. For example, a metric could be the number of clicks on an ad. To determine the winner, this indicator is measured statistically.
Let's apply this to an example using a frequency-based or traditional approach. In this script, you will create two declarations and change one variable, such as a copy of the declaration. Then select a metric, such as ad clicks.
In this example, the Frequent User A / B Test winner will be which ad clicked your target audience the most, based solely on the results of this experiment.
If you were to illustrate these components in a Bayesian A / B test, you would approach the test using different data.
What is a Bayesian A / B test?
The Bayesian approach takes information gathered from similar past experiments, combines it with current data, and draws a conclusion.
Basically, you would use the inference from previous Bayesian experiments as an option for a new test. This type of validation uses trial and error to create continuous tests until you find statistics to back up your desired results.
This definition may seem a little tricky to visualize without an example, so let's take a look at it.
If your previous Facebook ad had 867 unique visitors and 360 conversions for a 41% conversion rate, you would use that data to tell you what you expected.
If you were to assume that your next Facebook ad reached 5,000 unique visitors, you could conclude that, based on your previous experience, you would get 2,050 conversions. This will be option "A".
Let's say you're looking at the performance of a similar Facebook ad and you end up with a conversion rate of 52%. This is option "B."
What you did by collecting the data from the two options was to compute the posterior distribution, and the previous tests you ran are now the basis for your Bayesian test.
If you had conclusions about the conversion rates for each variable before calculating the posterior distribution, you can now update them based on the collected data.
You can ask hypothetical questions about your test, such as "How likely is" B "to be greater than option" A "?" In this case, you can conclude that the answer to this question is 9%.
Then some trial and error begins.
Bayesian methodology makes decisions by drawing some conclusions. You can calculate the expected loss by the rate of decrease in your metric when you select any variable.
Set a border, for example 2%, so that the metric goes lower. Once you have collected enough data to confirm that a variant has dropped below 2%, you will have a test winner.
Your estimated loss for a variation is the average of what your metric will decrease by if you choose that variation. The border should be small enough. This is necessary so that one can confidently suggest making such a big mistake.
The methodology assumes that you are more willing to make a mistake in a certain amount, and then move on to a more subtle experiment, instead of wasting time on an error that has fallen below that threshold.
If you run two experiments, they will stop when the expected losses are below this 4%-th limit. You should use the values of your options to calculate the average loss. You will then start testing again using these values as the wealth distribution.
Bayesian A / B testing proves that you can make a business decision that does not fall below the boundaries you set.
You can use the collected data to run tests continuously until you see an increase in metrics with each experiment.
When you use Bayesian testing, you can periodically change the test and improve the results as the test runs. Bayesian A / B testing uses constant innovation to give you concrete results, making small incremental improvements. You don't need to use the output as a result, but instead use it as an option.
If you run split tests in software or other channels, you don't need to change them to run Bayesian A / B tests.
Instead, you can look at the tools at your disposal in this software to get more calculated results. Then constantly run these tests and analyze them to choose your winners.
You can use a Bayesian A / B test instead of a traditional A / B test if you want to include more metrics in your findings. This is a really good test for calculating a more specific ROI on your ad. Of course, if you have less time, you can always use the frequent approach to get more of the "big picture".
Whichever method you choose, A / B testing is popular because it gives you insights that you might find useful in future campaigns.
Based on materials from the site: https://blog.hubspot.com.
❤️ Почему этот тест называется "Баесовским"?
It works according to a formula developed by the English mathematician, Thomas Baes.
✔️ What can help you do more accurate Beaesian A / B testing?
Read about beta distribution and Thompson sampling to determine previous A and B values.
✔️ What is the probability distribution in a Bayesian split test?
It is a set of possible parameter values along with a function that tells how likely any particular parameter is.
❤️ Why is Bayesian A / B testing important?
It takes into account the results of past checks and probabilities in the analysis, which makes the results of the new check more accurate.
❤️ What is Bayesian A / B Testing?
While accepting options that offer little improvement, Bayesian A / B testing argues that the false positive rate - the proportion of times we accept treatment when the treatment is not really better - is not very important.
❤️ What does Bayesian approach mean?
An approach to data analysis that provides a posterior probability distribution for some parameter (eg, a treatment effect) derived from the observed data, and a prior probability distribution for the parameter. The posterior distribution serves as the basis for statistical inference.
❤️ Where is Bayesian statistics used?
In any area of the application where you have a lot of heterogeneous or noisy data, or where you need a clear understanding of your uncertainty, you can use Bayesian statistics.
