Occam's Razor is a principle that is frequently used in various fields to determine the most likely explanation for a given phenomenon.
The principle states that entities should not be multiplied beyond necessity and that the simplest explanation that fits the data is usually the correct one. But is there a way to justify this principle mathematically?
In this post, I will argue that Occam's Razor can be justified by Bayesian probability theory, which is a mathematical framework for updating beliefs based on new evidence. I will define Occam's Razor in terms of Bayesian probability theory, and I will show that the principle follows naturally from the basic axioms of probability theory.
First, let's define Occam's Razor more precisely. We can think of Occam's Razor as a principle of parsimony: the simplest explanation that fits the data is usually the correct one. But what do we mean by "simplest"? We can define simplicity in terms of the number of entities postulated by a given hypothesis.
For example, a hypothesis that postulates the existence of only one entity (e.g. a single force or a single cause) is simpler than a hypothesis that postulates the existence of multiple entities (e.g. multiple forces or multiple causes).
We can formalize this idea by defining a "hypothesis space" - that is, the set of all possible explanations that could explain a particular phenomenon. For example, the hypothesis space for the motion of a ball might include various explanations such as "the ball is being pushed by a force," "the ball is rolling down a hill," "the ball is being blown by the wind," and so on.
We can then assign a "prior probability" to each hypothesis in the space - that is, the probability that a given hypothesis is true before any new evidence is considered.
Now, suppose we observe some data (e.g. we measure the position and velocity of the ball at various times). We can then use Bayesian probability theory to update our beliefs about which hypothesis is true.
Bayesian probability theory tells us to update our prior probabilities in light of the new evidence, using a "likelihood function" that measures the probability of observing the data given a particular hypothesis. The likelihood function depends on the details of the data and the particular hypothesis under consideration.
We can then use Bayes' theorem to calculate the "posterior probability" of each hypothesis - that is, the probability that the hypothesis is true given the data we have observed. Bayes' theorem tells us that the posterior probability of a hypothesis is proportional to the prior probability of the hypothesis multiplied by the likelihood of the data given the hypothesis.
So, how does Occam's Razor come into play? Suppose we have two competing hypotheses that are equally likely a priori. That is, we have no reason to prefer one hypothesis over the other before we see any data. We can then compare the posterior probabilities of the two hypotheses after we have observed the data. If one hypothesis has a higher posterior probability than the other, then we have evidence in favor of that hypothesis.
Related Posts
Now, suppose one of the hypotheses is simpler than the other. We can then show that the simpler hypothesis will generally have a higher posterior probability than the more complex hypothesis, assuming that both hypotheses are consistent with the data.
This follows from the fact that simpler hypotheses are more flexible and can fit a wider range of data. In contrast, more complex hypotheses are more specific and can only fit a narrower range of data. As a result, the simpler hypothesis will have a higher likelihood than the more complex hypothesis for a wider range of possible data sets.
We can quantify this effect using a "Bayes factor," which measures the strength of the evidence in favor of one hypothesis over another, and are commonly used in Bayesian inference to quantify the support for different hypotheses. The Bayes factor is calculated by dividing the likelihood of the data under one hypothesis by the likelihood of the data under another hypothesis, and then weighting the result by the prior probabilities of the hypotheses.