Content

First failed benchmark

Some BNN benchmark examples are applied to so-called trivial data. Under trivial we mean an elementary and most common cases when observed outputs or targets have a bell shape distribution density. Most publicly available data sets have this sort of distributions and building stochastic models for this kind of data is elementary, not challenging and can be accomplished in many different ways, including making ensembes by simple random initialization. The claimed advantage of BNN is recognition of non-trivial distributions of the outputs. In this experiment I simply took popular published BNN example and changed the data from trivial bell shape distribution into other.

Benchmark code and benchmark data

Here we consider benchmark example from the site specifically designed and dedicated to probabilistic models. Example: Bayesian Neural Network.

The training data is stochastic function of a single argument $y = F(x)$. The training record has 100 pairs, shown in the image below by 'x':


When library code is executed, it returns expectation model, shown as a solid blue line, and light blue area holding 90% confidence interval. It returns also reference to an object that generates Markov Chain Monte Carlo (MCMC) samples of the output for any provided input $x$. I generated multiple samples with 2000 returned values, passed them to a method building a histogram and obtained perfectly matched distribution of the outputs, below is one of them:


At this point I have to add that outputs were truly generated with normal distribution, it can be found in code, and the result perfectly matches the actual distibution. Here we say a 'good job' to the developers of software and try our own data.

Moving from ideal world into real one

Now we try to use this NUMPYRO example for the data that developer of this code have not seen. It is just a real life situation, not an attack. I only split data randomly into two blocks changing the distribution from unimodal into multimodal:


The expectation and confidence interval looks fine but the histogram is disappointing. It shows unimodal distribution for every point I tried. It appeared that mathematical software, for which authors introduced a special term 'probabilistic programming' did not recognize the basic property of the data that can be seen by the naked eye.


In case if referred site move or remove this example, it can be found in my repository github. The changes that I made to the code are properly commented, so they can be disabled and code may be executed for both original and modified data.