Content

Second failed benchmark
Published benchmarks examples, usually, have two components: code and data. The data, obviously, must be kind of challenging,
exposing code strength, but they are not always chosen that way. This example is taken from
Keras 
another big and well maintained collection of source code for deep machine learning.
The data used in this Keras example is Wine Quality.
It has 4 898 records with 11 observed features. The target is quality score 0,1,...10.
Since this data is experimental and only one output value is available for each record, the predicted
distribution can't be compared to actual one and accuracy can't be assessed. So, in order to assess accuracy we
replaced Wine Quality Data by synthetic
data:
where $C_j$ are uniformly distributed random values on $[0,1]$, $X_j$ are observed values, $X^*_j$
are used in computation of the outputs $y$, parameter $\delta$ is an error level. When $\delta=0$, the system becomes deterministic
and can be modelled with near 100% accuracy by neural network, so we are dealing with aleatoric uncertainty only.
The generated data set size was 10 000 records.
This formula was designed by mathematician Mike Poluektov. I call it Mike's benchmark data set. Recalculation of outputs
with different random terms $C_j$
allows estimation of probability densities for $y$ and they all are complex enough and depend on observed inputs $X_j$.
The code from Keras provides only
expectations and standard deviations. So I replaced experimental
Wine Quality Data and compared expectations and
standard deviations returned by model to so called 'actual' or
Monte Carlo simulated values.
The error level was $\delta = 0.8$. Single deterministic model for such data gives accuracy near $75\%$.
The accuracy for returned expectations was near $98\%$ and for standard deviations near $92\%$.
The used accuracy metric was Pearson correlation coefficient. The modified code can be found in my repository
backup location,
but it is almost same as original, only data is replaced and accuracy assessment is added.
Is it success or failure?
It may look like a good result. The data is very complex, single deterministic model is very inaccurate, correlation
for estimated $\hat{y}$ and given $y$ outputs for single model is only $75\%$, but BNN returns expectations with $98\%$ accuracy
and standard deviations with $92\%$ accuracy.
Ok, in the next section I will explain why this result is very weak.
'Vandalization' of the beautiful picture
My idea in counter example was to model variance by another deterministic model and compare accuracy to BNN. Assume we built single deterministic
expectation model $M_E$ by minimizing residual errors and it provides estimated outputs $\hat{y}$. Now we can compute squared errors for each
individual input $e_i = (M_E(X_i)  y_i)^2$ and build another model $M_V$ using $e_i$ as new targets.
The choice of the models $M_E, M_V$ should not necessarily be a neural network. I chose here the one I was using for several years in research
$$ M(x_1, x_2, x_3, ... , x_n) = \sum_{q=0}^{2n} \Phi_q\left(\sum_{p=1}^{n} \phi_{q,p}(x_{p})\right), $$
which is KolmogorovArnold representation. This model will be explained in details further in this site. For
this moment I only state that its accuracy in many comparison tests appeared to be near neural networks
but training needs much less time. The functions $\Phi_q, \phi_{q,p}$ are not specified prior to training, their
shapes is fully determined in the training process.
Now I simply announce the end result for KolmogorovArnold model. The accuracy for expectation is 98%
and for variance 92%. Two deterministic models did the same job. The code can be found
in my storage, the project name is residual.
Using two deterministic models for uncertainty estimation, obviously, is much more quick and reliable process. When technology
is significantly more complex, it should bring certain advantages, which I did not notice in this example.
Needless to say that training two deterministic models took few seconds and using BNN near two minutes on the
same machine.
 
