Toy 8 Modes Experiments Replication Problem

[mvpcom]

I used your code to replicate all experiments for comparision purpose. However, it is not possible to replicate any experiments related to toy 8 modes ("Some" not "all") in the same way that was shown in the paper. Did you use any specific parameter set? If yes would you please let me know how I can replicate similar results?

[renatobellotti]

I'm also experiencing problems. Did you get the plotting to work? I only get the following message and the timing, but cannot see any graph:

Javascript Error: IPython is not defined

[renatobellotti]

It works when you use Jupyter notebooks, but not when you use Jupyter lab.

@mvpcom I could get the correct plot. What was wrong for you?

[mvpcom]

@renatobellotti Oh no that's not my problem. I can run the code and I can achieve good results for the "all" experiment. But if you change the experiment from "all" to "some" (there is a flag for data generator that you can set), I can't reproduce the same results in the paper. I saw a video which they mentioned 5700 parameter update. It seems to me they used longer training time than me. But even with that change still no success to reproduce the same results.

[renatobellotti]

I've also seen the video.

What network did you use for the coefficient functions? The paper says it uses 3 Dense layers (not mentioned how many units in each) with ReLU activations "in the intermediate layers". I'm not sure if this means that the output layers of the coefficient networks don't have an activation...

In the demo notebook, however, they use the GLOWCouplingBlock that came out later (and 8 layers of this). According to the docstring, this means that the weights of s_1 are the same as those of t_1 (and the weights of s_2 are the same as those of t_2).

[mvpcom]

@renatobellotti I've used the same network in the notebook without any change but the data and number of epochs. Because of this, I've asked if they used any specific configuration to make sure this is not the problem.

[redraven984]

Try increasing the data size to 9600 - the maximum dimensions of a tensor, and then increasing the epochs. At 9600 and 500 epochs I get pretty close to the paper. They probably scaled it down because there is no way to tell what size someone's GPU might be.

[ardizzone]

I will transfer this issue to the paper's own repository. And I will try to get in touch with the person who made the toy experiments and the jupyter notebook to find out more!