Install issues on Arch Linux and Ubuntu Docker

[LSaldyt]

I understand Arch Linux isn't supported per the README, so this is just FYI. Maybe the ubuntu docker install is more concerning? I tried that as a backup, and it didn't work either.
I'm actively working on fixing this -- it's totally possible the issue is only my own, so if I find a solution, I'll make a PR.

TLDR:

I suspect either the boost version or setup.py for preventing installation on some systems. I think better tracking of required package versions could get around this.

Recall the Ubuntu Linux install instructions:

    sudo apt-get install -y libboost-all-dev libgsl0-dev ccache
    sudo apt-get install -y python-matplotlib  # If you don't already have it
    sudo apt-get install -y python-scipy       # If you don't already have it
    pip install venture-0.4.2.tar.gz

The Arch Linux equivalent is the following:

sudo pacman -S boost gsl ccache
pip2 install matplotlib scipy

I indeed have these packages:

$pacman -Q | egrep 'boost|gsl'
boost 1.67.0-4
boost-libs 1.67.0-4
gsl 2.5-1
$pip2 list | egrep 'matplotlib|scipy'
matplotlib                         2.2.2       
scipy                              1.1.0       
$ python2 --version
Python 2.7.15

Same thing with ubuntu docker image:

# apt list | egrep 'boost-all|libgsl'
libboost-all-dev/bionic,now 1.65.1.0ubuntu1 amd64 [installed]
libgsl-dev/bionic,now 2.4+dfsg-6 amd64 [installed]
# pip list | egrep 'matplotlib|scipy'
matplotlib (2.2.2)
scipy (1.1.0)
# python --version
Python 2.7.15rc1

They're higher versions than specified in conda/meta.yaml (which I'm referring to because there is no requirements.txt -- which is what pip would use), but this doesn't seem to be causing the problem (I'll try all downgraded versions next).

Here's the ubuntu case with the latest release:

# pip install v0.5.1.1.tar.gz 
Processing ./v0.5.1.1.tar.gz
    Complete output from command python setup.py egg_info:
    fatal: not a git repository (or any of the parent directories): .git
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-ZBvNtO-build/setup.py", line 85, in <module>
        pkg_version, full_version = get_version()
      File "/tmp/pip-ZBvNtO-build/setup.py", line 57, in get_version
        'git', 'describe', '--dirty', '--long', '--match', tag, '--tags',
      File "/usr/lib/python2.7/subprocess.py", line 223, in check_output
        raise CalledProcessError(retcode, cmd, output=output)
    subprocess.CalledProcessError: Command '['git', 'describe', '--dirty', '--long', '--match', 'release-0.5.1a20170816', '--tags']' returned non-zero exit status 128
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-ZBvNtO-build/

I've tried running this from inside the Venture repo, but this doesn't work.

I also tried the 4.2 release, which gives a multi-page template error, which I also get in the 5.1 case if I call python setup.py install manually. It's a huge error, but I think the relevant part is this:

backend/new_cxx/src/pyutils.cxx:23:10: fatal error: boost/python/numeric.hpp: No such file or directory
 #include <boost/python/numeric.hpp>
          ^~~~~~~~~~~~~~~~~~~~~~~~~~

However, this is already installed, so far as I can tell:

# apt-get install libboost-python-dev
libboost-python-dev is already the newest version (1.65.1.0ubuntu1).

Looks like others have run into this. I tried changing numeric.hpp to numpy.hpp, but maybe a downgrade of boost will work? I will let you know...

Anyway, all of that documentation is to say two important things: I suspect either the boost version or setup.py for preventing installation on some systems. I think better tracking of required package versions could get around this.

[LSaldyt]

I've tried changing the boost versions among other things..

I also get the following when trying to install an early version of venture (4.2):

$pip2 install venture-0.4.2.tar.gz
Installing collected packages: venture
  Running setup.py install for venture ... error
    Complete output from command /usr/bin/python2 -u -c "import setuptools, tokenize;__file__='/tmp/pip-req-build-LascNw/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-WhMfiR/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    generating python/lib/parser/church_prime/grammar.y -> python/lib/parser/church_prime/grammar.py
    /usr/bin/env PYTHONPATH=/tmp/pip-req-build-LascNw/external/lemonade/dist /tmp/pip-req-build-LascNw/external/lemonade/dist/bin/lemonade -s python/lib/parser/church_prime/grammar.y
    Traceback (most recent call last):
      File "/tmp/pip-req-build-LascNw/external/lemonade/dist/bin/lemonade", line 3, in <module>
        from lemonade.main import main
      File "/tmp/pip-req-build-LascNw/external/lemonade/dist/lemonade/main.py", line 6, in <module>
        from exceptions import *
    ModuleNotFoundError: No module named 'exceptions'
    error: command '/usr/bin/env' failed with exit status 1
    
    ----------------------------------------
Command "/usr/bin/python2 -u -c "import setuptools, tokenize;__file__='/tmp/pip-req-build-LascNw/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-WhMfiR/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-req-build-LascNw/

(And the exceptions module is indeed installed):

$python2
Python 2.7.15 (default, May  1 2018, 20:16:04) 
[GCC 7.3.1 20180406] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import exceptions
>>> 

Anyway, I'd appreciate some suggestions or an official list of dependencies past "apt-get install," since it seems quite a few things have progressed since Venture's inception.

Best,

Lucas

[fsaad]

Hi Lucas,

Thanks for the detailed report; the Venture build is known to be hard to run from scratch. It can be installed directly on ubuntu 16.04, but also via docker and conda.

The instructions for installing on a fresh Ubuntu 16.04 system are shown in https://github.com/probcomp/Venturecxx/blob/master/docker/ubuntu1604.

You can try the following to confirm that the dockerfile works as intended.

$ git clone git@gihtub.com:probcomp/Venturecxx
$ cd Venturecxx
$ docker build -f docker/ubuntu1604 -t LSaldyt/venture .

Many tests should run while building the image. After the image is built, you can obtain a python session with Venture inside the container x using:

$ docker run -it LSaldyt/venture /bin/bash
# ./pythenv.sh python
>>> import venture

[LSaldyt]

Great! I've confirmed that the above essentially works, and will let me start using/contributing to Venture. Still, I'd prefer an arch-native setup or a pip install. I'll take a look at that docker file and see if it helps me accomplish either of those. Anyway, I greatly appreciate your help, and consider this resolved. I'll send a pull request if I figure out alternate installs.

[fsaad]

@LSaldyt Okay, if you want a conda install see: https://github.com/probcomp/iventure/blob/master/docs/conda.md

For a pip-based install, see: https://github.com/probcomp/iventure/blob/master/docs/virtualenvs.md

The above instructions are tested assuming a fresh system of Ubuntu 16.04.

[LSaldyt]

Awesome, thank you so much for your help.

Feel free to disregard the following, I'm just putting it here for documentation.

As noted, if I follow the pip-based install on arch linux (actually, an antergos install of arch linux), then I get some issues that look like they stem from the customized version of the lemonade package:

Venturecxx$ python2 setup.py build
running build
running build_py
generating python/lib/parser/church_prime/grammar.y -> python/lib/parser/church_prime/grammar.py
/usr/bin/env PYTHONPATH=/home/flynn/projects/clones/Venturecxx/external/lemonade/dist /home/flynn/projects/clones/Venturecxx/external/lemonade/dist/bin/lemonade -s python/lib/parser/church_prime/grammar.y
Traceback (most recent call last):
  File "/home/flynn/projects/clones/Venturecxx/external/lemonade/dist/bin/lemonade", line 3, in <module>
    from lemonade.main import main
  File "/home/flynn/projects/clones/Venturecxx/external/lemonade/dist/lemonade/main.py", line 6, in <module>
    from exceptions import *
ModuleNotFoundError: No module named 'exceptions'

Still, I'm not really worried about it since I can just run Venture in docker (And all the examples/reference works fine, at least so far ;) ). Maybe this is something to think about once Venture moves out of alpha.

P.S. The issues documented with my original ubuntu docker used a newer version of ubuntu, and therefore a newer boost version, among other things, which caused the documented issues. In other words, this will needed to be updated once (if) Venture is released non-alpha.

P.P.S. I'm also hesitant to use something like anaconda. It's pretty heavy, and causes problems if it's used in tandem with pip. This is just personal preference though.