TESTED_VERSIONS is ignored in tox?

[pllim]

I set custom TESTED_VERSIONS in astropy/astrowidgets#103 but it does not seem to work, as the test log still shows:

Running tests with Astropy version 4.0.
Running tests in astrowidgets docs.

It works in my other projects. The only difference I can think of is that astrowidgets is using tox, unlike the others that I am using for comparison.

[weaverba137]

I am seeing similar behaviour in a variety of testing scenarios, all involving tox.

• My conftest.py file. It is almost exactly the same as the package-template version.
• A recent test that suggests the conftest.py is being ignored.. Neither the custom PYTEST_HEADER_MODULES nor TESTED_VERSIONS appear in the header.
• A test from the same build showing the expected header values.. Note that this is a coverage test.
• A test on astropy 2.0.16. This also has the correct header values.
• All tests that use pytest-astropy-header, except for the two cases noted above, appear to ignore conftest.py.

[pllim]

I think this is the issue that @bsipocz and @larrybradley also ran into.

[pllim]

Maybe this is not robust enough:

pytest-astropy-header/pytest_astropy_header/display.py

Lines 106 to 114 in 53b214e

	# TESTED_VERSIONS can contain the affiliated package version, too
	if len(TESTED_VERSIONS) > 1:
	for pkg, version in TESTED_VERSIONS.items():
	if pkg not in ['Astropy', 'astropy_helpers']:
	s = "\nRunning tests with {} version {}.\n".format(
	pkg, version)
	else:
	s = "\nRunning tests with Astropy version {}.\n".format(
	TESTED_VERSIONS['Astropy'])

And perhaps if we remove the old checks in #32 , things will be easier to debug. 🤞

[pllim]

Hmm... Using this plugin and tox in stsci-package-template does not show this problem. I am not sure why!

============================= test session starts ==============================
platform linux -- Python 3.7.9, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
cachedir: .tox/py37-test-oldestdeps/.pytest_cache

Running tests with packagename version 0.1.dev2+g7bfb23d.
Running tests in packagename docs.

xref spacetelescope/stsci-package-template#25

[saimn]

So after some investigation, the conftest.py is not always loaded if it is located in packagename/. It is loaded only if running pytest with some path argument. So the solution is to put conftest.py in the root directory (which is what is done for stsci-package-template but not for the other examples mentioned here).

[weaverba137]

Thanx, I'll give that a test.

[pllim]

I think the reason we used to have packagename/conftest.py layout was for that test runner (i.e., python -c "import packagename; packagename.test()"), or so I thought @astrofrog said once.

[saimn]

Yes, I think you're right. So maybe an option is to duplicate the files...

[pllim]

duplicate the files

Not as clean but if it works, we could recommend it.

[astrofrog]

I wonder if we could somehow have the main version be in the root directory and then copy it into the installation path when installing, to not actually have to duplicate it manually?

[pllim]

👎 to auto copying, as it really only affects the astropy test runner and pytest-astropy-header technically no longer depends on astropy (despite the name).

[pllim]

So, I just created a root level conftest.py to get around this problem. But for whatever reason, just having even an empty conftest.py in the root dir caused the path reported by tox run to change from:

..\..\.tox\py38-test\lib\site-packages\jdaviz\configs\...

to

..\..\jdaviz\configs\...

❗ ❗ ❗

What I am doing wrong? 🤯

xref spacetelescope/jdaviz#583