Py3DNS is a dependancy generated in poetry.lock even if not required

[Pavel Dedík]

Easy to reproduce:

```
poetry add dkimpy==1.0.5
```

That command adds dnspython as a dependency to poetry.lock, while:

```
poetry add dkimpy==1.1.4
```

Adds py3dns. And that happens even if dnspython is installed.

Thus, py3dns remains unused (and causes issues on mac - https://github.com/sdgathman/pyspf/issues/2)

It would be nice if it was possible to not install py3dns (not have it generated in poetry.lock) and use dnspython instead. This could be achieved via extras.

[Scott Kitterman (kitterman)]

Is py3DNS installed? If so, that's the expected behavior (but I agree it's not ideal). If not, then I think it's a poetry bug.

[Pavel Dedík (paveldedik)]

No, even if it isn't installed, poetry does add it as a dependency to poetry.lock. So there is no way to not end up having it installed in production if we use dkimpy - it will always end up in poetry.lock.

I'm not sure how poetry works, why it is adding it as a dependency. But in version of dkimpy 1.0.5, it was adding dnspython as a transitive dependency instead.

The solution would be to use extras:

pip install dkimpy - would install whatever is already installed
pip install dkimpy[dnspython] - would require dnspython, py3dns would not be an option

I don't know if that is possible. But it would help.

If it is a poetry issue, I don't know what to report to them exactly.

[Scott Kitterman (kitterman)]

I'm not sure if that's possible.

I'm also not sure what's different now. I looked and the dependency related code in 1.1.4 is the same as 1.0.5. Did you use a different version of Poetry?

[Pavel Dedík (paveldedik)]

I am also not sure what the difference is. I was checking the code here. I am running the same version of poetry. You can try yourself. Just run the poetry add dkmipy==1.0.5, check the lock file and than run the same command with the latest version.

I don't know how poetry is determining dependencies. It might need some investigation.

[Scott Kitterman (kitterman)]

OK. I went back and looked again. The code is different. 1.0.5 still had support for pyhon2:

try:
    import DNS
    if sys.version_info[0] == 2:
        kw['install_requires'] = ['PyDNS']
    else:
        kw['install_requires'] = ['Py3DNS']
except ImportError: # If PyDNS is not installed, prefer dnspython
    kw['install_requires'] = ['dnspython>=1.16.0']

I'm not sure how to deal with this, but I now agree it needs dealing with. The problem is that the pyproject.toml based build systems have no way to support this. It's one of the reasons I haven't moved dkimpy away from it.

[Pedro Vicente (pedrovfer)]

Hi Scott and Pavel,

I think I may found the problem since I hit this issue now when trying to update to 1.1.5 from 1.0.5. So I cannot update the package due to there is a problem if both py3dns and dns are installed at the same time.

```
(dns_py3dns_problem-afmo) pedrovicente@pedrovicente  ~/cytora/external_proyects/dns_py3dns_problem  pip freeze
dnspython==2.4.1
py3dns==4.0.0
(dns_py3dns_problem-afmo) pedrovicente@pedrovicente  ~/cytora/external_proyects/dns_py3dns_problem  python -c "import dns"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/pedrovicente/.virtualenvs/dns_py3dns_problem-afmo/lib/python3.10/site-packages/dns/__init__.py", line 27, in <module>
    from .Base import DnsRequest
  File "/Users/pedrovicente/.virtualenvs/dns_py3dns_problem-afmo/lib/python3.10/site-packages/dns/Base.py", line 44, in <module>
    from . import Lib
  File "/Users/pedrovicente/.virtualenvs/dns_py3dns_problem-afmo/lib/python3.10/site-packages/dns/Lib.py", line 35, in <module>
    import DNS
ModuleNotFoundError: No module named 'DNS'
```
And my project needs dnspython due to it's a requirement from another dependency.

To debug the problem I've done the following:

I've installed packages from pypi and from source distribution in pypi. Installing from source distribution I've found there is a difference between 1.0.5 and 1.0.6 that makes poetry install py3dns:

```
cat dkimpy-1.0.5/dkimpy.egg-info/requires.txt
dnspython>=1.16.0

[ARC]
authres

[asyncio]
aiodns

[ed25519]
pynacl

[testing]
authres
pynacl

```

```
cat dkimpy-1.0.6/dkimpy.egg-info/requires.txt
Py3DNS

[ARC]
authres

[asyncio]
aiodns

[ed25519]
pynacl

[testing]
authres
pynacl

```
```
cat dkimpy-1.1.5/dkimpy.egg-info/requires.txt
Py3DNS

[ARC]
authres

[asyncio]
aiodns

[ed25519]
pynacl

[testing]
authres
pynacl
```

This problem affects all versions since 1.0.6 till latest one 1.1.5.

I've tried to build the package myself with latest changes in main and package built ok in my machine, could you check if there is any problem in the machine that builds and publishes this package?

See build log and requires.txt from build

```
python -m build
* Creating venv isolated environment...
* Installing packages in isolated environment... (setuptools >= 40.8.0, wheel)
* Getting build dependencies for sdist...
running egg_info
writing dkimpy.egg-info/PKG-INFO
writing dependency_links to dkimpy.egg-info/dependency_links.txt
writing entry points to dkimpy.egg-info/entry_points.txt
writing requirements to dkimpy.egg-info/requires.txt
writing top-level names to dkimpy.egg-info/top_level.txt
reading manifest file 'dkimpy.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files found matching 'dkim/*.pyc'
warning: no previously-included files found matching 'dkim/tests/*.pyc'
adding license file 'LICENSE'
writing manifest file 'dkimpy.egg-info/SOURCES.txt'
...
cat dist/dkimpy-1.1.5/dkimpy.egg-info/requires.txt
dnspython>=2.0.0

[ARC]
authres

[asyncio]
aiodns

[ed25519]
pynacl

[testing]
authres
pynacl
```

[Pedro Vicente (pedrovfer)]

Hi Scott and Pavel,

I was able to reproduce the problem that I think happens when releasing package, described above and reason of why poetry install py3dns.

```
docker run -it python /bin/bash
wget https://files.pythonhosted.org/packages/08/2b/05298eac3618233edb942e0548a9922dfc294dd075919205d94cc8c5b8ac/dkimpy-1.1.5.tar.gz
 tar xfz dkimpy-1.1.5.tar.gz
pip install build
cd dkimpy-1.1.5
pip install Py3DNS
python -m build --no-isolation
```

See full output

```
docker run -it python /bin/bash
root@03f10985ebf3:/# wget https://files.pythonhosted.org/packages/08/2b/05298eac3618233edb942e0548a9922dfc294dd075919205d94cc8c5b8ac/dkimpy-1.1.5.tar.gz
--2023-07-31 08:44:18-- https://files.pythonhosted.org/packages/08/2b/05298eac3618233edb942e0548a9922dfc294dd075919205d94cc8c5b8ac/dkimpy-1.1.5.tar.gz
Resolving files.pythonhosted.org (files.pythonhosted.org)... 151.101.133.55, 2a04:4e42:1f::311
Connecting to files.pythonhosted.org (files.pythonhosted.org)|151.101.133.55|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 67064 (65K) [application/octet-stream]
Saving to: ‘dkimpy-1.1.5.tar.gz’

dkimpy-1.1.5.tar.gz 100%[=======================================================================================================================>] 65.49K --.-KB/s in 0.03s

2023-07-31 08:44:18 (2.40 MB/s) - ‘dkimpy-1.1.5.tar.gz’ saved [67064/67064]

root@03f10985ebf3:/# tar xfz dkimpy-1.1.5.tar.gz
root@03f10985ebf3:/# pip install build
Collecting build
  Downloading build-0.10.0-py3-none-any.whl (17 kB)
Collecting packaging>=19.0 (from build)
  Downloading packaging-23.1-py3-none-any.whl (48 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.9/48.9 kB 2.2 MB/s eta 0:00:00
Collecting pyproject_hooks (from build)
  Downloading pyproject_hooks-1.0.0-py3-none-any.whl (9.3 kB)
Installing collected packages: pyproject_hooks, packaging, build
Successfully installed build-0.10.0 packaging-23.1 pyproject_hooks-1.0.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip is available: 23.1.2 -> 23.2.1
[notice] To update, run: pip install --upgrade pip
root@03f10985ebf3:/# cd dkimpy-1.1.5
root@03f10985ebf3:/dkimpy-1.1.5# pip install Py3DNS
Collecting Py3DNS
  Downloading py3dns-4.0.0-py3-none-any.whl (29 kB)
Installing collected packages: Py3DNS
Successfully installed Py3DNS-4.0.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip is available: 23.1.2 -> 23.2.1
[notice] To update, run: pip install --upgrade pip
root@03f10985ebf3:/dkimpy-1.1.5# python
Python 3.11.4 (main, Jul 28 2023, 04:37:46) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import DNS
>>>
root@03f10985ebf3:/dkimpy-1.1.5# python -m build --no-isolation
* Getting build dependencies for sdist...
runni...

[Scott Kitterman (kitterman)]

This is a difficult problem for me to solve as I use none of these tools. If someone can propose a solution that doesn't involve dropping the option to use Py3DNS, I'd love to know what it is.

[Pedro Vicente (pedrovfer)]

Hey Scott, I'm not proposing to drop the use of Py3DNS,

The problem is on how the package is built, the python env (virtualenv) where package is built is not isolated and has Py3DNS on it, see above example `python -m build --no-isolation`, so py3dns is addedd in the package build as requirement and tools like poetry get it as requirement and install it.

The only thing you need to do is to ensure that package is build and released from a python env, virtual env without py3dns on it at the moment of building this package. `python -m build` that command should ensure it.

If this is not clear can you share how the package is built and released to pypi please? So I can help you out and check how Py3DNS is added at the built time instead of dnspython.

Cheers,
Pedro

[Scott Kitterman (kitterman)]

All that's on pypi is the source tarball:

https://pypi.org/project/dkimpy/#files

Any build issues are in whatever the target environment is.

The build process is:

python3 setup.py sdist
twine upload [file]

[Pedro Vicente (pedrovfer)]

Hi Scott, could you please check when you run `python3 setup sdist` that Py3DNS is not available to import.

python3 -c "import DNS"

if it's available that's the reason why the sdist package sent to pypi has the wrong require in `dkimpy.egg-info/requires.txt

root@62086b4fda19:/dkimpy-1.1.5/dist# cat dkimpy-1.1.5/dkimpy.egg-info/requires.txt
Py3DNS

[ARC]
authres

[asyncio]
aiodns

[ed25519]
pynacl

[testing]
authres
pynacl

instead of

root@5ced28f3f5b3:/dkimpy-1.1.5/dist/dkimpy-1.1.5# cat dkimpy.egg-info/requires.txt
dnspython>=2.0.0

[ARC]
authres

[asyncio]
aiodns

[ed25519]
pynacl

[testing]
authres
pynacl

[Pedro Vicente (pedrovfer)]

If that's the reason you only need to create an empty virtualenv to run python3 setup.py sdist so py3dns is not avaliable when sdist is generated

[Scott Kitterman (kitterman)]

That makes sense. I almost certainly have it installed in the environment where I do that.\

[Pedro Vicente (pedrovfer)]

Perfect!!! Could you try and generate a new version 1.1.6 and check that contents are like this:

# cat dkimpy.egg-info/requires.txt
dnspython>=2.0.0

[ARC]
authres

[asyncio]
aiodns

[ed25519]
pynacl

[testing]
authres
pynacl

If so please publish it to pypi so I can update the lib on my end?

KR,
Pedro

[Pedro Vicente (pedrovfer)]

Hey Scott, are you planning to do a new release fixing this package issue soon? I want to upgrade to this latest version 1.0.5 but I'm affected by this issue.

Regards,
Pedro

[Pedro Vicente (pedrovfer)]

Hi Scott, do you have any update on this?

[Scott Kitterman (kitterman)]

On Wednesday, August 16, 2023 3:42:39 AM EDT you wrote:
> Hi Scott, do you have any update on this?

Not really. Given that the Python ecosystem doesn't support the concept of
alternative dependencies (sigh), I'm still considering what to do about this
that's more than just a short-term fix.

Regenerating the sdist with dnspython being required instead of py3dns really
only shifts the problem.

I'm considering making them both optional and you have to pick one, but that's
not very satisfactory either, as the package should work as installed with no
options.

Scott K

[Pedro Vicente (pedrovfer)]

I understand it, but could you consider revert to the previous behaviour on 1.0.5 where packages where released with dnspython as main dep instead of Py3DNS?

You could later find a way of making it more rock solid?

Also checking code in dnsplug.py, dnspython is preferred and Py3DNS is alternative.

# Prefer dnspython if it's there, otherwise use pydns
try:
    import dns.resolver
    _get_txt = get_txt_dnspython
except ImportError:
    try:
        import DNS
        DNS.DiscoverNameServers()
        _get_txt = get_txt_pydns
    except:
        raise

What do you think?

Pedro

[Pedro Vicente (pedrovfer)]

Hey Scott, sorry to bother you what do you think about my previous message, any chance I can get a version with the dependency updated in the package?

[Pedro Vicente (pedrovfer)]

Hi Scott, not sure if you are off and don't have access to this, but please when you are back could you tell me if you can address a new release as asked? If not I'll need to think a way of doing it by myself, basically I want to remove Deprecation Warnings in app and I cannot update to latest version because I'm hitting this issue but with a new release backward compatible as commented above, it will work for me and I think for other people too.