What am I missing - can't get the build to pick up a sub folder 'migrations'

New to packaging - would be grateful if someone could explain why the subfolder ‘migrations’ in that is part of the project is IN the tar. file but when using pip install does not get copied in.
Tried a of lof different things - none seem to make a difference.

https://files.pythonhosted.org/packages/3d/8b/07edef5498e35f6033b7c42dbb27fdce6137191414ad1de31a1a36a29762/django-openai-assistant-0.28.tar.gz

First, you have a setup.py file and a setup.cfg file, but no pyproject.toml file. This is heavily discouraged. See Is setup.py deprecated? - Python Packaging User Guide. At the minimum, pyproject.toml should contain a [build-system] table, but moving metadata from setup.py to the [project] table is recommended.

Second, you have explicitly specified packages=['django_openai_assistant'] in your setup.py, omitting 'django_openai_assistant.migrations', so setuptools is just being obedient here.

Third, you have in your setup.py:

long_description = (this_directory / "README.md").read_text()

which is an antipattern because it reads the file with a locale-dependent encoding instead of always UTF-8, and this can produce errors on Windows if you put any non-ASCII characters in the README.

Fourth, license = "MIT" is not the recommended practice, it’s best to use a classifier for well-known licenses. See Writing your pyproject.toml - Python Packaging User Guide

Here’s what I would do: remove all of setup.py, setup.cfg and MANIFEST.in, and add a pyproject.toml with the following contents:

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "django-openai-assistant"
version = "0.28"
dependencies = ["openai", "markdown"]
readme = "README.md"
authors = [{name = "Jean-Luc Vanhulst", email = "jl@valor.vc"}]
maintainers = [{name = "Jean-Luc Vanhulst", email = "jl@valor.vc"}]
classifiers = ["License :: OSI Approved :: MIT License"]
urls = {Homepage = "https://github.com/jlvanhulst/django-openai"}
keywords = ["django", "celery", "openai", "assistants"]

I tested the build, it produced functional distributions, with the subpackage properly included.

Note that I’ve switched the backend to Hatchling, but if you prefer to stick with setuptools (which is, in my humble opinion, much more complicated and difficult to use), you might need to add back a MANIFEST.in to include LICENSE.

2 Likes

Oh, and one last thing: since you say you’re new to packaging, what tutorial did you follow? Because whatever it is, it seems to contain lots of old and/or bad practices. There is an up-to-date and official tutorial at Packaging Python Projects - Python Packaging User Guide

And one more “last” thing: I see that you’ve only uploaded the source distribution to PyPI (here). But you also need to build a wheel (.whl file) and upload it too. python -m build will build both of them.

Thanks for your quick and detailed response(s).
I just did a build based on the toml file.

When you said ‘I tried and it build correctly’ - what I am wondering about is if you tried to INSTALL the tar file and then check if the migrations subfolder was installed? Because THAT is my key problem. THe folder is IN the tar file as a subfolder - but when using pip install on the TAR it never gets copied.

I’m pretty sure that LICENSE is one of the files which setuptools includes by default, so for that particular file you don’t need a manifest. But you could easily need one for other files.

The thing that setuptools still has going for it is probably the best docs of any of the build backends.

I don’t exactly disagree with you suggesting hatchling, but I’m still not sure it’s the right choice for beginners to be given hatch (the frontend/workflow tool). Wouldn’t Flit be a reasonable contender here as well, since it focuses on being minimal?

Could you link to the source for this project? Having to pull the sdist to see what it contains is a bit of an improvement to participation, and the homepage link on pypi is a 404 dead end.

Are the files you’re expecting to have copied inside of your source dir or adjacent to it? It’s common for build backends to include such files in the sdist but not the wheel.

When you run python -m build, it generates the sdist (.tar.gz), but also a wheel (.whl), which is actually a ZIP archive whose contents follow certain norms. When pip installs a package, it basically either downloads a wheel from PyPI, or builds the wheel (if you only uploaded the sdist to PyPI, or if you install a local folder as in pip install .), then it spreads the wheel and does some light bookkeeping. So, I didn’t actually install your project, I just inspected the wheel, and I found that the subdirectory was included, which means it will be installed.

We must think quite differently, because a large part of the setuptools documentation is practically incomprehensible to me…

Well my problem is that when you actually install the pip install - it does not install that sub folder- it installs the the main folder, not the /migrations folder that lives under it.

Sorry about that - the github is now public

Yes, I have understood that. And I am telling you that my proposed change solves your problem.

You can experiment with this for yourself. In your repository, cloned and not modified:

~/repos/django-openai-assistant $ python -m build
* Creating virtualenv isolated environment...
* Installing packages in isolated environment... (setuptools >= 40.8.0, wheel)
* Getting build dependencies for sdist...
[log snipped, setuptools is super verbose]
* Building wheel from sdist
* Creating virtualenv isolated environment...
* Installing packages in isolated environment... (setuptools >= 40.8.0, wheel)
* Getting build dependencies for wheel...
[log snipped, setuptools is super verbose]
Successfully built django-openai-assistant-0.28.tar.gz and django_openai_assistant-0.28-py3-none-any.whl

~/repos/django-openai-assistant $ cd dist/

~/repos/django-openai-assistant/dist $ unzip django_openai_assistant-0.28-py3-none-any.whl
Archive:  django_openai_assistant-0.28-py3-none-any.whl
  inflating: django_openai_assistant/__init__.py  
  inflating: django_openai_assistant/apps.py  
  inflating: django_openai_assistant/assistant.py  
  inflating: django_openai_assistant/models.py  
  inflating: django_openai_assistant-0.28.dist-info/LICENSE  
  inflating: django_openai_assistant-0.28.dist-info/METADATA  
  inflating: django_openai_assistant-0.28.dist-info/WHEEL  
  inflating: django_openai_assistant-0.28.dist-info/top_level.txt  
  inflating: django_openai_assistant-0.28.dist-info/RECORD

As you can see, the django_openai_assistant/migrations/ folder is not included in the wheel, so it won’t be installed. Now with my changes:

~/repos/django-openai-assistant $ python -m build
* Creating virtualenv isolated environment...
* Installing packages in isolated environment... (hatchling)
* Getting build dependencies for sdist...
* Building sdist...
* Building wheel from sdist
* Creating virtualenv isolated environment...
* Installing packages in isolated environment... (hatchling)
* Getting build dependencies for wheel...
* Building wheel...
Successfully built django_openai_assistant-0.28.tar.gz and django_openai_assistant-0.28-py2.py3-none-any.whl

~/repos/django-openai-assistant $ cd dist/

~/repos/django-openai-assistant/dist $ unzip django_openai_assistant-0.28-py2.py3-none-any.whl
Archive:  django_openai_assistant-0.28-py2.py3-none-any.whl
  inflating: django_openai_assistant/__init__.py  
  inflating: django_openai_assistant/apps.py  
  inflating: django_openai_assistant/assistant.py  
  inflating: django_openai_assistant/models.py  
  inflating: django_openai_assistant/migrations/0001_initial.py  
  inflating: django_openai_assistant/migrations/__init__.py  
  inflating: django_openai_assistant-0.28.dist-info/METADATA  
  inflating: django_openai_assistant-0.28.dist-info/WHEEL  
  inflating: django_openai_assistant-0.28.dist-info/licenses/LICENSE  
  inflating: django_openai_assistant-0.28.dist-info/RECORD  

Now the subfolder is included in the wheel, so it will be installed.

I appreciate your help - and indeed version 0.29 is now deployed WITH the migrations folder.
The two changes are: added the toml file as provided. And created by using python3 -m build

Still trying to understand what caused the problem. Now with twine I’m uploading the whl not the tar.gz
Should I remove the setup.py? setup.cfg

Also, speaking about best practices - what do you recommend for where/how to add updates per version?

And you commented on the documentation include - which again was found ‘somehwere’ on the internet - so totally open for better. I need to add more/better documentation. Any suggestions there are welcome as well.

And of course if there is anyone with a Django/Celery app that wants to play with OpenAI Assistants I’m curious to get feedback. We run a lot of Assistants on this internally.

The basic cause of the problem was

packages=['django_openai_assistant']

in your setup.py file. In setuptools, the packages argument specifies packages which are included in a shallow way, i.e., this code tells setuptools to include files immediately in django_openai_assistant/, but not in any subfolder. On the other hand, the code

packages=['django_openai_assistant', 'django_openai_assistant.migrations']

would have included the subfolder.

Generally speaking, the behavior and options of setuptools around specifying files to include/exclude are quite complex, which is also why I suggested Hatchling.

Don’t – you should upload both the sdist and the wheel.

Yes. They are completely ignored now that you are using Hatchling. And you can remove MANIFEST.in too.

Sorry, I don’t understand this question. Could you rephrase it?

Perhaps it’s just that after years of using setuptools I can no longer tell what is strange arcana and what’s straightforward usage! :joy:


The norm for a new package is not to include either of these files.

They’re not doing any harm, but I think you’ll find life simpler with fewer files to manage.

I’m unclear on the question. Are you asking where to store your package version number? Or something else?

Packages should generally define their version in pyproject.toml. If you need the version number at runtime, use importlib.metadata.

I can’t comment on the documentation content too much, not being a user of the relevant tech stack. But I would recommend setting up a separate documentation site (readthedocs is quite popular and I enjoy using it, although there is a bit of a learning curve of course).
Projects may start with documentation all in a single readme, but it’s relatively rare, in my experience, that the docs can stay that small. Setting up a docs site early in a project is usually an effort-saver. You can turn the readme into a quickstart doc and put all of the details into your docs site.