Need help installing data files with pyproject.toml

Python Help
,
1

Dear Experts,

I have a project that looks like:

src/
├── data_checks
│   ├── filter_file.py
│   ├── ntuple_filter.py
│   ├── selector.py
│   └── utilities.py
├── data_checks_data
│   ├── __init__.py
│   ├── config
│   │   ├── __init__.py
│   │   └── dt_2024_turbo.toml
│   └── pfns
│       ├── __init__.py
│       └── dt_2024_turbo.json
└── data_checks_scripts
    ├── __init__.py
    └── save_pfns.py

I am trying to install the toml and json files with:

[project]
name        = 'data_checks'
version     = '0.0.1'
dependencies= ['tqdm', 'rx_scripts', 'apd'] 

[project.optional-dependencies]
dev  = []

[tools.setuptools.packages.find]
where=['src']

[tools.setuptools.package-data]
"data_checks"=['data_checks_data/*.json', 'data_checks_data/*toml']

[project.scripts]
save_pfns='data_checks_scripts.save_pfns:main'

but it does not work. Could you please help me with this?

Cheers.

2

You show:

[tools.setuptools.package-data]
"data_checks"=['data_checks_data/*.json', 'data_checks_data/*toml']

But there are no data files in the /src/data_checks directory.

Maye you should have:

[tools.setuptools.package-data]
"data_checks_data" = ["*.json", "*toml"]

or:

[tools.setuptools.package-data]
"" = ["*.json", "*toml"]

I am not 100% sure, I have not tested this setup in particular, and I have not done this kind of things in a long while in general.

3

Dear @sinoroc

Thanks for your answer. I tried it, however I still do not think those files are been installed, I see:

image

I am installing with:

pip install -e .
4

OK, what works is:

[tool.setuptools.package-data]
'data_checks_data' = ['*.json', '*.toml']

but the layout has to change to:

src/
├── data_checks
│   ├── filter_file.py
│   ├── ntuple_filter.py
│   ├── selector.py
│   └── utilities.py
├── data_checks_data
│   ├── __init__.py
│   ├── dt_2024_turbo.json
│   └── dt_2024_turbo.toml
└── data_checks_scripts
    ├── __init__.py
    └── save_pfns.py

which might be acceptable, however not ideal, given that things will get messy in data_checks_data.

5

I believe it should be possible to return to your original source tree directory layout, if you work with the glob patterns and/or the package names.

Maybe try one of those:

'data_checks_data.config' = ['*.toml']
'data_checks_data.pfns' = ['*.json']

'data_checks_data' = ['**.json', '**.toml']

'' = ['**.json', '**.toml']

'data_checks_data' = ['config/*.toml', 'pfns/*.json']

The documentation is not explicit about the handling of sub-packages.

6

Hello @sinoroc

Thanks for your answer. I had tried:

'data_checks_data' = ['config/*.toml', 'pfns/*.json']

at some point and it did not work. Now I tried with a different project:

[tool.setuptools.package-data]                                                                                                           
dmu_data=['text/*.txt', 'text/*.toml']

and it worked. Maybe the quotes in the first one were the problem.

Cheers.