The current behavior of csv.QUOTE_STRINGS (as of Python 3.13, with the bug fix for CSV reader should support QUOTE_NOTNULL and QUOTE_STRINGS · Issue #113732 · python/cpython · GitHub) is that it cannot read all data types it has written.
Example Python data to be written:
field_names = ['str', 'int', 'float', 'bool', 'none', 'list', 'dict', 'set']
field_rows = [
{
'str': 'foo', 'int': 42, 'float': 3.14, 'bool': True, 'none': None,
'list': ['a', 'b'], 'dict': {'a': 1, 'b': 2}, 'set': {'a', 'b'},
},
{
'str': '', 'int': 0, 'float': 0.0, 'bool': False, 'none': None,
'list': ['a'], 'dict': {'a': 1}, 'set': {'a'},
},
{
'str': '', 'int': 0, 'float': 0.0, 'bool': False, 'none': None,
'list': [], 'dict': {}, 'set': {},
},
Resulting .csv file written with csv.DictWriter with delimiter=",", quotechar='"', quoting=csv.QUOTE_STRINGS:
"str","int","float","bool","none","list","dict","set"
"foo",42,3.14,True,,"['a', 'b']","{'a': 1, 'b': 2}","{'a', 'b'}"
"",0,0.0,False,,['a'],{'a': 1},{'a'}
"",0,0.0,False,,[],{},{}
The format in this .csv file is quite nice because:
- It distinguishes boolean values from strings
- It distinguishes None from empty strings
and thus, it provides the ability to (mostly) re-establish the original Python types when reading it. (The only minor issue is that empty dict and empty set cannot be distinguished)
However, reading that .csv file with:
with open(csvfile_name, 'r') as csvfile:
rows = list(csv.DictReader(
csvfile, lineterminator="\n", delimiter=",",
quotechar='"', quoting=csv.QUOTE_STRINGS))
results in this failure (pytest output):
self = <csv.DictReader object at 0x108299a90>
def __next__(self):
if self.line_num == 0:
# Used only for its side effect.
self.fieldnames
> row = next(self.reader)
E ValueError: could not convert string to float: 'True'
/opt/homebrew/Cellar/python@3.13/3.13.0_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/csv.py:178: ValueError
Even though True is not quoted in the .csv file, it is interpreted as a string, and because it was not quoted, it is attempted to be converted to float, following the csv.QUOTE_STRINGS rule to interpret an unquoted empty string as None and to convert all other non-quoted fields to float.
A similar problem exists for lists and dicts and sets that end up unquoted in the .csv file.
In addition, int and float values can be distinguished in the .csv file by the presence of a dot, but both int and float values are converted to float when reading.
I propose that the behavior of csv.QUOTE_STRINGS when reading is changed to the following:
- convert unquoted empty strings to
None - convert unquoted boolean values to bool
- convert unquoted lists to list
- convert unquoted dicts (including empty dict) to dict
- convert unquoted (non-empty) sets to set
- convert unquoted values that match ints, to int
- convert unquoted values that match floats, to float
- convert other unquoted values to str
- convert quoted values to str
The user could then check for strings that represent dicts or lists or sets and convert them. Doing that in the csv module would go too far IMO, because there may be intentional string typed values with such content.
I think this is change would be reasonably backwards compatible, because it changes only a situation that previously lead to an exception, and I think it is unlikely that this exception could have been recovered by users. The distinction between float and int would now cause int to be returned where float was returned previously, but that would probably work in most cases, unless the user code checks for types.
If this change is considered too backwards incompatible, a new quote constant with the proposed reading behavior (and the writing behavior of csv.QUOTE_STRING) could be introduced.
BTW, I tried the other csv.QUOTE_* constants, but none of them came close to writing the .csv file as distinguishable as csv.QUOTE_STRINGS.
Please comment on this proposal, and let me know whether this could be brought forward as an issue on GitHub · Where software is built.