We have a huge data processing code written in python 3.6.5 version running inside Linux Docker container it works for smaller dataset but for larger dataset > 80K records it is failing with PicklingError.
Code Structure:
from functools import partial
from multiprocessing import Pool
class workclass(object):
def f(self,a,b,c):
# data processing
def main():
w = workclass
sample_dict = {}
sample_dict = loading fromdb
sample_list = list(sample_dict.keys())
func = partial(w.f, sample_dict, 'env')
p = Pool(5)
result = p.map(func, sample_list)
print(result)
if __name__=='__main__':
main()
ails in python/reduction.py save_global line 922 Error: picklingerror: cant pickle ‘java.lang.Integer’
Any suggestion will be of great help
Strangly the same program works fine in Windows OS but fails in Linux and Linux Docker container
We have a huge data processing code written in python 3.6.5 version
running inside Linux Docker container it works for smaller dataset but
for larger dataset > 80K records it is failing with PicklingError.
Code Structure:
[… snip …]
This can’t be your code. It isn’t even indented properly, and has other
syntax errors.
ails in python/reduction.py save_global line 922 Error: picklingerror:
cant pickle ‘java.lang.Integer’
That sounds like you’ve got some Java-implemented library somewhere in
your data handling. It isn’t evident in what you’ve shown us.
Strangly the same program works fine in Windows OS but fails in Linux
and Linux Docker container
Might be just lucky, or maybe the support libraries in Linux are
different from those on Windows.
Anyway, I don’t think we’ve got enough information here to help you
beyond tritely observing that you’re trying to pickle something that
isn’t a pure Python type. Pickle is, IIRC, used to transfer information
to subprocesses in the multiprocessing module.