I have a script running obtaining data from a server through an API call. The data are per date. The date range is too big. It counts for 3 years and data are per day. Since the python script I am running at the moment is not 100% stable, I cannot trust it to run unconditionally for all those thousand days. I need a clause to put in my code and make a copy of saved data for every 3 months. If that’s the case, even if server or the execution crashes for some reason, I won’t repeat it again from scratch.
I need something like that maybe
Is my approach correct?
Furthermore. I use colab environment. Where do I need to place such a function? I was thinking that it should be at the same cell, right after the main function that sends the queries to server and retrieves the data. Right after it, so this while loop works at the same time and saves the data every three months during the main function execution.
If I place this part of code at a subsequent cell right after the main function cell, then, the program will wait until the main function completes its action before it save the data for every three months, I guess. That way, if time runouts or server crushes then I will lose the whole progress, since, I will need to repeat the main function execution from beginning. But if I place this while loop at the same cell where the main function is and put where thrmonth_data(START,END) the main function then I will have a saving per 3 months and even if it crushes or server times out, then, at least, I will have some progress saved.
What do you think?
fmt = '%Y-%m-%d' cur = datetime.datetime.strptime(START, fmt).date() end = datetime.datetime.strptime(END, fmt).date() # prepare the loop: delta = datetime.timedelta(days=1) while cur < end: START = cur.strftime(fmt) # current date cur += delta # next day END = cur.strftime(fmt) thrmonth_data(START, END) with open('datafor3months.json', 'w') as fp: json.dump(thrmonth_data, fp)