I have a problem with the file csv

I have a piece of code that allows me to download a csv file from the web data server, like eg. temperature, humidity etc.
My data download has a start date and an end date by having it written from the terminal with raw_input, (eg I want the data starting from 01-10-2020 to 10-08-2020) with the various measurements and up to here it works.
Now the problem I’m having is that I would need this data over a month or a year to say that I need to hang more csv files. As I did, it can’t download for more than a month or a year, maybe it’s the server that’s causing problems.
So how can I write to the previously created csv file, another csv file overwritten to the other without changing the one that was written before with a new date?
If you can help me thank you in advance.

Hi Valentina,

I’m afraid I cannot work out what problem you are asking us to solve, or

what you have already tried.

You can help us to help you if you ask better questions. It

might help you to read this first:

Thank you.

I share my python file for better understanding.

import numpy as np
import pandas as pd
import datetime as dt
import requests
import csv

numAQP = raw_input (“Inserisci il numero della piattaforma:”)
start_date = raw_input (“Inserisci la data iniziale:”)
end_date = raw_input (“Inserisci la data finale:”)

url = ‘url’ + numAQP +’/csv?start_date=’ + start_date + '&end_date= ’ + end_date
response = requests.get(url)
url_content = response.content
csv_file = open(‘out.csv’, ‘w’)

csv_file.write(url_content)
csv_file.close()

This allows me to download a csv file from the web data server, such as eg. temperature, humidity etc.
My data download has a start date (start_date) and an end date (end_date), writing it from the terminal with raw_input.
(eg. I want the data from 01-08-2020 to 20-08-2020) with the various measures and up to here it works perfectly).

My problem I’m having is that I would need this data to be a month or a year, I need to hang multiple CSV files.
As I did, it fails to download, evidently it is due to the server and I was given this task.

So how can I hang the previously created csv file with another csv file in the other without changing the one that was written before with a new date? For example. I download 01-03-2020 until 20-03-2020, I hang the other file that goes from 21-03-2020 to 30-03-2020 and so on … so that I can have a csv file with the data inside ranging from 01-03-2020 until 30-09-2020 so it’s 6 months.
If you can help me, thank you in advance.

Hey, so if I understand you correctly you want to download a large amount of data from a web server, but downloading it all at once would fail so you are trying to chunk the request into smaller steps, then combine the data at the end to get your full dataset.

To do this, you can make use of the datetime module. It has utilities to parse strings into date objects, which can then be incremented by any timestep using datetime.timedelta. For example, here is how you would find the string representation of tomorrow’s date:

from datetime import datetime, timedelta
today = datetime.now()   # a datetime object representing the time right now
# then we can convert that to a string using strftime:
today_date_string = today.strftime('%d-%m-%Y')  # '09-10-2020'
# now we can add one day using timedelta, and get that date as a string:
tomorrow_date_string = (today + timedelta(days=1)).strftime('%d-%m-%Y') # '10-10-2020'

You can then use this by starting at the start date and adding say 7 days each time until you get to the end date:

date = datetime(year=2020, month=3, day=1)
end_date = datetime(year=2020, month=9, day=1)
while date < end_date:
    start_date_str = date.strftime('%d-%m-%Y')
    date = min(date + timedelta(days=7), end_date)
    end_date_str = date.strftime('%d-%m-%Y')
    # your code here to get the data using these start/end strings

To combine all of the data in a csv, you can actually just append the new data to the end of an existing csv file. To do that you have to open it in append mode ('a') instead of write mode ('w'), otherwise the data will be deleted by the new write:

with open('out.csv', 'a') as f:
    f.write(url_content)

Best of luck with your project :slight_smile: