Dear python experts,
I’m fairly new to python and try to code a script for the following task:
A lot of APIs should be queried by HTTP POST request. In order to speed up the responses, blocks of 3 requests should be processed asynchronously or in parallel.
An ID is assigned to each request which is not part of the API but is needed to process the response afterwards.
In order to make testing easy and clear I limited the requests to 9 and coded a small PHP script that simulates the API:
<?php
header('Content-Type: application/json; charset=utf-8');
$start=microtime(true);
if ($_POST['param1']==1) {
sleep(3);
} else {
sleep(1);
}
$resp=["param1"=>$_POST['param1'], "time"=>microtime(true)-$start];
echo json_encode($resp);
I started with ospider’s answer here on stackoverflow:
as I got the impression that it’s the most modern approach.
Correct me if this is wrong.
This is my first attempt:
import time
import aiohttp
import asyncio
params = [1, 2, 3, 4, 5, 6, 7, 8, 9]
ids = [11, 12, 13, 14, 15, 16, 17, 18, 19]
url = r'http://localhost//_python/async-requests/the-api.php'
# Fetch data for one parameter asynchronously:
async def fetch(session, url, params, id):
# Send request with method POST:
start = time.perf_counter()
async with session.post(url, data=params) as response:
# Decode JSON in response:
resp = await response.json()
print(str(time.perf_counter() - start) + ' ' + resp['param1'])
# Add ID to response:
resp['id'] = id
# return response:
return resp
async def main():
async with aiohttp.ClientSession() as session:
idx = 0
tasks = []
start = time.perf_counter()
# Loop through params:
for param in params:
# Append task for fetching the current param to list,
# ID as an additional parameter:
tasks.append(fetch(session, url, {'param1': param}, ids[idx]))
idx += 1
# 3 tasks prepared?
if(idx % 3 == 0):
# Process tasks asynchronously:
resps = await asyncio.gather(*tasks)
for resp in resps:
print(resp)
# print(time.perf_counter() - start)
# reset task list:
tasks = []
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
I have to admit that I don’t completely understand what I coded but it seems to work fine.
I added the ID mentioned above as a parameter to the function fetch and added it to the response from the server.
It seemed to me that, although the procedure is asynchron in one block of 3, the correct order is kept in the responses. In order to verify this, I used a larger value for the sleep of the very first request and indeed, the correct order was restored. Therefore I made an additional version where the ID is added to the response directly without handing over to fetch and adding to the reponse:
import time
import aiohttp
import asyncio
params = [1, 2, 3, 4, 5, 6, 7, 8, 9]
ids = [11, 12, 13, 14, 15, 16, 17, 18, 19]
url = r'http://localhost//_python/async-requests/the-api.php'
# Fetch data for one parameter asynchronously:
async def fetch(session, url, params):
# Send request with method POST:
start = time.perf_counter()
async with session.post(url, data=params) as response:
# Decode JSON in response:
resp = await response.json()
print(str(time.perf_counter() - start) + ' ' + resp['param1'])
# return response:
return resp
async def main():
async with aiohttp.ClientSession() as session:
idx = 0
idxStart = 0
tasks = []
# Loop through params:
for param in params:
# Append task for fetching the current param to list,
tasks.append(fetch(session, url, {'param1': param}))
idx += 1
# 3 tasks prepared?
if(idx % 3 == 0):
# Process tasks asynchronously:
resps = await asyncio.gather(*tasks)
# print all responses including corresponding ID:
for resp in resps:
print(str(resp) + ' ' + ids[idxStart])
idxStart += 1
# reset task list:
tasks = []
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
Question: Is this really safe?
Another question: The PHP scripts sleeps for 1 or 3 seconds, however it takes appr. 1.4 seconds until the response arrives. This is a bit surprising as the PHP script is on my local disc. I suspect that transferring the data through the local web server requires the additional time?
Any comments and improvements are welcome.
Best regards - Ulrich