We Keep Coding

concurrent.futures return same response for different api calls

I'm trying to speed up API requests using multi threading.
I don't understand why but I often get the same API response for different calls (they should not have the same response). At the end I get a lot of duplicates in my new file and a lot of rows are missing.
example : request.post("id=5555") --> response for the request.post("id=444") instead of request.post("id=5555")
It looks like workers catch the wrong responses.
Have anybody faced this issue ?
` def request_data(id, useragent):
- ADD ID to data and useragent to headers -
time.sleep(0.2)
resp = requests.post(
-URL-,
params=params,
headers=headerstemp,
cookies=cookies,
data=datatemp,
)
return resp
df = pd.DataFrame(columns=["ID", "prenom", "nom", "adresse", "tel", "mail", "prem_dispo", "capac_acc", "tarif_haut", "tarif_bas", "presentation", "agenda"])
ids = pd.read_csv('ids.csv')
ids.drop_duplicates(inplace=True)
ids = list(ids['0'].to_numpy())
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor:
future_to_url = {executor.submit(request_data, id, usera): id for id in ids}
for future in concurrent.futures.as_completed(future_to_url):
ok=False
while(ok==False):
try:
resp = future.result()
ok=True
except Exception as e:
print(e)
df.loc[len(df)] = parse(json.loads(resp))
`
I tried using asyncio, first response from Multiple async requests simultaneously but it returned the request and not the API response...

Slack blocks + sending response doesn't work in python, not even when async - only in CURL

Sending block to Slack - it shows on slack correctly.
Button is clicked, I am getting the interactive event webhook.
webhook returns 200 OK to slack, not before scheduling an async POST to the response_url
the return message is posted to response_url AFTER the 200 ack is returned from the webhook function
Slack responds 404 to the response_url
that same response URL works when curled outside of python....
I don't get why slack rejects the posted return message with 404 when the very same response_url works when used in curl outside of python.
my webhook processor:
def slack_webhook(request):
json_dict = json.loads(request.body.decode('utf-8'))
token = json_dict['token'] if 'token' in json_dict else None
message = json_dict['message'] if 'message' in json_dict else None
trigger_id = json_dict['trigger_id'] if 'trigger_id' in json_dict else None
response_url = json_dict['response_url'] if 'response_url' in json_dict else None
actions = json_dict['actions'] if 'actions' in json_dict else None
for action in actions:
print(f"** received action {action['action_id']}")
print(f"** response_url: {response_url}")
print(f"** trigger_id: {trigger_id}")
payload = {
"replace_original": True,
"text": "Success!",
}
# async via Celery...
send_slack_interactive_response.delay(response_url, payload, trigger_id)
return HttpResponse(status=200)
the async celery task which sends the
#app.task(bind=True, retry_kwargs={'max_retries': 10, 'countdown': 5})
def send_slack_interactive_response(self, response_url, payload, trigger_id):
print(f"** -> response_url: {response_url}")
print(f"** -> trigger_id: {trigger_id}")
if response_url and payload and trigger_id:
headers = {"Content-Type": "application/json"}
payload['trigger_id'] = trigger_id
print(json.dumps(payload))
r = requests.post(response_url, data=payload, headers=headers)
print(r.__dict__)
This function fails with 404. however, when i take the response_url, trigger_id and create the exact same POST with command line curl - it works.
What am I doing wrong?

Just a comment on your code : You can do token = json_dict.get("token", None) that save you lot of code
On your problem :
Be sure Celery parameters is not weird encoded (response_url is send to messagerie and is encoded)
Be sure request parameters are well used (like using json instead of data...)

How to feed client request headers and server response headers in aiohttp server response body?

A simple aiohttp server code below, I want to know how to return all client's http request head information and server response http head info in server's response.
A simple target is when I use web browser open the http://127.0.0.1:8080, the web page can display client's http request head and server response http head immediately.
Any help is appreciated.
from aiohttp import web
async def handle(request):
request_head = web.Request.headers //Q1?
response_head = web.Response.headers //Q2?
return web.Response("\n".join((request_head,response_head)))
app = web.Application()
app.router.add_get('/', handle)
web.run_app(app)

managed got a dirty solution with flaw regarding the response head part - not very sure the 2nd web.Response will change the header itself, any improvement is appreciated, thanks.
from aiohttp import web
async def handle(request):
request_head = tuple((k.encode('utf-8'), v.encode('utf-8')) for k, v in request.headers.items())
resp = web.Response(text = str(request_head))
response_head = tuple((k.encode('utf-8'), v.encode('utf-8')) for k, v in resp.headers.items())
resp = web.Response(text = "\n".join((str(request_head),str(response_head))))
return resp
app = web.Application()
app.router.add_get('/', handle)
web.run_app(app)

DELETE Request in Python "requests" module not working with body

I have been using this function to handle http requests with no problems:
def do_request(self, method, url, **kwargs):
params = kwargs.get('params', None)
headers = kwargs.get('headers', None)
payload = kwargs.get('data', None)
request_method = {'GET':requests.get, 'POST': requests.post, 'PUT': requests.put, 'DELETE': requests.delete}
request_url = url
req = request_method[method]
try:
res = req(request_url, headers=headers, params=params, data=json.dumps(payload))
except (requests.exceptions.ConnectionError, requests.exceptions.RequestException) as e:
data = {'has_error':True, 'error_message':e.message}
return data
try:
data = res.json()
data.update({'has_error':False, 'error_message':''})
except ValueError as e:
msg = "Cannot read response, %s" %(e.message)
data = {'has_error':True, 'error_message':msg}
if not res.ok:
msg = "Response not ok"
data.update({'has_error':True, 'error_message':msg})
if res.status_code >= 400:
msg = 'Error code: ' + str(res.status_code) + '\n' + data['errorCode']
data.update({'has_error':True, 'error_message': msg})
return data
When I have to do a DELETE request without body entity I have no problems but when I try to add one (when required by the server) I get an error message from the server telling that the body cannot be null as if no body has been sent. Any ideas why this might be happening? I'm using requests module and python 2.7.12. As far as I know data can be send in a DELETE request. Thanks!

There are problems with some clients and some servers when DELETE includes entity body: Is an entity body allowed for an HTTP DELETE request? for example & lots of search results.
Some servers (apparently) convert the DELETE into a POST, others simply perform the DELETE but drop the body. In your case, you've investigated that indeed, the body of a DELETE is dropped by the server & it has been suggested that you change the DELETE to POST.

Mmm... I can send a DELETE with body with Postman and works OK. But I cant get the same result with Requests 2.17.3
This is a issue related to Requests

Can django server send request to another server

I'm looking for help. My django server has instant messaging function achieved by django-socketio. If I run the server by cmd 'runserver_socketio' then there is no problems.
But now I want to run server by 'runfcgi' but that will make my socketio no working. So I want the socketio server handles the request which is conveyed by fcgi server. Can it work?
Following is my code:
def push_msg(msg):
params = urllib.urlencode({"msg":str(msg)})
'''headers = {"Content-type":"text/html;charset=utf8"}
conn = httplib.HTTPConnection("http://127.0.0.1:8000")
print conn
conn.request("POST", "/push_msg/", data=params, headers=headers)
response = conn.getresponse()
print response'''
h = httplib2.http()
print h
resp, content = h.request("http://127.0.0.1:8000/push_msg/", method="POST", body=params)
url(r'^push_msg/$', 'chat.events.on_message')
chat.events.on_message:
def on_message(request):
msg = request.POST.get('msg')
msg = eval(msg)
try:
print 'handle messages'
from_id = int(msg['from_id'])
to_id = int(msg['to_id'])
user_to = UserProfile.objects.get(id = msg['to_id'])
django_socketio.broadcast_channel(msg, user_to.channel)
if msg.get('type', '') == 'chat':
ct = Chat.objects.send_msg(from_id=from_id,to_id=to_id,content=data['content'],type=1)
ct.read = 1
ct.save()
except:
pass
return HttpResponse("success")
I have tried many times, but it can't work, why?

1) Of course Django can make request to another server
I have not much idea about django-socketio
and one more suggestion why you are using httplib you can use other advance version like httplib2 or requests apart from that Django-Piston is dedicated for REST request you can also try with that