Currently trying to manage a website with a Python script to control IoT object.
From what I discovered, control is doing in 2 times :
POST method to get an ID, needed to control the device.
POST method using the ID
The first one is working with this Python script and ID is displayed in the response.
import requests
url = 'http://local_IP/login.cgi'
payload = {'lgname': 'theLogin', 'lgpin': 'thePin'}
r = requests.post(url, data=payload)
For the second POST (to control the device when the user is logged in), I captured the command with Wireshark and here is the information:
POST /user/keyfunction.cgi HTTP/1.1\r\n
Content-Type: text/plain;charset=UTF-8\r\n
Referer: http://LOCAL_IP/login.cgi\r\n
and then I have:
Line-based text data: text/plain
sess=IDReceivedWithTheFirstPOST&comm=80&Data0=2&data2=18&data1=1
So basicaly, I need a way to do a POST in Python with this "Line-based text data: text/plain" but I have no idea how to deal with it.
Hope you'll be able to help me,
Thank you,
Baptiste
EDIT: If it can help anyone someday, here is my working code:
import requests
from collections import OrderedDict
session = requests.Session()
url = 'http://LOCAL_IP/login.cgi'
payload = {'lgname': 'User', 'lgpin': 'Password'}
r_login = session.post(url, data=payload)
with open('data.txt', 'w') as output:
output.write(r_login.text)
text = 'function getSession(){return'
with open('./data.txt', 'rb') as f:
for line in f:
if line.find(text) == 1:
id = line.split()[2][1:17]
print(id)
data = OrderedDict()
data['sess']=id
data['comm']=80
data['data0']=2
data['data2']=1
data['data1']=16
url = 'http://LOCAL_IP/user/keyfunction.cgi'
r_keyfunction = session.post(url, data=data)
with open('data2.txt', 'w') as output:
output.write(r_keyfunction.text)
Updated based on OPs results to use requests.Session()
using requests.Session() will capture all cookies and forward them on subsequent requests. It also pools connections and does lots of other cool things.
import requests
session = requests.Session()
payload = {'lgname': 'theLogin', 'lgpin': 'thePin'}
r_login = session.post('http://local_IP/login.cgi', data=payload)
# Figure out the ID here somehow
id_thing = 'IDReceivedWithTheFirstPOST'
payload = {
'sess': id_thing,
'comm': 80,
'Data0': 2,
'data2': 18,
'data1': 1
}
r_keyfunction = session.post('http://local_IP/user/keyfunction.cgi', params=payload)
# do something here
Related
I am trying to fill html form and get the intended result as i get when i fill manually. But I fail.
I am trying to fill the site https://www.desco.org.bd/ebill/login.php with value 32000001. So far my try is as below-
import requests
#LOGIN_URL = 'https://www.desco.org.bd/ebill/login.php'
#LOGIN_URL = 'https://www.desco.org.bd/ebill/authentication.php'
LOGIN_URL = 'https://www.desco.org.bd/ebill/billinformation.php'
payload = {
'username': '32000001',
'login':'Login',
'login':'true'
}
with requests.Session() as s:
p = s.post(LOGIN_URL, data=payload)#, verify=False)
# print the html returned or something more intelligent to see if it's a successful login page.
print (p.text)
I have found that login.php redirects to authentication.php and it further redirects to billinformation.php which delivers the true data i needed.
Thanks in advance.
N.B. I am not planning to use selenium since it is too slow for my case i.e. collect huge data from this site.
i am working for similar case, may be you would try using websockets:
import websockets
def process_function(message):
# process the message
def server(ws:str, path:int):
while True:
message_received = await ws.recv() # receive from ui
print(f'Msg [{message_received}]')
message_to_send = process_function(message)
await ws.send(message_to_send) # send feedback to ui
server = websockets.serve(server, '127.0.0.1', 5678) # set the html to run in the same ip:port
another try:
import json, requests
def do_auth(url):
headers = {"Content-Type": "application/json", "Accept":'*/*'}
body = json.dumps({'username': 'user', 'password': 'pass'})
r = requests.post(url=url, headers=headers, data=body, verify=False)
print(r.status_code);
d = json.loads(r.text);
print(d['access_token']);
print(d['refresh_token'])
return d['access_token'], d['refresh_token']
do_auth(url_auth) # authorization
requests.get(url_content, headers=headers, verify=False) # get data
We have Jira authenticated to Zapier, but it does not have tagging functionality, so I hacked up a simple python module below. However, it doesn't seem that I can reuse the already authenticated Jira account. Is there a way to hide the password somehow so it's not just clear text?
# jira_label and jira_url come from upstream zaps
# declaring output hash with defaults set
output = {'jira_label': jira_label}
### Python code
import requests
user = 'my_personal_user'
dpass = 'xxx' # <--- gotta do something about it
url1 = jira_url
pdata = '{"fields": {"labels": ["' + jira_label +'"]}}'
header1 = {'Content-Type': 'application/json'}
r = requests.put(url2, auth=(user, dpass), data=pdata, headers=header1)
Please use, Base64 encoding :
>>> import base64
>>> print base64.b64encode("mypassword")
bXlwYXNzd29yZA==
>>> print base64.b64decode("bXlwYXNzd29yZA==")
mypassword
With this your request will look like this:
r = requests.put(url2, auth=(user, base64.b64decode("bXlwYXNzd29yZA==")), data=pdata, headers=header1)
Lately I've come across a number of questions and articles very briefly covering using urllib, requests, mwapi, poster, and various other tools to either perform an HTTP POST, or working with the API to upload one or more files to a MediaWiki instance. Thus far, nothing has worked.
So, could someone kindly provide a simple code block that will reliably upload a file to such a Wiki? My preference is in Requests and/or Python 3, but at this point I'm pretty desperate and am open to almost anything.
Edit:
Per the request in the comments, below is the last bit of code I attempted. It completes with no errors, but of course no file is uploaded or any change to the Wiki logs.
from urllib.parse import quote
import requests
user = 'username'
passw = quote('password')
baseurl = 'http://127.0.0.1:8020/mediawiki/'
apiurl = baseurl + 'api.php'
login_params = '?action=login&lgname=%s&lgpassword=%s&format=json'% (user, passw)
# Login request
r1 = requests.post(apiurl+login_params)
login_token = r1.json()['login']['token']
# Login confirm
login_params2 = login_params+'&lgtoken=%s'% login_token
r2 = requests.post(apiurl+login_params2, cookies=r1.cookies)
# Get edit token
params3 = '?format=json&action=query&meta=tokens&continue='
r3 = requests.get(apiurl+params3, cookies=r2.cookies)
edit_token = r3.json()['query']['tokens']['csrftoken']
edit_cookie = r2.cookies.copy()
edit_cookie.update(r3.cookies)
# Upload file
with open('91.png', 'rb') as f:
headers = {'content-type': 'multipart/form-data'}
payload = {'action': 'upload', 'filename': 'Image', 'file': '91.png', 'token': edit_token}
files = {'files': f}
r4 = requests.post(apiurl, headers=headers, data=payload, files=files, cookies=edit_cookie)
I'm glad you got mwclient to work, but I think I can answer your preference of using just Python 3 and requests.
I was having major headaches doing the same thing, and finally got the following to work. I also posted it at https://www.mediawiki.org/wiki/API_talk:Upload#Python_with_requests but since this was a question I found while trying to solve my problem, I'll reproduce below...
You probably strictly speaking do not need a BotPassword, but it's a good idea.
import requests
api_url = 'https://project/w/api.php'
USER,PASS=u'BotUsername#Instancename',u'[[Special:BotPasswords]] password'
#Ensure bot instance is permissioned for createeditmovepage, uploadfile, uploadeditmovefile
FILENAME='/path/to/file'
REMOTENAME='remote_filename.ext'
USER_AGENT='Descriptive User Agent per [[:meta:User-Agent_policy]]'
# get login token and log in
payload = {'action': 'query', 'format': 'json', 'utf8': '',
'meta': 'tokens', 'type': 'login'}
r1 = requests.post(api_url, data=payload)
login_token=r1.json()['query']['tokens']['logintoken']
login_payload = {'action': 'login', 'format': 'json', 'utf8': '',
'lgname': USER, 'lgpassword': PASS, 'lgtoken': login_token}
r2 = requests.post(api_url, data=login_payload, cookies=r1.cookies)
cookies=r2.cookies.copy()
# We have now logged in and can request edit tokens thusly:
def get_edit_token(cookies):
edit_token_response=requests.post(api_url, data={'action': 'query',
'format': 'json',
'meta': 'tokens'}, cookies=cookies)
return edit_token_response.json()['query']['tokens']['csrftoken']
# Now actually perform the upload:
upload_payload={'action': 'upload',
'format':'json',
'filename':REMOTENAME,
'comment':'<MY UPLOAD COMMENT>',
'text':'Text on the File: page... description, license, etc.',
'token':get_edit_token(cookies)}
files={'file': (REMOTENAME, open(FILENAME,'rb'))}
headers={'User-Agent': USER_AGENT}
upload_response=requests.post(api_url, data=upload_payload,files=files,cookies=cookies,headers=headers)
To me, it seems weird that your last code didn't error. Assuming that you set up MediaWiki with the standard configuration, api.php is in baseurl/w/api.php, not baseurl/api.php. This means that you requested the wrong page all the time. Try again, but this time replace apiurl = baseurl + 'api.php' with apiurl = baseurl + 'w/api.php'.
It turns out that the one I hadn't tried worked. Using the documented examples for mwclient, uploading was successful.
I use requests.api lib to send post request. What I want is to send multidimensional POST data and I always come up short with this code:
import requests
url = 'http://someurl.com';
request_data = {}
request_data['someKey'] = 'someData'
request_data['someKeytwo'] = 'someData2'
request_data['requestData'] = {'someKey3': 'someData3'}
login = requests.post(url, data=login_data)
On the receiving end i get a POST with "requestData" => "someKey3" instead of "requestData" => ["someKey3" => 'someData3']
How do I send the correct POST?
The correct answer for my question is:
import requests
url = 'http://someurl.com';
request_data = {}
request_data['requestData[someKey3]'] = 'someData3'
login = requests.post(url, data=request_data)
Simply use:
import json
login = requests.post(rul, data=json.dumps(login_data))
This way you receive a json on the the receiving side.
I have a simple website crawler, it works fine, but sometime it stuck because of large content such as ISO images, .exe files and other large stuff. Guessing content-type using file extension is probably not the best idea.
Is it possible to get content-type and content length/size without fetching the whole content/page?
Here is my code:
requests.adapters.DEFAULT_RETRIES = 2
url = url.decode('utf8', 'ignore')
urlData = urlparse.urlparse(url)
urlDomain = urlData.netloc
session = requests.Session()
customHeaders = {}
if maxRedirects == None:
session.max_redirects = self.maxRedirects
else:
session.max_redirects = maxRedirects
self.currentUserAgent = self.userAgents[random.randrange(len(self.userAgents))]
customHeaders['User-agent'] = self.currentUserAgent
try:
response = session.get(url, timeout=self.pageOpenTimeout, headers=customHeaders)
currentUrl = response.url
currentUrlData = urlparse.urlparse(currentUrl)
currentUrlDomain = currentUrlData.netloc
domainWWW = 'www.' + str(urlDomain)
headers = response.headers
contentType = str(headers['content-type'])
except:
logging.basicConfig(level=logging.DEBUG, filename=self.exceptionsFile)
logging.exception("Get page exception:")
response = None
Yes.
You can use the Session.head method to create HEAD requests:
response = session.head(url, timeout=self.pageOpenTimeout, headers=customHeaders)
contentType = response.headers['content-type']
A HEAD request similar to GET request, except that the message body would not be sent.
Here is a quote from Wikipedia:
HEAD
Asks for the response identical to the one that would correspond to a GET request, but without the response body. This is useful for retrieving meta-information written in response headers, without having to transport the entire content.
Use requests.head() for this. It will not return the message body. You should use head method if you are interested only in the headers. Check this link for detail.
h = requests.head(some_link)
header = h.headers
content_type = header.get('content-type')
Sorry, my mistake, I should read documentation better. Here is the answer:
http://docs.python-requests.org/en/latest/user/advanced/#advanced (Body Content Workflow)
tarball_url = 'https://github.com/kennethreitz/requests/tarball/master'
r = requests.get(tarball_url, stream=True)
if int(r.headers['content-length']) > TOO_LONG:
r.connection.close()
# log request too long
Because requests.head() does NOT auto redirect, so a URL is redirected, requests.head() will get 0 for Content-Length. So make sure allow_redirects=True is added.
r = requests.head(url, allow_redirects=True)
length = r.headers['Content-Length']
Refer to Requests Redirection And History