Currently I have an HTML form which chooses the file and upload it to server.
How to do it without HTML form.
<html>
<head></head>
<body>
<form action="/upload" method="post" enctype="multipart/form-data">
Select a file: <input type="file" name="uploadinc" />
<input type="submit" value="Start upload" />
</form>
</body>
</html>
And my bottle server contains the following code to upload.
#route('/UploadFiles', method='POST')
def UploadFiles():
print "inside upload files"
uploadinc = request.files.get('uploadinc')
uploadinc.save("/home/user/files/"+uploadinc.filename)
I want to directly save the file without HTML UI.
Like..
request.files.get("file location in local machine if it is fixed(C:\\a.txt)")
But it is getting as none. How to do it?
I am able to call the Rest API from a rest client like this.
How to do this call programatically ?
You may try Requests lib: POST a Multipart-Encoded File
How to do this call programatically ?
Modified example from Requests lib documentation:
import requests
url = 'http://10.208.53.89:7778/UploadFiles'
multiple_files = [
('images', ('foo.png', open('foo.png', 'rb'), 'image/png')),
('images', ('bar.png', open('bar.png', 'rb'), 'image/png'))
]
r = requests.post(url, files=multiple_files)
You want to upload files from the command line, instead of in a browser? Just use curl:
curl -F "image=#foo.png" -F "image2=#bar.png" http://localhost:8888/uploadFiles
Source.
Related
I'm looking for a way to get files such as the one in this link, which can be downloaded by clicking a "download" button. I couldn't find a way despite reading many posts that seemed to be relevant.
The code I got so far:
import requests
from bs4 import BeautifulSoup as bs
with open('ukb49810.html', 'r') as f:
html = f.read()
index_page = bs(html, 'html.parser')
for i in index_page.find_all('a', href=True)[2:]:
if 'coding' in i['href']:
file = requests.get(i['href']).text
download_page = bs(file, 'html.parser').find_all('a', href=True)
From the download_page variable I got "URLs" with the code
for ii in download_page:
print(ii['href'])
which printed
http://
index.cgi
browse.cgi?id=9&cd=data_coding
search.cgi
catalogs.cgi
download.cgi
https://bbams.ndph.ox.ac.uk/ams/resApplications
help.cgi?cd=data_coding
field.cgi?id=22001
field.cgi?id=22001
label.cgi?id=100313
field.cgi?id=31
field.cgi?id=31
label.cgi?id=100094
I tried to use these supposedly-URLs to compose the download URL but the link I got didn't work.
Thanks.
None of these links are to the download page. If you view source on the page, you will see how the download is done:
<form method="post" action="codown.cgi">
<input type="hidden" name="id" value="9"></td><td>
<input class="btn_glow" type="submit" value="Download">
</form>
So you would need to submit a POST request to codown.cgi with your value, something like:
curl --request POST \
--url https://biobank.ndph.ox.ac.uk/showcase/codown.cgi \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data id=9
However the thing I would suggest is searching the site for a more convenient option than scraping. On something like this it's likely to available (and indeed, it is in this case!)
It looks like all of the data you can get from that page (and its variants) can be obtained from the Downloads->Schema page, and those all offer simple download links you can use, eg:
https://biobank.ndph.ox.ac.uk/showcase/schema.cgi?id=5
I'm trying to get the HTML content of a password protected site using Ghost.py.
The web server I have to access, has the following HTML code (I cut it just to the important parts):
URL: http://192.168.1.60/PAGE.htm
<html>
<head>
<script language="JavaScript">
function DoHash()
{
var psw = document.getElementById('psw_id');
var hpsw = document.getElementById('hpsw_id');
var nonce = hpsw.value;
hpsw.value = MD5(nonce.concat(psw.value));
psw.value = '';
return true;
}
</script>
</head>
<body>
<form action="PAGE.HTM" name="" method="post" onsubmit="DoHash();">
Access code <input id="psw_id" type="password" maxlength="15" size="20" name="q" value="">
<br>
<input type="submit" value="" name="q" class="w_bok">
<br>
<input id="hpsw_id" type="hidden" name="pA" value="180864D635AD2347">
</form>
</body>
</html>
The value of "#hpsw_id" changes every time you load the page.
On a normal browser, once you type the correct password and press enter or click the "submit" button, you land on the same page but now with the real contents.
URL: http://192.168.1.60/PAGE.htm
<html>
<head>
<!–– javascript is gone ––>
</head>
<body>
Welcome to PAGE.htm content
</body>
</html>
First I tried with mechanize but failed, as I need javascript. So now I´m trying to solve it using Ghost.py
My code so far:
import ghost
g = ghost.Ghost()
with g.start(wait_timeout=20) as session:
page, extra_resources = session.open("http://192.168.1.60/PAGE.htm")
if page.http_status == 200:
print("Good!")
session.evaluate("document.getElementById('psw_id').value='MySecretPassword';")
session.evaluate("document.getElementsByClassName('w_bok')[0].click();", expect_loading=True)
print session.content
This code is not loading the contents correctly, in the console I get:
Traceback (most recent call last): File "", line 8, in
File
"/usr/local/lib/python2.7/dist-packages/ghost/ghost.py", line 181, in
wrapper
timeout=kwargs.pop('timeout', None)) File "/usr/local/lib/python2.7/dist-packages/ghost/ghost.py", line 1196, in
wait_for_page_loaded
'Unable to load requested page', timeout) File "/usr/local/lib/python2.7/dist-packages/ghost/ghost.py", line 1174, in
wait_for
raise TimeoutError(timeout_message) ghost.ghost.TimeoutError: Unable to load requested page
Two questions...
1) How can I successfully login to the password protected site and get the real content of PAGE.htm?
2) Is this direction the best way to go? Or I'm missing something completely which will make things work more efficiently?
I'm using Ubuntu Mate.
This is not the answer I was looking for, just a work-around to make it work (in case someone else has a similar issue in the future).
To skip the javascript part (which was stopping me to use python's request), I decided to do the expected hash on python (and not on web) and send the hash as the normal web form would do.
So the Javascript basically concatenates the hidden hpsw_id value and the password, and makes a md5 from it.
The python now looks like this:
import requests
from hashlib import md5
from re import search
url = "http://192.168.1.60/PAGE.htm"
with requests.Session() as s:
# Get hpsw_id number from website
r = s.get(url)
hpsw_id = search('name="pA" value="([A-Z0-9]*)"', r.text)
hpsw_id = hpsw_id.group(1)
# Make hash of ID and password
m = md5()
m.update(hpsw_id + 'MySecretPassword')
pA = m.hexdigest()
# Post to website to login
r = s.post(url, data=[('q', ''), ('q', ''), ('pA', pA)])
print r.content
Note: the q, q and pA are the elements that the form (q=&q=&pA=f08b97e5e3f472fdde4280a9aa408aaa) is sending when I login normally using internet browser.
If someone however knows the answer of my original question I would be very appreciated if you post it here.
I would like to submit a form on a webpage.
The page has however several forms :
<form method="post" action="https://mywebsite.com/pageA" id="order" class="order ajaxForm">
<input type="text" class="decimal" name="value" id="fieldA" value="0" />
</label>
</form>
<form method="post" action="https://mywebsite.com/pageB" id="previousorder" class="order ajaxForm">
<input type="text" class="decimal" name="value" id="fieldB" value="0" />
</label>
</form>
Is there an easy way to trigger a specific form using python & request ?
I'd go with some more advanced tools like mechanize or MechanicalSoup. The latter is actually based on requests internally (I assume you meant requests package by "request"). Both of these tools allow to "select a desired form" and then submit it specifying the required parameters.
For instance, submitting the order form with MechanicalSoup would look something like this:
import mechanicalsoup
browser = mechanicalsoup.StatefulBrowser()
browser.open("https://yourwebsite.com")
# Fill-in the order form
browser.select_form('#order')
browser["value"] = "100"
browser.submit_selected()
You have to look at the DevTools Network tab while posting a form.
Every form will have different request url and post parameters. Generally, what you will need to do with requests is something like that:
req = requests.post('https://mywebsite.com/pageB',
data = {'fieldB':'value_you_want_to_submit'})
But better first investigate it with DevTools.
Try something like this: (prob need to make some modifications but it will be close to what you want this example is for login form):
install lxml
import requests
from lxml import html
payload = {
"username": "<USER NAME>",
"password": "<PASSWORD>",
"csrfmiddlewaretoken": "<CSRF_TOKEN>"
}
sessionReq = requests.session()
login_url = "https://example.be/account/login.php"
result = sessionReq.get(login_url)
tree = html.fromstring(result.text)
authenticity_token = list(set(tree.xpath("//input[#name='csrfmiddlewaretoken']/#value")))[0]
result = sessionReq.post(login_url,data = payload, headers = dict(referer = login_url)
url = 'https://bitbucket.org/dashboard/overview'
I hope this helps you :)
I am working in a Linux VNC. It has an older python version 2.7 and so I couldn't use any of predefined GUI packages which are not available in this version. So I need an alternative to do that. Also with these below codes, the code is getting redirected to html page but at the same time the cgi script also runs and prints the output in the terminal as "None" value. So I want the html part to be done first and after giving the input in webpage and clicking the submit button, i must get that value in terminal. How can i do that? Also cgi-bin folder is not there, so on clicking the submit button, getting a File Not Found error as well. Please help me through this...
test.py
import webbrowser
def func1():
f = open('helloworld.html','w')
message = """<html>
<head></head>
<body>
<form name="get" action="/cgi-bin/cgi_test.py" method="get">
Name : <input type="text" name="nm"/></br>
<input type="submit" value="Submit"/>
</form>
</body>
</html>"""
f.write(message)
f.close()
webbrowser.open('helloworld.html')
cgi_test.py
import test
import cgi
test.func1()
form=cgi.FieldStorage()
n=form.getvalue('nm')
print (n)
I have a critical issue. I would like integrate my application with another much older application. This service is simply a web form, probably behind a framework (I think ASP Classic maybe). I have an action URL, and I have the HTML code for replicating this service.
This is a piece of the old service (the HTML page):
<FORM method="POST"
url="https://host/path1/path2/AdapterHTTP?action_name=myactionWebAction&NEW_SESSION=true"
enctype="multipart/form-data">
<INPUT type="text" name="AAAWebView-FormAAA-field1" />
<INPUT type="hidden" name="AAAWebView-FormAAA-field2" value="" />
<INPUT type="submit" name="NAV__BUTTON__press__AAAWebView-FormAAA-enter" value="enter" />
</FORM>
My application should simulate form submission of this old application from code-behind with Python. For now, I didn't have so much luck.
For now I do this
import requests
payload = {'AAAWebView-FormAAA-field1': field1Value, \
'AAAWebView-FormAAA-field2': field2Value, \
'NAV__BUTTON__press__AAAWebView-FormAAA-enter': "enter"
}
url="https://host/path1/path2/AdapterHTTP?action_name=myactionWebAction&NEW_SESSION=true"
headers = {'content-type': 'multipart/form-data'}
r = requests.post(url, data=payload, headers=headers)
print r.status_code
I receive a 200 HTTP response code, but if I click on submit button on the HTML page, the action saves the values, but my code does not do the same. How do I fix this problem?
The owner of an old application sent me this Java exception log. Any ideas?
org.apache.commons.fileupload.FileUploadException: the request was rejected because no multipart boundary was found
Try passing an empty dictionary as files with requests.post. This will properly construct a request with multipart boundary I think.
r = requests.post(url, data=payload, headers=headers, files={})