I apologize if this question is real entry level type programmer..
But if I am posting data with the requests package, is the data secure? OR while the http message is 'in the air' between my PC and http bin; could someone intercept/replicate what I am doing?... Basically corrupt my data and create havoc for what I am trying to do...
import time, requests
stuff = {}
stamp = time.time()
data = 120.2
stuff['Date'] = stamp
stuff['meter_reading'] = data
print("sending this dict",stuff)
r = requests.post('https://httpbin.org/post', data=stuff)
print("Status code: ", r.status_code)
print("Printing Entire Post Request")
print(r.text)
With the script above on the level of security would it matter if I am posting to a server that is running http or https? The code above is similar to my real world example (that I run on a rasp pi scheduled task) where I am posting data with a time stamp to an http (Not https) server (flask app on pythonanywhere cloud site) which then saves the data to sql. This data can then be rendered thru typical javacript front end web development...
Thanks for any advice I am still learning how to make this 'secure' on the data transfer from the rasp to to cloud server.. Asking about client side web browsing security to view the data that has already been transferred maybe a totally different question/topic..
This is a question about protocols mainly. The HTTP protocol is less secure as someone can 'listen' to what you are sending over it. That's why you should always use the newer HTTPS protocol, since it uses TLS (encrypted) connection. You can read more about it e.g. here.
Requests verifies SSL certificates for HTTPS requests, just like a web browser. By default, SSL verification is enabled, and Requests will throw a SSLError if it’s unable to verify the certificate.
https://requests.readthedocs.io/en/master/user/advanced/#ssl-cert-verification
If you're transmitting data that you do not want others to be able to see, use https. For this use case I can't imagine it would matter too much.
Related
i need a guide to establish a connection to a public web service, send request to it and get response back. for example this web service:
http://webservices.oorsprong.org/websamples.countryinfo/CountryInfoService.wso?WSDL
i've tried to test this API with SoapUI application. this API has a bunch of methods such as sending you a country's capital by getting it's ISO name (send IR as request and get Tehran as respond). now somehow i want to do this through Python. i want to have access to all it's methods and send requests. only by connecting to API's address or any other way (maybe by loading each method's XML code and running it in Python? idk). is it possible? any guide?
I'd recommend checking out this library:
https://requests.readthedocs.io/en/latest/
You can send HTTP requests to URL endpoints, parse out data, etc. Hope this helps!
I have successfully used suds and SOAPpy in the past. I see people recommend Zeep nowadays but I haven't used it.
I am developing an application that spends tokens to execute an operation.
each user has a limited amount of token and they will earn it as time goes by.
the number of tokens of each user is stored in a database as an integer, when the client application wants to execute the operation it sends an HTTP request to the server and on the server it checks the database if it has the necessary number of tokens, if the user has the necessary tokens, the server application subtracts the tokens from the database and sends an HTTP response like the following:
body_response:
{
"spent_tokens": [amount]
"can_execute": true
}
my question is if this is too insecure or not?, if it is possible to falsify the HTTP response?
and if it is unsure what other method I could use instead.
For the server application I am using python with flask and C# WPF in the client application making the HTTP requests with the HTTPClient standard library
thank you for your help
Yes, it is possible to spoof requests in both HTTP and HTTPS using tools such as Burp or curl. Not only is it possible, it is also extremely simple. Forging malicious HTTP/HTTPS requests is the bread and butter of any bad actor looking for flaws in your products. You should never trust data provided by the client. All sensitive state should be managed by a trusted server.
I'm making a bunch of webcalls to scrape for data utilizing Joblib and Requests which work fine on my own personal computer but when I moved the code to Amazon EC2 t3a.micro service, I run into a request error 444. requests.exceptions.HTTPError: 444 Client Error: Unknown for url. I believe this is because I am making too many requests in a limited time frame but was not sure how to exactly fix this problem or what the problem is.
According to this httpstatus page the error comes from:
A non-standard status code used to instruct nginx to close the
connection without sending a response to the client, most commonly
used to deny malicious or malformed requests.
So if you are 100% sure you use the same payload on EC2 and it is also correct, then it could indeed be the amount for requests. The way you limit the amount of requests is usually dependent on the code you wrote. I have seen people use the sleep(5) code to avoid flooding the server with requests. Not sure if it would work in your specific case though.
I've developed a fairly simple web service using Flask (Python 2.7, current Flask and dependencies), where clients POST a hunk of JSON to the server and get a response.
This is working 100% of the time when no authentication is enabled; straight up POST to my service works great.
Adding HTTP Digest authentication to the endpoint results in the client producing a 'Broken Pipe' error after the 401 - Authentication Required response is sent back... but only if the JSON hunk is more than about 22k.
If the JSON hunk being transmitted in the POST is under ~22k, the client gets its 401 response and cheerfully replies with authentication data as expected.
I'm not sure exactly what the size cut-off is... the largest I've tested successfully with is 21766 bytes, and the smallest that's failed is 43846 bytes. You'll note that 32k is right in that range, and 32k might be a nice default size for a buffer... and this smells like a buffer size problem.
The problem has been observed using a Python client (built with the 'requests' module) and a C++ client (using one of Qt's HTTP client classes). The problem is also observed both when running the Flask app "stand-alone" (that is, via app.run()) and when running behind Apache via mod_wsgi. No SSL is enabled in either case.
It goes as follows:
your client POSTs JSON data without authentication
server receives the request (not necessarily in one long chunk, it might come in parst)
server evaluates the requests and finds it is not providing credentials, so decides stopping processing the request and replies 401.
With short POST size server consumes all and does not have time to break the POST requests in the middle. With growing POST size the chances to interrupt unauthorized POST request is higher.
You client has two options:
Either start sending credentials right away.
Or try / catch broken pipe and react to it by forming proper Digest based request.
The first feeling is, something is broken, but it is rather reasonable approach - imagine, someone could post huge POST request, consume resources on your server while not being authorized to do so. The reaction of the server seems reasonable in this context.
Originally, I tried to post an ajax request from my client side to a third party url, but it seems that the browser have security issues with that. I thought about sending an ajax to the server side, from there to send a GET request to the third party, get the response and send it back to the client side. How can I do that with flask?
Install the requests module (much nicer than using urllib2) and then define a route which makes the necessary request - something like:
import requests
from flask import Flask
app = Flask(__name__)
#app.route('/some-url')
def get_data():
return requests.get('http://example.com').content
Depending on your set up though, it'd be better to configure your webserver to reverse proxy to the target site under a certain URL.
Flask alone does not have this capability, but it is a simple matter to write a request handler that makes a request to another server using an HTTP client library and then return that response.
# third-party HTTP client library
import requests
# assume that "app" below is your flask app, and that
# "Response" is imported from flask.
#app.route("/proxy-example")
def proxy_example():
r = requests.get("http://example.com/other-endpoint")
return Response(
r.text,
status=r.status_code,
content_type=r.headers['content-type'],
)
However, this will not achieve exactly the same result as you might expect from a client-side request. Since your server cannot "see" any cookies that the client browser has stored for the target site, your proxied request will be effectively anonymous and so, depending on the target site, may fail or give you a different response than you'd get requesting that resource in the browser.
If you have a relationship with the third-party URL (that is, if you control it or are able to work with the people who do) they can give access for cross-domain requests in the browser using CORS (which is only supported in modern browsers) or JSON-P (an older workaround that predates CORS).
The third-party provider could also give you access to the data you want at an endpoint that is designed to accept requests from other servers and that provides a mechanism for you to authenticate your app. The most popular protocol for this is OAuth.
As the other answers have stated using the requests module for python would be the best way to tackle this from the coding perspective. However as the comments mentioned (and the reason I came to this question) this can give an error that the request was denied. This error is likely cause by SELinux.
To check if this is the issue first make sure SELinux is enabled with this command:
sestatus
If 'Current Mode' is 'enforcing' then SELinux is enabled.
Next get the current bool values that apply directly to apache with this command:
getsebool -a | grep httpd
Look for the setting 'httpd_can_network_connect' this determines if apache is allowed to make TCP requests out to the network. If it is on then all apache TCP requests will be allowed. To turn it on run the following as root:
setsebool -P httpd_can_network_connect 1
If you only need database access (I had this problem before which is why I suspected SELinux here) then it would probably be better to only turn on 'httpd_cna_network_connect'.
The purpose of this policy is that if a hacker was to hijack your apache server they would not be able to get out through the server to the rest of your internal network.
This probably would've been better as a comment but I don't have enough rep..
Sources:
https://tag1consulting.com/blog/stop-disabling-selinux
https://wiki.centos.org/TipsAndTricks/SelinuxBooleans