We Keep Coding

What is blocking my requests from successfully logging in?

So im using requests to login into netflix (education purposes only) and when i print the response text it keeps bringing back the same thing even when i have a valid netflix account, what is blocking my entrance to login?
from fake_useragent import UserAgent as ua
from bs4 import BeautifulSoup
s = requests.Session()
login = s.get("https://www.netflix.com/login", headers={"User-Agent": ua().random})
soup = BeautifulSoup(login.text, "html.parser")
loginForm = soup.find('form')
authURL = loginForm.find('input', {"name": "authURL"}).get('value')
headers = {"user-agent": ua().random,"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9", "accept-language": "en-US,en;q=0.9", "accept-encoding": "gzip, deflate, br", "referer": "https://www.netflix.com/login", "content-type": "application/x-www-form-urlencoded","cookie":""}
data = {"userLoginId:": "validnetflix", "password": "netflix123", "rememberMeCheckbox": "true", "flow": "websiteSignUp", "mode": "login", "action": "loginAction", "withFields": "rememberMe,nextPage,userLoginId,password,countryCode,countryIsoCode", "authURL": authURL, "nextPage": "https://www.netflix.com/browse","countryCode": "+1","countryIsoCode": "US"}
request = s.post("https://www.netflix.com/login", headers=headers, data=data)
cookie = dict(flwssn=s.get("https://www.netflix.com/login", headers ={"User-Agent": ua().random}).cookies.get("flwssn"))
if 'Sorry, we can\'t find an account with this email address. Please try again or' or 'Incorrect password' in request.text:
print("bad")
else:
print("good")
print(request.text)```

Scrapy/Python XMLHTTPRequest

I built a scraper that works up to a point: It navigates to a list of records, parses the records to key ones for further crawling, goes to those individual records but is unable to parse tables in the records because they are loaded via JavaScript. JavaScript issues a POST request (xmr) to populate them. So if JavaScript is not enabled it returns something like 'No records found.'
So I read this question: Link
I inspected Request Headers with browser dev tools. Headers include:
fetch("https://example.com/Search/GridQuery?query=foo", {
"headers": {
"accept": "text/plain, */*; q=0.01",
"accept-language": "en-US,en;q=0.9,es;q=0.8",
"cache-control": "no-cache",
"content-type": "application/x-www-form-urlencoded",
"pragma": "no-cache",
"sec-fetch-dest": "empty",
"sec-fetch-mode": "cors",
"sec-fetch-site": "same-origin",
"x-requested-with": "XMLHttpRequest"
},
"referrer": "https://example.com/SiteSearch/Search?query=bar",
"referrerPolicy": "no-referrer-when-downgrade",
"body": "page=1&size=10&useFilters=false",
"method": "POST",
"mode": "cors",
"credentials": "include"
});
The browser does indicate a cookie although not output by copying fetch...
I then tried this:
url = response.urljoin(response.css('div#Foo a::attr(href)').get())
yield Request(url=url,
method='POST',
body='{"filters": ["page": "1", "size": "10", "useFilters": "False"]}',
headers={'x-requested-with': 'XMLHttpRequest'},
callback=self.parse_table)
I get a response but it still says 'No records found'. So the POST request is not working right.
Do I need to put everything in the request header? How do I determine what must be included? Are cookies required?

I did not test this, since you didn't provide real url, but I see a couple of problems there.
Note that content type is application/x-www-form-urlencoded, and you are sending JSON object in the body (that's for application/json)
Instead, you should be sending FormRequest:
url = "https://example.com/Search/GridQuery?query=foo"
form_data = {"page": "1", "size": "10", "useFilters": "False"}
yield FormRequest(url, formdata=form_data, callback=self.parse_table)
Or simply add parameters as query parameters in the URL (still POST request, just omit the body).
url="https://example.com/Search/GridQuery?query=foo&page=1&size=10&useFilters=False"
Either way, you do not need that "filters":[], just use simple key-value object.

HTTP 415 Unsupported Media Type client when sending data to server

Is it that my server is not supporting this request or I am sending some wrong data which doesn't seem to be the case.
# importing the requests library
import requests
# defining the api-endpoint
API_ENDPOINT = "http://10.176.14.170:5000/api/du/v1"
# data to be sent to api
data = {"name":"du1"}
# sending post request and saving response as response object
r = requests.post(url = API_ENDPOINT, data = data)
print(r)
Response:
<Response [415]>
Success from a web CLI that is running on same server.
post /api/du/v1
POST api/du/v1 {"name":"du1"}
I also get success when I send the POST request using curl command from same server

While the original curl command was not included, though it was stated that the data was sent via POST to the server as is. I am going to assume that the curl -d '{"name":"du1"}' was used along with the appropriate headers being sent. Now to demonstrate this, the following command is used:
$ curl http://httpbin.org/post -d '{"name":"du1"}' -H 'Content-Type: application/json'
{
"args": {},
"data": "{\"name\": \"du1\"}",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Content-Length": "15",
"Content-Type": "application/json",
"Host": "httpbin.org",
"User-Agent": "curl/7.61.0"
},
"json": {
"name": "du1"
},
"url": "http://httpbin.org/post"
}
Note the bolded data key - that's what likely was being read by your application. On the contrary, doing the same thing as done with the provided usage of requests at the same endpoint:
>>> import requests
>>> r = requests.post('http://httpbin.org/post', data={"name":"du1"})
>>> print(r.text)
{
"args": {},
"data": "",
"files": {},
"form": {
"name": "du1"
},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Content-Length": "8",
"Content-Type": "application/x-www-form-urlencoded",
"Host": "httpbin.org",
"User-Agent": "python-requests/2.21.0"
},
"json": null,
"url": "http://httpbin.org/post"
}
Note how the requests had been decoded into a form simply because a) the data submitted was a raw dictionary, and not a JSON encoded string and b) the header was not supplied. To fix this, simply encode the data and supply the appropriate headers:
>>> import requests
>>> import json
>>> data = json.dumps({"name":"du1"})
>>> r = requests.post(
... 'http://httpbin.org/post', data=data,
... headers={'Content-Type': 'application/json'})
>>> print(r.text)
{
"args": {},
"data": "{\"name\": \"du1\"}",
"files": {},
"form": {},
...
"json": {
"name": "du1"
},
...
}
Note that the server is now able to interpret the provided data, and can decode that as JSON.

Send body of POST request

I inspected how a request was sent to a website in firefox:
(Unfortunately I had to change the website URL to a fake one to prevent the server form being requested too much).
I tried to do this request in python:
import requests
import json
seq = 'ATGGCAGACTCTATTGAGGTC'
url = 'http://www.test.com'
body = {'QUERY': seq}
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
r = requests.post(url, data=json.dumps(body), headers=headers)
print(r.text)
However when doing this the website says: Empty gene sequence passed for blast analysis. Please enter a valid gene sequence. So that means that the sequence (i.e. QUERY) is not sent correctly to the server. What am I missing here?
(P.s. hopefully missing of the website is not a problem to answer this question, if it is please let me know maybe I can ask to mention their website)

I am guessing the string / sequence that you are submitting to that particular website is the problem. I ran your sample code against a POST accepting website:
import requests
import json
seq = 'ATGGCAGACTCTATTGAGGTC'
url = 'http://httpbin.org/post'
body = {'QUERY': seq}
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
r = requests.post(url, data=json.dumps(body), headers=headers)
print(r.text)
And got this result, which shows your query properly formed:
{
"args": {},
"data": "{\"QUERY\": \"ATGGCAGACTCTATTGAGGTC\"}",
"files": {},
"form": {},
"headers": {
"Accept": "text/plain",
"Accept-Encoding": "gzip, deflate",
"Content-Length": "34",
"Content-Type": "application/json",
"Host": "httpbin.org",
"User-Agent": "python-requests/2.22.0"
},
"json": {
"QUERY": "ATGGCAGACTCTATTGAGGTC"
},
"origin": "2.122.222.8, 2.122.222.8",
"url": "https://httpbin.org/post"
}

Are you sure the it's supposed to be "QUERY=<>?" Cause it could be incorrect formatting of the body. Normally it's in JSON format as in "title: information." Note the ':' rather than the '='

curl vs python "requests" when hitting APIs

I am trying to hit the Bitbucket API for my account, and a successful attempt looks like:
curl --user screename:mypassword https://api.bitbucket.org/1.0/user/repositories
in the command line. In python, I try:
import requests
url = 'https://api.bitbucket.org/1.0/user/repositories'
then
r = requests.post(url, data={'username': myscreename, 'password':mypassword})
and
r = requests.post(url, data="myscreename:mypassword")
and
r = requests.post(url, data={"user": "myscreename:mypassword"})
all get 405 error. The API is https://confluence.atlassian.com/bitbucket/rest-apis-222724129.html.
I wonder:
What am I doing wrong in the requests version, they all look similar to my curl attempt
What is the difference between requesting with curl and python requests module? What general pattern can I recognize when reading an API with a curl example and then writing it in python?
Thank you
ANSWER:
it required the right headers
https://answers.atlassian.com/questions/18451025/answers/18451117?flashId=-982194107
UPDATE:
# ===============
# get user
# ===============
import requests
import json
# [BITBUCKET-BASE-URL], i.e.: https://bitbucket.org/
url = '[BITBUCKET-BASE-URL]/api/1.0/user/'
headers = {'Content-Type': 'application/json'}
# get user
# [USERNAME], i.e.: myuser
# [PASSWORD], i.e.: itspassword
r = requests.get(url, auth=('[USERNAME]', '[PASSWORD]'), headers=headers)
print(r.status_code)
print(r.text)
#print(r.content)

Here's a way to do basic HTTP auth with Python's requests module:
requests.post('https://api.bitbucket.org/1.0/user/repositories', auth=('user', 'pass'))
With the other way you're passing the user/pass through the request's payload, which is not desired since HTTP basic auth has its own place in the HTTP protocol.
If you want to "see" what's happening under the hood with your request I recommend using httpbin:
>>> url = "http://httpbin.org/post"
>>> r = requests.post(url, data="myscreename:mypassword")
>>> print r.text
{
"args": {},
"data": "myscreename:mypassword",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Content-Length": "22",
"Host": "httpbin.org",
"User-Agent": "python-requests/2.5.1 CPython/2.7.6 Darwin/14.3.0"
},
"json": null,
"origin": "16.7.5.3",
"url": "http://httpbin.org/post"
}
>>> r = requests.post(url, auth=("myscreename", "mypassword"))
>>> print r.text
{
"args": {},
"data": "",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Authorization": "Basic bXlzY3JlZW5hbWU6bXlwYXNzd29yZA==",
"Content-Length": "0",
"Host": "httpbin.org",
"User-Agent": "python-requests/2.5.1 CPython/2.7.6 Darwin/14.3.0"
},
"json": null,
"origin": "16.7.5.3",
"url": "http://httpbin.org/post"
}
And with curl:
curl -X POST --user myscreename:mypassword http://httpbin.org/post
{
"args": {},
"data": "",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Authorization": "Basic bXlzY3JlZW5hbWU6bXlwYXNzd29yZA==",
"Host": "httpbin.org",
"User-Agent": "curl/7.37.1"
},
"json": null,
"origin": "16.7.5.3",
"url": "http://httpbin.org/post"
}
Notice the resemblance between the last python example and the cURL one.
Now, getting right the API's format is another story, check out this link: https://answers.atlassian.com/questions/94245/can-i-create-a-bitbucket-repository-using-rest-api
The python way should be something like this:
requests.post('https://api.bitbucket.org/1.0/repositories', auth=('user', 'pass'), data = "name=repo_name")

With python3, you can use json={...} instead of data={...}, and it will set the header automatically to application/json:
import requests
url = 'https://api.bitbucket.org/1.0/user/repositories'
data = {
'data1': 'asd',
'data2': 'asd'
}
req = requests.post(url, auth=('user', 'password'), json = data)
data = req.json()
# data['index']