I have basic script for uploading video files and small pipeline to run its
The script is https://developers.google.com/youtube/v3/guides/uploading_a_video
It been worked fine couple of last months pretty fine, but I started getting 404 error from api server
There is the rich output
/usr/bin/python2.7 push_video.py --file="/tmp/HofQ.mp4" --description="File name" --keywords="test" --category="22" --privacyStatus="private"
connect: (www.googleapis.com, 443)
send: 'GET /discovery/v1/apis/youtube/v3/rest HTTP/1.1\r\nHost: www.googleapis.com\r\naccept-encoding: gzip, deflate\r\nauthorization: Bearer ya29.a0AfH6SMCzOovQCiAa0I-Mrz7oD-wWeikotEGLIRzk2Z6D2N7umFciU5RDQWZMKtBOXu-7gI_-v_ArhcNtTE9kNPnYHVYi32697vPBUC3be0sAi-kPHN9Utpi00gS1KDpa5gko8ZR_D_euZSzM_3VJrinOMe1jWIsS-WY\r\nuser-agent: Python-httplib2/0.17.2 (gzip)\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json; charset=UTF-8
header: Vary: Origin
header: Vary: X-Origin
header: Vary: Referer
header: Content-Encoding: gzip
header: Date: Fri, 18 Sep 2020 09:35:20 GMT
header: Server: scaffolding on HTTPServer2
header: Cache-Control: private
header: X-XSS-Protection: 0
header: X-Frame-Options: SAMEORIGIN
header: X-Content-Type-Options: nosniff
header: Alt-Svc: h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
header: Transfer-Encoding: chunked
Uploading file...
connect: (youtube.googleapis.com, 443)
send: u'POST /upload/youtube/v3/videos?uploadType=resumable&alt=json&part=status%2Csnippet HTTP/1.1\r\nHost: youtube.googleapis.com\r\ncontent-length: 142\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: (gzip)\r\nx-upload-content-length: 155903894\r\nx-upload-content-type: video/mp4\r\ncontent-type: application/json\r\nauthorization: Bearer xxxxxx_-v_xxxx\r\nx-goog-api-client: gdcl/1.8.0 gl-python/2.7.18rc1\r\n\r\n{"status": {"privacyStatus": "private"}, "snippet": {"tags": ["test"], "categoryId": "22", "description": "File name", "title": "Test Title"}}'
reply: 'HTTP/1.1 404 Not Found\r\n'
header: Content-Type: text/plain; charset=utf-8
header: X-GUploader-UploadID: ABg5-UyUv6D3yHDBjVs6znCaTrtwA5GthyyHgrqOZNzB2uRy_QnO10h40rBmFEJMBQQvzwKggt7J-k4ulclMI2e9H90emRvk-A
header: Content-Length: 9
header: Date: Fri, 18 Sep 2020 09:35:20 GMT
header: Server: UploadServer
header: Alt-Svc: h3-Q050=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
An HTTP error 404 occurred:
Not Found
What I have tried to do:
Run from another hosts, same problem
Renew authorization, same problem
Many thanks for any help
Youtube fixed the issue by changing the rootUrl in https://www.googleapis.com/discovery/v1/apis/youtube/v3/rest from https://youtube.googleapis.com/ to https://www.googleapis.com/
Related
I am attempting to make a web scraping tool to scrape facebook marketplace and ran into an issue when using my code on an AWS E2 Server.
It seems as though when I run the code locally, things work okay. But when I run them from my AWS E2 Server, I get redirected to a login page:
Locally
HTTP/1.1 200 OK
Vary: Accept-Encoding
accept-ch-lifetime: 4838400
accept-ch: sec-ch-prefers-color-scheme
x-fb-rlafr: 0
document-policy: force-load-at-top
Pragma: no-cache
Cache-Control: private, no-cache, no-store, must-revalidate
Expires: Sat, 01 Jan 2000 00:00:00 GMT
X-Content-Type-Options: nosniff
X-XSS-Protection: 0
X-Frame-Options: DENY
Strict-Transport-Security: max-age=15552000; preload
Content-Type: text/html; charset="utf-8"
X-FB-Debug: hRIETkwli3E8bVAgkC5W4QkI8QUsRTBrYr3VNlSA2uhlf6t+W79d7uwuiu7bO4XjOZSc40kGAmrfUXC1/3bHCg==
Date: Tue, 03 May 2022 20:38:25 GMT
Priority: u=3,i
Transfer-Encoding: chunked
Alt-Svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
Connection: keep-alive
AWS
HTTP/2 302
location: https://www.facebook.com/login/?next=https%3A%2F%2Fwww.facebook.com%2Fmarketplace%2Fnyc
x-fb-rlafr: 0
document-policy: force-load-at-top
pragma: no-cache
cache-control: private, no-cache, no-store, must-revalidate
expires: Sat, 01 Jan 2000 00:00:00 GMT
x-content-type-options: nosniff
x-xss-protection: 0
x-frame-options: DENY
strict-transport-security: max-age=15552000; preload
content-type: text/html; charset="utf-8"
x-fb-debug: 0BY9brNcXJnElH5BpyMTVoppfjVs+5MRBpn+HlJd477H/E3T3P6XZNh/bu3oEMT1+Np/zProRfu4XEqkU2CAOA==
content-length: 0
date: Tue, 03 May 2022 20:38:15 GMT
priority: u=3,i
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
Do I need to run my requests through a proxy server in order to interact with Facebook from my AWS E2 Server in order to send requests anonymously? Or is there something configured on my server that is causing this 302 response?
Any help would be greatly appreciated here!
I get the below messages for every test step, which is bit annoying. I need to process the console logs in a different way.
send: b'PUT /api/v2/superadmin_personal/item/14278b98-4430-4d2e-8301-1e30501da3b3 HTTP/1.1\r\nHost: abc.lab.com:8080\r\nUser-Agent: python-requests/2.27.1\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nAuthorization: Bearer 2c0717a7-b477-4e02-b1b5-df2a2757db70\r\nContent-Length: 137\r\nContent-Type: application/json\r\n\r\n'
send: b'{"endTime": "1646987482101", "status": "PASSED", "issue": null, "launchUuid": "f380b026-d7c9-4596-b80a-dcaec6fa82f2", "attributes": null}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
header: Content-Type: application/json
header: Date: Fri, 11 Mar 2022 08:30:58 GMT
header: Expires: 0
header: Pragma: no-cache
header: X-Content-Type-Options: nosniff
header: X-Frame-Options: DENY
header: X-Xss-Protection: 1; mode=block
header: Content-Length: 93
You can set rp.http.logging=false in the reportportal.prop file or as a JVM parameter.
There is a common switch for all HTTP requests/responses Python sends:
from http.client import HTTPConnection
HTTPConnection.debuglevel = 0
Unfortunately Python uses just print to log HTTP (as here), ignoring his own logging framework. That's really silly, but here where Python is. Therefore there is no any straight way to configure what you want log and what you would like to skip. You can just turn on or off console printing for all HTTP requests.
I tried through several means, but nowhere do I find a satisfatory answer to this -
What are the differences between Python "requests" module and Linux "curl" command? Does "requests" use "curl" underlying, or is it totally different way of dealing with HTTP request/response?
For most of the requests, they both behave in the same way (as it should be), but sometimes, I find a difference in response and it is really hard to figure out why is it so.
eg. Using curl for HEAD request:
curl --head https://historia.sherpadesk.com
HTTP/2 302
content-type: text/html; charset=utf-8
date: Mon, 28 Feb 2022 20:31:30 GMT
access-control-expose-headers: Request-Context
cache-control: private
location: /login/?ref=portal
set-cookie: ASP.NET_SessionId=nghpw4qp5cw2ntwmwfuxw3oi; path=/; HttpOnly; SameSite=Lax
content-length: 135
request-context: appId=cid-v1:d5f9900e-ecd4-442f-9e92-e11b4cdbc0c9
x-frame-options: SAMEORIGIN
x-xss-protection: 1
x-content-type-options: nosniff
strict-transport-security: max-age=31536000
and if I use -L to follow redirects,
curl --head https://historia.sherpadesk.com -L
HTTP/2 302
content-type: text/html; charset=utf-8
date: Mon, 28 Feb 2022 20:31:37 GMT
access-control-expose-headers: Request-Context
cache-control: private
location: /login/?ref=portal
set-cookie: ASP.NET_SessionId=trzp0bql4nibswux5z5wfayy; path=/; HttpOnly; SameSite=Lax
content-length: 135
request-context: appId=cid-v1:d5f9900e-ecd4-442f-9e92-e11b4cdbc0c9
x-frame-options: SAMEORIGIN
x-xss-protection: 1
x-content-type-options: nosniff
strict-transport-security: max-age=31536000
HTTP/2 302
content-type: text/html; charset=utf-8
date: Mon, 28 Feb 2022 20:31:38 GMT
access-control-expose-headers: Request-Context
location: https://app.sherpadesk.com/login/?ref=portal
content-length: 161
request-context: appId=cid-v1:d5f9900e-ecd4-442f-9e92-e11b4cdbc0c9
x-frame-options: SAMEORIGIN
x-xss-protection: 1
x-content-type-options: nosniff
strict-transport-security: max-age=31536000
HTTP/2 200
content-type: text/html; charset=utf-8
date: Mon, 28 Feb 2022 20:31:39 GMT
access-control-expose-headers: Request-Context
cache-control: no-store, no-cache
expires: -1
pragma: no-cache
set-cookie: ASP.NET_SessionId=aqmnxu2s3qkri3sravsrs1cq; path=/; HttpOnly; SameSite=Lax
content-length: 8935
request-context: appId=cid-v1:d5f9900e-ecd4-442f-9e92-e11b4cdbc0c9
x-frame-options: SAMEORIGIN
x-xss-protection: 1
x-content-type-options: nosniff
strict-transport-security: max-age=31536000
and here is the (debug) output when I use Python's requests module requests.head(url):
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): historia.sherpadesk.com:443
send: b'HEAD / HTTP/1.1\r\nHost: historia.sherpadesk.com\r\nUser-Agent: python-requests/2.26.0\r\nAccept-Encoding: gzip, deflate, br\r\nAccept: */*\r\nConnection: keep-alive\r\n\r\n'
reply: 'HTTP/1.1 403 Forbidden: Access is denied.\r\n'
header: Content-Length: 58
header: Content-Type: text/html
header: Date: Mon, 28 Feb 2022 20:36:18 GMT
header: X-Frame-Options: SAMEORIGIN
header: X-XSS-Protection: 1
header: X-Content-Type-Options: nosniff
header: Strict-Transport-Security: max-age=31536000
DEBUG:urllib3.connectionpool:https://historia.sherpadesk.com:443 "HEAD / HTTP/1.1" 403 0
INFO:root:URL: https://historia.sherpadesk.com/
INFO:root:<Response [403]>
which just results in 403 response code. Response is same whether allow_redirects is True/False. I have also tried using proxy with python code, as I thought maybe its getting blocked as this URL might be recognising Python's request to be a bot, but that also fails. Also, if that was the case, why does curl succeed?
So, my main question here is: what are the major differences between curl and requests, which might cause difference in responses in certain cases? If possible, I would really like thorough explanation which could help me debug and resolve these issues.
The two libraries are different but the problem here is related to user agent.
When I try with curl, specifying the python-requests user agent:
$ curl --head -A "python-requests/2.26.0" https://historia.sherpadesk.com/
HTTP/2 403
content-type: text/html
date: Mon, 28 Feb 2022 22:30:02 GMT
content-length: 58
x-frame-options: SAMEORIGIN
x-xss-protection: 1
x-content-type-options: nosniff
strict-transport-security: max-age=31536000
With curl default user agent:
$ curl --head https://historia.sherpadesk.com/
HTTP/2 302
...
Apparently, they have some type of website security that is blocking HTTP clients like python-requests, but not curl for some reason.
I'm trying to debug why i'm not able to POST an xml file with requests .post() to an API. The following wget command works fine:
wget -vv --no-check-certificate --post-file dynobjadd.xml \
"https://1.1.1.1/api/?type=user-id&action=set&key=$MYSUPERSECRETKEY=&file-name=dynobjadd.xml&client=wget" \
--no-http-keep-alive -O response.out
successful wget output:
...
URI encoding = ‘UTF-8’
--2017-01-05 13:21:11-- https://1.1.1.1/api/?type=user-id&action=set&key=$MYSUPERSECRETKEY=&file-name=dynobjadd.xml&client=wget
Certificates loaded: 165
Connecting to 1.1.1.1:443... connected.
...
---request begin---
POST https://1.1.1.1/api/?type=user-id&action=set&key=$MYSUPERSECRETKEY=&file-name=dynobjadd.xml&client=wget HTTP/1.1
User-Agent: Wget/1.18 (linux-gnu)
Accept: */*
Accept-Encoding: identity
Host: 1.1.1.1
Connection: Close
Content-Type: application/x-www-form-urlencoded
Content-Length: 175
---request end---
[writing BODY file dynobjadd.xml ... done]
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 200 OK
Server:
Date: Thu, 05 Jan 2017 20:21:12 GMT
Content-Type: application/xml; charset=UTF-8
Content-Length: 255
Connection: close
ETag: "437cf-12b-56e39c36"
Pragma: no-cache
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Thu, 19 Nov 1981 08:52:00 GMT
X-FRAME-OPTIONS: SAMEORIGIN
Set-Cookie: PHPSESSID=123123; path=/; secure; HttpOnly
---response end---
200 OK
python code i am trying:
xml = open('dynobjadd.xml').read()
url = 'https://1.1.1.1/api/?type=user-id&action=set&key=$MYSUPERSECRETKEY=&file-name=dynobjadd.xml&client=requests'
r = requests.post(url, data=xml, verify=False )
r.content() output:
<response status = 'error' code = '400'><result><msg>No file uploaded</msg></result></response>
You should use argument 'files'.
From http://docs.python-requests.org/en/master/user/quickstart/#post-a-multipart-encoded-file :
>>> url = 'http://httpbin.org/post'
>>> files = {'file': open('report.xls', 'rb')}
>>> r = requests.post(url, files=files)
>>> r.text
{
...
"files": {
"file": "<censored...binary...data>"
},
...
}
I have the exact same code on 2 servers. With one of them I can connect to amazon SQS, while the other one can't. Here is the output from the non-working server:
send: 'GET /?Action=GetQueueUrl&QueueName=Erablitek&Version=2012-11-05 HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nHost: queue.amazonaws.com\r\nAuthorization: AWS4-HMAC-SHA256 Credential=AKIAIOQSTDBQVPXWYK7A/20150219/us-east-1/sqs/aws4_request,SignedHeaders=host;x-amz-date,Signature=9f5b0a187b178974f7b9b28e0028c2f9c034ee6fa2b1ee3ea9fcf9c3370219d5\r\nX-Amz-Date: 20150219T155308Z\r\nUser-Agent: Boto/2.34.0 Python/2.7.3 Linux/3.12.31+\r\n\r\n'
reply: 'HTTP/1.1 403 Forbidden\r\n'
header: Server: Server
header: Date: Thu, 19 Feb 2015 10:53:13 GMT
header: Content-Type: text/xml
header: Content-Length: 367
header: Connection: keep-alive
header: x-amzn-RequestId: 717fcf52-963b-5c4b-8f22-820d54e28cb4
And here is the output from the working server
send: 'GET /?Action=GetQueueUrl&QueueName=Erablitek&Version=2012-11-05 HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nHost: queue.amazonaws.com\r\nAuthorization: AWS4-HMAC-SHA256 Credential=AKIAIOQSTDBQVPXWYK7A/20150219/us-east-1/sqs/aws4_request,SignedHeaders=host;x-amz-date,Signature=a9538654d3b281156cbb5a410717e80381cac1e19c9ffcd8d96589c25ed6256d\r\nX-Amz-Date: 20150219T110853Z\r\nUser-Agent: Boto/2.35.2 Python/2.7.3 Linux/3.12.31+\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: Server
header: Date: Thu, 19 Feb 2015 11:08:55 GMT
header: Content-Type: text/xml
header: Content-Length: 321
header: Connection: keep-alive
header: x-amzn-RequestId: f29ed0a6-d762-5079-b26c-9df911e5c178
To my knowledge, both servers are configured and maintained pretty much the same way, however they're installed in 2 different locations. I have checked and credentials are the same on both servers.
Edit: I have also tried several versions of boto, including 2.35.2 which is the one installed on the working server.
I have no idea what else I should be checking
The QueueName parameter appears to be different in the two requests. In the first one (the non-working one) the queue name is:
QueueName=ErabliTEK
and in the second, working example it is:
QueueName=Erablitek
Could that be your problem?