We Keep Coding

How to make Python go through URLs in a text file, check their status codes, and exclude all ones with 404 error?

I tried the following script, but unfortunately the output file is identical to the input file. I'm not sure what's wrong with it.
import requests
url_lines = open('banana1.txt').read().splitlines()
remove_from_urls = []
for url in url_lines:
remove_url = requests.get(url)
print(remove_url.status_code)
if remove_url.status_code == 404:
remove_from_urls.append(url)
continue
url_lines = [url for url in url_lines if url not in remove_from_urls]
print(url_lines)
# Save urls example
with open('banana2.txt', 'w+') as file:
for item in url_lines:
file.write(item + '\n')

There seems to be no error in your code, but there are few things that would help to make it more readable and consistent. The first course of action should be to make sure there is at least one url that would return a 404 status code.
Edit: After providing the actual URL.
The 404 problem
In your case, the problem is the Twitter actually does not return 404 error for your "Not found" url. You can test it using curl:
$ curl -o /dev/null -w "%{http_code}" "https://twitter.com/davemeltzerWON/status/1321279214365016064"
200
Or using Python:
import requests
response = requests.get("https://twitter.com/davemeltzerWON/status/1321279214365016064")
print(response.status_code)
The output for both should be 200.
Since Twitter is a JavaScript application that loads its content after it has been processed in browser, you cannot find the information you are looking for in the HTML response. You would need to use something like Selenium to actually process the JavaScript for you and then you would be able to look for actual text like "not found" on the web page.
Code review
Please make sure to close the file properly. Also, file object is a lines iterator, you can convert it to list very easily. Another trick to make the code more readable is to make use of Python set. So you may read the file like this:
with open("banana1.txt") as fid:
url_lines = set(fid)
Then you simply remove all the links that do not work:
not_working = set()
for url in url_lines:
if requests.get(url).status_code == 404:
not_working.add(url)
working = url_lines - not_working
with open("banana2.txt", "w") as fid:
fid.write("\n".join(working))
Also, if some of the links point to the same server, you should make use of requests.Session class:
from requests import Session
session = Session()
Then replace requests.get with session.get, you should get some performance boost since the Session uses keep-alive connection and many other things.

Access Sonarqube Webapi with python requests

i quite new to pyhton. I just try a simple way to get an HTTP response with python to a simple get from the sonar Web API
i use the request library and try a simple use :
project = requests.get(url=Sonar_Api_Projects_Search, params=param_Projects, verify=False, headers={'Authorization': 'token {}'.format(token)})
the request is well formatted and work fine when i use it in e web browser.
but as a response i get this strange output :
{"err_code":500,"err_msg":"undefined method empty?' for
nil:NilClass\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/lib/authenticated_system.rb:132:in
login_from_basic_auth'\n\torg/jruby/RubyProc.java:290:in
call'\n\torg/jruby/RubyProc.java:224:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/http_authentication.rb:126:in
authenticate'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/http_authentication.rb:116:in
authenticate_with_http_basic'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/lib/authenticated_system.rb:129:in
login_from_basic_auth'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/lib/authenticated_system.rb:11:in
current_user'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/app/controllers/application_controller.rb:102:in set_user_session'\n\torg/jruby/RubyKernel.java:2223:in
send'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activesupport-2.3.15/lib/active_support/callbacks.rb:178:in
evaluate_method'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activesupport-2.3.15/lib/active_support/callbacks.rb:166:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/filters.rb:225:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/filters.rb:629:in
run_before_filters'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/filters.rb:615:in
call_filters'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/filters.rb:610:in
perform_action_with_filters'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/benchmarking.rb:68:in
perform_action_with_benchmark'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activesupport-2.3.15/lib/active_support/core_ext/benchmark.rb:17:in
ms'\n\tjar:file:/D:/sonarqube-5.6.6_20170214/lib/server/jruby-complete-1.7.9.jar!/META-INF/jruby.home/lib/ruby/1.8/benchmark.rb:308:in
realtime'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activesupport-2.3.15/lib/active_support/core_ext/benchmark.rb:17:in
ms'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/benchmarking.rb:68:in
perform_action_with_benchmark'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/rescue.rb:160:in
perform_action_with_rescue'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/flash.rb:151:in perform_action_with_flash'\n\torg/jruby/RubyKernel.java:2223:in
send'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/base.rb:532:in
process'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/filters.rb:606:in
process_with_filters'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/base.rb:391:in
process'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/base.rb:386:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/routing/route_set.rb:450:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/dispatcher.rb:87:in
dispatch'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/dispatcher.rb:85:in
dispatch'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/dispatcher.rb:121:in
_call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/dispatcher.rb:130:in
build_middleware_stack'\n\torg/jruby/RubyProc.java:290:in
call'\n\torg/jruby/RubyProc.java:224:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/query_cache.rb:29:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/query_cache.rb:34:in
cache'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/query_cache.rb:9:in
cache'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/query_cache.rb:28:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:361:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/config/environment.rb:67:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/string_coercion.rb:25:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/rack-1.1.6/lib/rack/head.rb:9:in call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/rack-1.1.6/lib/rack/methodoverride.rb:24:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/params_parser.rb:15:in
call'\n\tfile:/D:/sonarqube-5.6.6_20170214/lib/server/jruby-rack-1.1.13.2.jar!/jruby/rack/session_store.rb:70:in
context'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/rack-1.1.6/lib/rack/session/abstract/id.rb:58:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/failsafe.rb:26:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/dispatcher.rb:106:in
call'\n\tfile:/D:/sonarqube-5.6.6_20170214/lib/server/jruby-rack-1.1.13.2.jar!/rack/adapter/rails.rb:34:in
serve_rails'\n\tfile:/D:/sonarqube-5.6.6_20170214/lib/server/jruby-rack-1.1.13.2.jar!/rack/adapter/rails.rb:39:in
call'\n\tfile:/D:/sonarqube-5.6.6_20170214/lib/server/jruby-rack-1.1.13.2.jar!/rack/handler/servlet.rb:22:in
call'\n"}
Can someone help me ?
Thanks a lot
Best regards
Arnaud

Direct use of requests never worked for me.
I do the following and it is working fine:
(below code is to list projects in Sonar)
import json , requests, pprint
url = 'http://sonar_url:9000/api/projects/search'
myToken = 'fa2377941a95125443f4efade615512jjkd221211a48'
session = requests.Session()
session.auth = myToken, ''
call = getattr(session, 'get')
res = call(url)
print(res.status_code)
binary = res.content
output = json.loads(binary)
pprint.pprint(output)
...
#Parse json result

In Sonarqube 8.9, requests is working for me.
First, you should should create an API token. Per the docs:
This is the recommended way. Benefits are described in the page User Token. The token is sent via the login field of HTTP basic authentication, without any password.
The docs go on to provide a weird curl usage example:
# note that the colon after the token is required in curl to set an empty password
curl -u THIS_IS_MY_TOKEN: https://sonarqube.com/api/user_tokens/search
In requests, this looks something like this:
response = requests.get(
"http://your-sonar-instance.com/api/blah",
auth=HTTPBasicAuth("Some Sonarqube API token", "")
)
return json.loads(response.text)
See https://docs.sonarqube.org/latest/extend/web-api/ for API details.
Also note that auth=HTTPBasicAuth("token", "") seems to behave differently from auth=HTTPBasicAuth("token", None).

I know its an old question. Thankfully there is a wrapper library available now - https://github.com/shijl0925/python-sonarqube-api. It works quite well and is easy to setup.
If possible people from Sonarsource could make it the official one so that more people start using it and it gets maintained in the future too.

add POST HOOK to a bitbucket repository using their REST API and python-requests

I'm trying to add a POST hook to a bitbucket repository, but I'm getting 404 results on each attempt.
I'm doing:
payload = {'type': 'POST', 'URL': announce_post_hook}
content_type = {"Content-Type": "application/json"}
request_url = 'https://api.bitbucket.org/1.0/repositories/{repo_owner}/{repo_slug}/services/'
request_url = request_url.format(repo_owner=repo_owner, remote_url=remote_url)
requests.post(request_url, auth=(repo_user, repo_pass), data=json.dumps(payload), headers=content_type)
I also tried using this URL:
https://bitbucket.org/api/1.0/repositories/{repo_owner}/{repo_slug}/services/
Since is the one listed in differnt parts of their api (and I'm using api.bitbucket.org instead of bitbucket.org/api/ to set up deployment keys, for example).
If I try to do it using curl, as in:
curl -X POST -u user:pass https://api.bitbucket.org/1.0/repositories/repowner/reposlug/services/ --data "type=POST&URL=https://hooks.urladdress.com"
then it'll work. But attempting to do it through python-requests like in the other api call will fail...
Anyone has any idea on what's going on? It'll just respond that the resource is not found, which doesn't seem to be right (since it works through curl)
Found this question with a very similar problem, but there are no answers there...

For future reference, if somoeone else has the same problem, apparently this particular endpoint doesn't accept json encoded data.
So the request would need to be
requests.post(request_url, auth=(repo_user, repo_pass), data='type=POST&URL=hooks.url.com')

How to view Terminal output when running Requests in Python?

I have a simple Python scripts that POSTs a local file to a given URL via Requests:
import requests
url = 'http://myWebsite.com/extension/extension/extension'
files = {'file': open("myLocalFile.csv")}
r = requests.post(url, files=files)
print r.headers
When I run a cURL command in Terminal that does the exact same thing:
curl -k -F docfile=#myLocalFile.csv http://myWebsite.com/extension/extension/extension
I get the output:
{"success":true, "data":{"uploaded":39, "errors":0, "unchanged":39, "skipped":0, "updated":0, "created":0, "failed":[]}, "numRows":1}
which indicates that I have successfully uploaded the file. How can I view this same output when I run my python script? I want to be able to parse through this output and check to see if "success" is true/false.
Sorry for the weird formatting, I tried to fix it but couldn't :(

print r.text
gives you the response body and then you can parse through the response body to check if success=true or false.

Not having used the library before the example on their home page seems helpful
python-requests.
More specifically:
print(r.json()) #possibly print(r.content)
# prints {u'private_gists': 419, u'total_private_repos': 77, ...}
Also see this question

Trouble with pycurl.POSTFIELDS

I'm familiar with CURL in PHP but am using it for the first time in Python with pycurl.
I keep getting the error:
Exception Type: error
Exception Value: (2, '')
I have no idea what this could mean. Here is my code:
data = {'cmd': '_notify-synch',
'tx': str(request.GET.get('tx')),
'at': paypal_pdt_test
}
post = urllib.urlencode(data)
b = StringIO.StringIO()
ch = pycurl.Curl()
ch.setopt(pycurl.URL, 'https://www.sandbox.paypal.com/cgi-bin/webscr')
ch.setopt(pycurl.POST, 1)
ch.setopt(pycurl.POSTFIELDS, post)
ch.setopt(pycurl.WRITEFUNCTION, b.write)
ch.perform()
ch.close()
The error is referring to the line ch.setopt(pycurl.POSTFIELDS, post)

I do like that:
post_params = [
('ASYNCPOST',True),
('PREVIOUSPAGE','yahoo.com'),
('EVENTID',5),
]
resp_data = urllib.urlencode(post_params)
mycurl.setopt(pycurl.POSTFIELDS, resp_data)
mycurl.setopt(pycurl.POST, 1)
...
mycurl.perform()

I know this is an old post but I've just spent my morning trying to track down this same error. It turns out that there's a bug in pycurl that was fixed in 7.16.2.1 that caused setopt() to break on 64-bit machines.

It would appear that your pycurl installation (or curl library) is damaged somehow. From the curl error codes documentation:
CURLE_FAILED_INIT (2)
Very early initialization code failed. This is likely to be an internal error or problem.
You will possibly need to re-install or recompile curl or pycurl.
However, to do a simple POST request like you're doing, you can actually use python's "urllib" instead of CURL:
import urllib
postdata = urllib.urlencode(data)
resp = urllib.urlopen('https://www.sandbox.paypal.com/cgi-bin/webscr', data=postdata)
# resp is a file-like object, which means you can iterate it,
# or read the whole thing into a string
output = resp.read()
# resp.code returns the HTTP response code
print resp.code # 200
# resp has other useful data, .info() returns a httplib.HTTPMessage
http_message = resp.info()
print http_message['content-length'] # '1536' or the like
print http_message.type # 'text/html' or the like
print http_message.typeheader # 'text/html; charset=UTF-8' or the like
# Make sure to close
resp.close()
to open an https:// URL, you may need to install PyOpenSSL:
http://pypi.python.org/pypi/pyOpenSSL
Some distibutions include this, others provide it as an extra package right through your favorite package manager.
Edit: Have you called pycurl.global_init() yet? I still recommend urllib/urllib2 where possible, as your script will be more easily moved to other systems.