I'm having a problem related to proxies in my Python script. Running the script below, to access NCBI Blast through biopython, the company's network where I work blocks the access because of security reasons. While talking with the IT guys they gave me a proxy for this kind of situations that has to be incorporated in my script. I've tried a lot of potential solutions but nothing seems to be working. Am I missing something here?
def main(seq):
import os
from Bio.Blast import NCBIWWW
import time
start_time = time.time()
try:
print('Connecting to NCBI...')
blast_handle = NCBIWWW.qblast('blastn','nt',sequence = seq, format_type = 'Text', megablast=True)
text = blast_handle.read()
print(text)
print("--- %s seconds ---" % (time.time() - start_time))
except Exception as e:
print(e)
if __name__ == '__main__':
import os
os.environ['http_proxy'] = 'http://123.456.78.90:80' # The proxy IT guys gave me
seq = 'CAACTTTTTTTTTTATTACAGACAATCAAGAAATTTTCTATTGAAATAAAATATTTTAAA\
ACTTTCAACAACGGATCTCTTGGTTCTCGCATCGATGAAGAACGTAGCGAATTGCGATAA\
GTAATGTGAATTGCAGATTCTCGTGAATCATTGAATTTTTGAACGCACATTGCGCCCTCT\
GGTATTCCAGAGGGCATGCCTGTTTGAGCGTCATTTCCTTCTCAAAAACCCAGTTTTTGG\
TTGTGAGTGATACTCTGCTTCAGGGTTAACTTGAAAATGCTATGCCCCTTTGGCTGCCCT\
TCTTTGAGGGGACTGCGCGTCTGTGCAGGATGTAACCAATGTATTTAGGTATTCATACCA\
ACTTTCATTGTGCGCGTCTTATGCAGTTGTAGTCCACCCAACCTCAGACACACAGGCTGG\
CTGGGCCAACAGTATTCATAAAGTTTGACCTCA'
main(seq)
Thank you very much.
It seems that NCBIWWW.qblast doesn't have support for proxies, so you will need to adapt the code yourself.
In your local BioPython installation, go find the biopython/Bio/Blast/NCBIWWW.py file and add your proxy settings at line 203:
request = Request(url_base, message, {"User-Agent": "BiopythonClient"})
request.set_proxy('http://123.456.78.90:80', 'http') # <-- Add this line
handle = urlopen(request)
Related
Me and my friend have been trying to solve this code for about a week now with no success. We would appreciate some feedback from experienced programmers.
We have developed below code to connect to a web-socket. Our python script runs smoothly for 7 hours, but after 7 hours it crashes. We got "Error to many open files" couple of times. I have searched stackoverflow for a while to find smilar mistake in coding but we couldn't relate to our actual problem.
We also watch closely proc/"pid of our python script"/fd for open pipes. Whenever it reached to 1024 the websocket connection dies. We edited ulimit -n to increased the limit also but still script dies.
I am sharing the below code, I would really appreciate if you guys can give us some feedback in order for us to solve our longlasting headache.
import time
import datetime
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
from bt import Bt
from app import db
from app.models import LOG_HISTORY, BOOKING_ORDERS, BOOKING_CANCELLING,
add_row, delete_rows
import config
logger = config.Logger('bt log websocket.log')
def get_authenticateWss():
authenticated_wss = Bt(key=config.socket_api_key,
secret=config.socket_api_secret)
authenticated_wss.start()
while not authenticated_wss.conn.connected.is_set():
time.sleep(1)
authenticated_wss.authenticate()
time.sleep(5)
return authenticated_wss
def main(authenticated_wss):
while authenticated_wss.conn.connected.is_set():
booking_orders = BOOKING_ORDERS.query.all()
for booking_order in booking_orders:
payload = {
'cid': booking_order.cid,
'symbol': 't%s' % booking_order.symbol.split(":") .
[-1].strip(),
'type': "EXCHANGE LIMIT",
'amount': str(booking_order.amount),
'price': str(booking_order.price),
'hidden': 1
}
authenticated_wss.new_order(**payload)
logger.info("Creating the Order: %s" % str(payload))
db.session.delete(booking_order)
if float(booking_order.amount) >= 0:
add_row(LOG_HISTORY, [datetime.datetime.now(),
booking_order.symbol, "Buy Order", str(payload)])
else:
add_row(LOG_HISTORY, [datetime.datetime.now(),
booking_order.symbol, "Selling Order", str(payload)])
time.sleep(5)
booking_cancels = BOOKING_CANCELLING.query.all()
for booking_cancel in booking_cancels:
payload = {
'id': booking_cancel.order_id,
'cid': booking_cancel.order_cid,
'cid_date': booking_cancel.create_mts
}
authenticated_wss.cancel_order(**payload)
logger.info("Cancelling the Order: %s" % str(payload))
db.session.delete(booking_cancel)
add_row(LOG_HISTORY, [datetime.datetime.now(),
booking_cancel.symbol, "Cancelling Order", str(payload)])
time.sleep(5)
# time.sleep(10)
if __name__ == "__main__":
delete_rows(BOOKING_ORDERS)
delete_rows(BOOKING_CANCELLING)
while True:
logger.info("-------------- START ------------------")
authenticated_wss = get_authenticateWss()
try:
main(authenticated_wss)
except Exception as e:
logger.error(e)
finally:
logger.info("---------- STOP -----------------")
authenticated_wss.stop()
We have solved the issue, it was totally web-socket compatibility issue. We have updated the version of the module.
I have a python script that downloads files using libtorrent python binding. I just want to know how to remove the torrent once the download is complete.
I'm posting here the example script I used to make mine (I'm not posting mine because it's too large, it has database parts).
import libtorrent as lt
import time
ses = lt.session()
params = { 'save_path': '/home/downloads/'}
link = "magnet:?xt=urn:btih:4MR6HU7SIHXAXQQFXFJTNLTYSREDR5EI&tr=http://tracker.vodo.net:6970/announce"
handle = lt.add_magnet_uri(ses, link, params)
print 'downloading metadata...'
while (not handle.has_metadata()): time.sleep(1)
print 'got metadata, starting torrent download...'
while (handle.status().state != lt.torrent_status.seeding):
print '%d %% done' % (handle.status().progress*100)
time.sleep(1)
Thanks.
you call remove_torrent() on the session object, passing in the torrent_handle to remove.
http://libtorrent.org/reference-Core.html#remove_torrent()
In your script:
ses.remove_torrent(handle)
PS: I have a similar question with Requests HTTP library here.
I am using python v2.7 on windows 7 OS. I am using urllib2 module. I have two code snippets. One file is named as myServer.py The server class has 2 methods named as getName(self,code) and getValue(self).
The other script named as testServer.py simply calls the methods from the server class to retrieve the values and prints them. The server class basically retrieves the values from a Server in my local network. So, unfortunately I can't provide you the access for testing the code.
Problem: When I execute my testServer.py file, I observed in the task manager that the memory consumption keeps increasing. Why is it increasing and how to avoid it? If I comment out the following line
print serverObj.getName(1234)
in testServer.py then there is no increase in memory consumption.
I am sure that the problem is with the getName(self,code) of the server class. But unfortunately, I couldn't figure out what the problem is.
Code: Please find the code snippets below:
#This is the myServer.py file
import urllib2
import json
import random
class server():
def __init__(self):
url1 = 'https://10.0.0.1/'
username = 'user'
password = 'passw0rd'
passwrdmgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
passwrdmgr.add_password(None, url1, username, password)
authhandler = urllib2.HTTPBasicAuthHandler(passwrdmgr)
opener = urllib2.build_opener(authhandler)
urllib2.install_opener(opener)
def getName(self, code):
code = str(code)
url = 'https://10.0.0.1/' + code
response = urllib2.urlopen(url)
data = response.read()
name = str(data).strip()
return name
def getValue(self):
value = random.randrange(0,11)
return value
The following is the testServer.py snippet
from myServer import server
import time
serverObj = server()
while True:
time.sleep(1)
print serverObj.getName(1234)
print serverObj.getValue()
Thank you for your time!
This is question is quite similar to my other question. So I think the answer is also quite similar. The answer can be found here https://stackoverflow.com/a/23172330/2382792
So, I'm a complete noob when it comes to this kind of thing, and I need some help. I work in software QA for an ecommerce company, and we started using Saucelabs for our automated testing. I'm in the process of learning python but really know next to nothing at this point. I can build a decent test in Selenium IDE, export in Python/Selenium Webdriver, and run the test. Not an issue. However, how do I set the pass/fail flag on the interface? One of our devs wrote a parallel script so I can run a large number of tests at one time, but in order to do so I need to be able to see at a glance which tests have passed and which ones have failed. Can you help me? Thanks!
Also, any tutorials you are aware of on Selenium Webdriver would be helpful too! Really want to learn this stuff!
I did it this way, first you need to import some things
# These next imports for reporting Test status to Sauce Labs
import sys
import httplib
import base64
try:
import json
except ImportError:
import simplejson as json
Then you need this config
#Config to connect to SauceLabs REST API
config = {"username": "yourusernamehere",
"access-key": "youraccesskeyhere"}
Then you put your tests. At the end, before your TearDown you need to include
# Curl call to SauceLabs API to report Job Result
def set_test_status(self, jobid, passed):
base64string = base64.encodestring('%s:%s' % (config['username'], config['access-key']))[:-1]
body_content = json.dumps({"passed": passed})
connection = httplib.HTTPConnection("saucelabs.com")
connection.request('PUT', '/rest/v1/%s/jobs/%s' % (config['username'], jobid),
body_content,
headers={"Authorization": "Basic %s" % base64string})
result = connection.getresponse()
return result.status == 200
Then in your tearDown you need to include some kind of if logic. I did it this way (and it works)
def tearDown(self):
# sys.exc_info should be (None, None, None) if everything is OK, it fills with some values if something went wrong
# This if reports to Sauce Labs the outcome of the Test where True = Pass and False = Failed
if sys.exc_info() == (None, None, None):
self.set_test_status(self.driver.session_id, True)
else:
self.set_test_status(self.driver.session_id, False)
self.driver.quit()
self.assertEqual([], self.verificationErrors)
That did the trick for me
You can use Sauce labs REST API to mark your test pass/failed. You can find some example code given here
Here is the situation, I'm trying to get this API to work for me, but I can't seem to figure out all of its quirks. Even when I run the example below (taken from https://bitbucket.org/basti/python-amazon-product-api/src/2b6b628300c4/examples/all-galileo-titles.py) I get random errors. Now I'm getting an error that says the __init__() takes 4 arguments, and I'm only feeding it two. I put all of the credentials in the correct place, though, and I can't find anywhere in the modules where __init__() is demanding extra arguments. Anyone have any thoughts?
Here is what I was running. More details about the program can be found at that link above.
from amazonproduct.api import API
import lxml
if __name__ == '__main__':
# Don't forget to create file ~/.amazon-product-api
# with your credentials (see docs for details)
api = API(locale = 'uk')
result = api.item_search('Books', Publisher= 'RosettaBooks',
ResponseGroup='Large')
# extract paging information
total_results = result.results
total_pages = len(result) # or result.pages
for book in result:
print 'page %d of %d' % (result.current, total_pages)
#~ from lxml import etree
#~ print etree.tostring(book, pretty_print=True)
print book.ASIN,
print unicode(book.ItemAttributes.Author), ':',
print unicode(book.ItemAttributes.Title),
if hasattr(book.ItemAttributes, 'ListPrice'):
print unicode(book.ItemAttributes.ListPrice.FormattedPrice)
elif hasattr(book.OfferSummary, 'LowestUsedPrice'):
print u'(used from %s)' % book.OfferSummary.LowestUsedPrice.FormattedPrice
I had the same problem, I found on the bitbucket site for python-amazon-product-api a question from a couple of years ago that enabled me to work around the problem. See 11/1/2011 new requirement AssociateTag -- update required?
What I did to work around is not use the .amazon-product-api file to specify the credentials but instead pass them as part of the API call:
AWS_KEY = '……….'
SECRET_KEY = '……………'
ASSOCIATE_TAG = '…………….'
api = API(access_key_id=AWS_KEY, secret_access_key=SECRET_KEY, locale="us", associate_tag=ASSOCIATE_TAG)