I am trying to list items in a S3 container with the following code.
import boto.s3
from boto.s3.connection import OrdinaryCallingFormat
conn = boto.connect_s3(calling_format=OrdinaryCallingFormat())
mybucket = conn.get_bucket('Container001')
for key in mybucket.list():
print key.name.encode('utf-8')
Then I get the following error.
Traceback (most recent call last):
File "test.py", line 5, in <module>
mybucket = conn.get_bucket('Container001')
File "/usr/lib/python2.7/dist-packages/boto/s3/connection.py", line 370, in get_bucket
bucket.get_all_keys(headers, maxkeys=0)
File "/usr/lib/python2.7/dist-packages/boto/s3/bucket.py", line 358, in get_all_keys
'', headers, **params)
File "/usr/lib/python2.7/dist-packages/boto/s3/bucket.py", line 325, in _get_all
response.status, response.reason, body)
boto.exception.S3ResponseError: S3ResponseError: 301 Moved Permanently
<?xml version="1.0" encoding="UTF-8"?>
PermanentRedirectThe bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.99EBDB9DE3B6E3AF
Container001
<HostId>5El9MLfgHZmZ1UNw8tjUDAl+XltYelHu6d/JUNQsG3OaM70LFlpRchEJi9oepeMy</HostId><Endpoint>Container001.s3.amazonaws.com</Endpoint></Error>
I tried to search for how to send requests to the specified end point, but couldn't find useful information.
How do I avoid this error?
As #garnaat mentioned and #Rico answered in another question connect_to_region works with OrdinaryCallingFormat:
conn = boto.s3.connect_to_region(
region_name = '<your region>',
aws_access_key_id = '<access key>',
aws_secret_access_key = '<secret key>',
calling_format = boto.s3.connection.OrdinaryCallingFormat()
)
bucket = conn.get_bucket('<bucket name>')
in terminal run
nano ~/.boto
if there is some configs try to comment or rename file and connect again. (it helps me)
http://boto.cloudhackers.com/en/latest/boto_config_tut.html
there is boto config file directories. take a look one by one and clean them all, it will work by default configs. also configs may be in .bash_profile, .bash_source...
I guess you must allow only KEY-SECRET
also try to use
calling_format = boto.s3.connection.OrdinaryCallingFormat()
Related
I have a Firestore database like this:(https://i.stack.imgur.com/QSZ8m.png)
My code intends to update the fields "intensity" and "seconds" (under the document "1", under collection "Event") with the value "test" and 123 respectively.
import firebase_admin
from firebase_admin import credentials
from firebase_admin import db
# Initialize Firebase admin
cred = credentials.Certificate('taiwaneew-firebase-adminsdk-odl9d-222bd18a4e.json')
firebase_admin.initialize_app(cred, {
'databaseURL': 'https://taiwaneew.firebaseio.com/'
})
# Define a function to send data to the Firebase database
def send_data(param1, param2):
ref = db.reference(path='/TaiwanEEW/Event/1')
ref.update({
'intensity': param1,
'seconds': param2
})
# Invoke our function to send data to Firebase
send_data("test", 123)
The code, however, causes the following error:
File "/Users/joelin/PycharmProjects/pythonProject/venv/lib/python3.10/site-packages/firebase_admin/db.py", line 929, in request
return super(_Client, self).request(method, url, **kwargs)
File "/Users/joelin/PycharmProjects/pythonProject/venv/lib/python3.10/site-packages/firebase_admin/_http_client.py", line 119, in request
resp.raise_for_status()
File "/Users/joelin/PycharmProjects/pythonProject/venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://taiwaneew.firebaseio.com/TaiwanEEW/Event/1.json?print=silent
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/joelin/PycharmProjects/pythonProject/eewPush.py", line 20, in <module>
send_data("777", 778)
File "/Users/joelin/PycharmProjects/pythonProject/eewPush.py", line 14, in send_data
ref.update({
File "/Users/joelin/PycharmProjects/pythonProject/venv/lib/python3.10/site-packages/firebase_admin/db.py", line 341, in update
self._client.request('patch', self._add_suffix(), json=value, params='print=silent')
File "/Users/joelin/PycharmProjects/pythonProject/venv/lib/python3.10/site-packages/firebase_admin/db.py", line 931, in request
raise _Client.handle_rtdb_error(error)
firebase_admin.exceptions.NotFoundError: 404 Not Found
I have tried to identify the cause of error but the same error persists. I would really like to hear some opinions if you have any experiences on this. Thank you so much!
I have double checked that my credentials json file is correct, under the same directory as the python file, and my database premissions to write and read set to true.
I tried both '/TaiwanEEW/Event/1' and '/taiwaneew/Event/1' for the reference path because I am not sure if it should be the project name or the database name.
I coud not find the error here, so I come with a workaround.
You can use firebase_admin.firestore.
Then, your db object will be instanciated by db = firestore.client(), and have access to all collections and document (see doc).
Complete solution:
import firebase_admin
from firebase_admin import credentials, firestore
cred = credentials.Certificate('taiwaneew-firebase-adminsdk-odl9d-222bd18a4e.json')
# no need for an url here if your credentials already contain the project id.
firebase_admin.initialize_app(cred)
db = firestore.client()
# Define a function to send data to the Firebase database
def send_data(param1, param2):
doc = db.document('Event/1') # or doc = db.collection('Event').document('1')
doc.update({
'intensity': param1,
'seconds': param2
})
# Invoke our function to send data to Firebase
send_data("test", 123)
I am a OpenStack noob here.
I try to use REST API to automate my openstack work. I can successfully authenticate my account and get token from curl with below command:
export AUTH_URL="https://my_url:5000/v2.0/tokens"
curl -v -X POST $AUTH_URL -d '{"auth":{"passwordCredentials":{"username": "myusername", "password":"myuser_password"}, "tenantId":"my_tenant_id"}}' -H 'Content-type: application/json' | python -m json.tool
But, when I try to use Pyton, with this code, it gave me this error:
Traceback (most recent call last):
File "./test.py", line 10, in <module>
keystone = ksclient.Client(auth_url=auth_url, username=user_name, password=user_pwd, tenant_name=tenant_name)
File "/usr/lib/python2.7/site-packages/keystoneclient/v2_0/client.py", line 176, in __init__
self.authenticate()
File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
return wrapped(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/keystoneclient/httpclient.py", line 581, in authenticate
resp = self.get_raw_token_from_identity_service(**kwargs)
File "/usr/lib/python2.7/site-packages/keystoneclient/v2_0/client.py", line 220, in get_raw_token_from_identity_service
_("Authorization Failed: %s") % e)
keystoneauth1.exceptions.auth.AuthorizationFailure: Authorization Failed: The resource could not be found. (HTTP 404) (Request-ID:
My python code is below:
#!/usr/bin/env python
import keystoneclient.v2_0.client as ksclient
auth_url = "https://my_url:5000/v2.0/tokens"
user_name = "myusername"
user_pwd = "myuser_password"
tenant_name = "my_tenant_id"
keystone = ksclient.Client(auth_url=auth_url, username=user_name, password=user_pwd, tenant_name=tenant_name)
print keystone.auth_token
Can you please take a look at my code and see why the python code does not work?
Thanks!
You should only provide https://my_url:5000/v2.0 as the path when authenticating using the client.
See this documentation for more information.
You could also try something like this.
from keystoneauth1 import loading
from keystoneauth1 import session
from keystoneclient import client as keystoneclient
auth_url = 'https://my_url:5000'
user = '<user>'
password = '<password>'
tenant = '<tenant>'
loader = loading.get_plugin_loader('password')
keystone_auth = \
loader.load_from_options(
auth_url=auth_url,
username=user,
password=password,
user_domain_name=tenant
)
keystone_session = session.Session(
auth=keystone_auth,
verify=False
)
print(keystone_session.get_token())
Keep in mind that the Keystone V2 API has been deprecated, and removed in the latest versions of Openstack. If possible I would recommend moving over to the V3 API as soon as possible.
I need to automate download of reports from serviceNow.
I've been able to automate it using python and selenium and win32com by following method.
https://test.service-now.com/sys_report_template.do?CSV&jvar_report_id=92a....7aa
And using selenium to access serviceNow as well as modify firefox default download option to dump the file to a folder on windows machine.
However, Since all of this may be ported to a linux server , we would like to port it to SOAP or CURL.
I came across serviceNow libraries for python here.
I tried it out and following code is working if I set login , password and instance-name as listed at the site using following from ServiceNow.py
class Change(Base):
__table__ = 'change_request.do'
and following within clientside script as listed on site.
# Fetch changes updated on the last 5 minutes
changes = chg.last_updated(minutes=5)
#print changes client side script.
for eachline in changes:
print eachline
However, When I replace URL with sys_report_template.do, I am getting error
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\SOAPpy\Parser.py", line 1080, in _parseSOAP
parser.parse(inpsrc)
File "C:\Python27\Lib\xml\sax\expatreader.py", line 107, in parse
xmlreader.IncrementalParser.parse(self, source)
File "C:\Python27\Lib\xml\sax\xmlreader.py", line 125, in parse
self.close()
File "C:\Python27\Lib\xml\sax\expatreader.py", line 220, in close
self.feed("", isFinal = 1)
File "C:\Python27\Lib\xml\sax\expatreader.py", line 214, in feed
self._err_handler.fatalError(exc)
File "C:\Python27\Lib\xml\sax\handler.py", line 38, in fatalError
raise exception
SAXParseException: <unknown>:1:0: no element found
Here is relevent code
from servicenow import ServiceNow
from servicenow import Connection
from servicenow.drivers import SOAP
# For SOAP connection
conn = SOAP.Auth(username='abc', password='def', instance='test')
rpt = ServiceNow.Base(conn)
rpt.__table__ = "sys_report_template.do?CSV"
#jvar_report_id replaced with .... to protect confidentiality
report = rpt.fetch_one({'jvar_report_id': '92a6760a......aas'})
for eachline in report:
print eachline
So, my question is , what can be done to make this work?
I looked on web for resources and help, but didn't find any.
Any help is appreciated.
After much research I was able to use following method to get report in csv format from servicenow. I thought I will post over here in case anyone else runs into similar issue.
import requests
import json
# Set the request parameters
url= 'https://myinstance.service-now.com/sys_report_template.do?CSV&jvar_report_id=929xxxxxxxxxxxxxxxxxxxx0c755'
user = 'my_username'
pwd = 'my_password'
# Set proper headers
headers = {"Accept":"application/json"}
# Do the HTTP request
response = requests.get(url, auth=(user, pwd), headers=headers )
response.raise_for_status()
print response.text
response.text now has report in csv format.
I need to next figure out, how to parse the response object to extract csv data in correct format.
Once done, I will post over here. But for now this answers my question.
I tried this and its working as expected.
`import requests
import json
url= 'https://myinstance.service-now.com/sys_report_template.do?CSV&jvar_report_id=929xxxxxxxxxxxxxxxxxxxx0c755'
user = 'my_username'
pwd = 'my_password'
response = requests.get(url, auth=(user, pwd), headers=headers )
file_name = "abc.csv"
with open(file_name, 'wb') as out_file:
out_file.write(response.content)
del response`
I have tried everything I can find to get this to work...
I'm working on a plugin for a python-based task program (called GTG). I'm running Gnome on Opensuse Linux.
Code (Python 2.7):
def initialize(self):
"""
Intialize backend: try to authenticate. If it fails, request an authorization.
"""
super(Backend, self).initialize()
path = os.path.join(CoreConfig().get_data_dir(), 'backends/gtask', 'storage_file-%s' % self.get_id())
# Try to create leading directories that path
path_dir = os.path.dirname(path)
if not os.path.isdir(path_dir):
os.makedirs(path_dir)
self.storage = Storage(path)
self.authenticate()
def authenticate(self):
""" Try to authenticate by already existing credences or request an authorization """
self.authenticated = False
credentials = self.storage.get()
if credentials is None or credentials.invalid == True:
self.request_authorization()
else:
self.apply_credentials(credentials)
# Request periodic import, avoid waiting a long time
# self.start_get_tasks()
def apply_credentials(self, credentials):
""" Finish authentication or request for an authorization by applying the credentials """
http = httplib2.Http(ca_certs = '/etc/ssl/certs/ca_certs.pem', disable_ssl_certificate_validation=True)
http = credentials.authorize(http)
# Build a service object for interacting with the API.
self.service = build_service(serviceName='tasks', version='v1', http=http, developerKey='AIzaSyAmUlk8_iv-rYDEcJ2NyeC_KVPNkrsGcqU')
# self.service = build_service(serviceName='tasks', version='v1')
self.authenticated = True
def _authorization_step2(self, code):
credentials = self.flow.step2_exchange(code)
# credential = self.flow.step2_exchange(code)
self.storage.put(credentials)
credentials.set_store(self.storage)
return credentials
def request_authorization(self):
""" Make the first step of authorization and open URL for allowing the access """
self.flow = OAuth2WebServerFlow(client_id=self.CLIENT_ID,
client_secret=self.CLIENT_SECRET,
scope='https://www.googleapis.com/auth/tasks',
redirect_uri='http://localhost:8080',
user_agent='GTG')
oauth_callback = 'oob'
auth_uri = self.flow.step1_get_authorize_url(oauth_callback)
# credentials = self.flow.step2_exchange(code)
# url = self.flow.step1_get_authorize_url(oauth_callback)
browser_thread = threading.Thread(target=lambda: webbrowser.open_new(auth_uri))
browser_thread.daemon = True
browser_thread.start()
# Request the code from user
BackendSignals().interaction_requested(self.get_id(), _(
"You need to <b>authorize GTG</b> to access your tasks on <b>Google</b>.\n"
"<b>Check your browser</b>, and follow the steps there.\n"
"When you are done, press 'Continue'."),
BackendSignals().INTERACTION_TEXT,
"on_authentication_step")
def on_authentication_step(self, step_type="", code=""):
if step_type == "get_ui_dialog_text":
return _("Code request"), _("Paste the code Google has given you"
"here")
elif step_type == "set_text":
try:
credentials = self._authorization_step2(code)
except FlowExchangeError, e:
# Show an error to user and end
self.quit(disable = True)
BackendSignals().backend_failed(self.get_id(),
BackendSignals.ERRNO_AUTHENTICATION)
return
self.apply_credentials(credentials)
# Request periodic import, avoid waiting a long time
self.start_get_tasks()
The browser window opens up and I am presented with a code from Google. The program opens a small window where I can enter the code from Google.When that happens I get this in the console :
No handlers could be found for logger "oauth2client.util"
Created new window in existing browser session.
[522:549:0108/063825:ERROR:nss_util.cc(821)] After loading Root Certs, loaded==false: NSS error code: -8018
but the SSL icon is green in Chrome...
then when I submit the code, I get :
Exception in thread Thread-10:
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.7/site-packages/GTG/backends/backend_gtask.py", line 204, in on_authentication_step
credentials = self._authorization_step2(code)
File "/usr/lib/python2.7/site-packages/GTG/backends/backend_gtask.py", line 151, in _authorization_step2
credentials = self.flow.step2_exchange(code)
File "/usr/lib/python2.7/site-packages/oauth2client/util.py", line 132, in positional_wrapper
return wrapped(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/oauth2client/client.py", line 1283, in step2_exchange
headers=headers)
File "/usr/lib/python2.7/site-packages/httplib2/__init__.py", line 1586, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/usr/lib/python2.7/site-packages/httplib2/__init__.py", line 1328, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/usr/lib/python2.7/site-packages/httplib2/__init__.py", line 1250, in _conn_request
conn.connect()
File "/usr/lib/python2.7/site-packages/httplib2/__init__.py", line 1037, in connect
raise SSLHandshakeError(e)
SSLHandshakeError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
The file is called backend_gtask.py...
I have tried importing the certificate as stated here : How to update cacerts.txt of httplib2 for Github?
I have tried to disable verification (httplib2.Http(disable_ssl_certificate_validation=True)) as stated all over the web,
I have updated the python packages (which seemed to make things worse)
I have copied ca_certs.pem back and forth between /etc/ssl... and /usr/lib/python2.7/...
When I visit the auth page in a browser, it says the certificate is verified...
What else can I possibly check?
SHORT TEST CODE :
from oauth2client.client import OAuth2WebServerFlow
from oauth2client.tools import run
from oauth2client.file import Storage
CLIENT_ID = 'id'
CLIENT_SECRET = 'secret'
flow = OAuth2WebServerFlow(client_id=CLIENT_ID,
client_secret=CLIENT_SECRET,
scope='https://www.googleapis.com/auth/tasks',
redirect_uri='http://localhost:8080')
storage = Storage('creds.data')
credentials = run(flow, storage)
print "access_token: %s" % credentials.access_token
Found that here: https://github.com/burnash/gspread/wiki/How-to-get-OAuth-access-token-in-console%3F
OK...
Big thanks to Steffen Ullrich.
httplib2 version 0.9 tries to use the system certificates and not the certs.txt file that used to be shipped with it. It also enforces verification.
httplib2 can take a couple of useful parameters - notably ca_certs. Use it to point to the actual *.pem file in you ssl installation. I cannot be a folder, must be a real file.
I use the following in the initialization of the plugin :
self.http = httplib2.Http(ca_certs = '/etc/ssl/ca-bundle.pem')
Then, for all subsequent calls to httplib or google client libraries, I pass my pre-built http object as a parameter like this:
credentials = self.flow.step2_exchange(code, self.http)
self.http = credentials.authorize(self.http)
Now ssl connections work with the new httplib2...
I will eventually have to make sure the plugin can find certificates on any system, but at least I know what the problem was.
Thanks again to Steffen Ullrich for walking me through this.
See this answer for an easier fix without touching your code: just set your certificate bundle pem file path in an environment variable:
export HTTPLIB2_CA_CERTS="\path\to\your\ca-bundle"
I need to develop python soapclient for Travelport Galileo uAPI.
This is 30-day trial credentials for Travelport Universal API
Universal API User ID: Universal API/uAPI2514620686-0edbb8e4
Universal API Password: D54HWfck9nRZNPbXmpzCGwc95
Branch Code for Galileo (1G): P7004130
URLs: https://emea.universal-api.pp.travelport.com/B2BGateway/connect/uAPI/
This is quote from documentation galileo
HTTP Header
The HTTP header includes:
SOAP endpoints, which vary by:
Geographical region.
Requested service. In the preceding example, the HotelService is used for the endpoint; however, the service name is modified based on the request transaction.
gzip compression, which is optional, but strongly recommended. To accept gzip compression in the response, specify “Accept-Encoding: gzip,deflate” in the header.
Authorization, which follows the standard basic authorization pattern.
The text that follows “Authorization: Basic” can be encoded using Base 64. This functionality is supported by most programming languages.
The syntax of the authorization credentials must include the prefix "Universal API/" before the User Name and Password assigned by Travelport.
POST https://americas.universal-api.pp.travelport.com/
B2BGateway/connect/uAPI/HotelService HTTP/2.0
Accept-Encoding: gzip,deflate
Content-Type: text/xml;charset=UTF-8
SOAPAction: ""
Authorization: Basic UniversalAPI/UserName:Password
Content-Length: length
This is i my python code
import urllib2
import base64
import suds
class HTTPSudsPreprocessor(urllib2.BaseHandler):
def http_request(self, req):
message = \
"""
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:air="http://www.travelport.com/schema/air_v16_0" xmlns:com="http://www.travelport.com/schema/common_v13_0" -->
<soapenv:header>
<soapenv:body>
<air:availabilitysearchreq xmlns:air="http://www.travelport.com/schema/air_v16_0" xmlns:com="http://www.travelport.com/schema/common_v13_0" authorizedby="Test" targetbranch="P7004130">
<air:searchairleg>
<air:searchorigin>
<com:airport code="LHR">
</com:airport></air:searchorigin>
<air:searchdestination>
<com:airport code="JFK">
</com:airport></air:searchdestination>
<air:searchdeptime preferredtime="2011-11-08">
</air:searchdeptime></air:searchairleg>
</air:availabilitysearchreq>
</soapenv:body>
"""
auth = base64.b64encode('Universal API/uAPI2514620686-0edbb8e4:D54HWfck9nRZNPbXmpzCGwc95')
req.add_header('Content-Type', 'text/xml; charset=utf-8')
req.add_header('Accept', 'gzip,deflate')
req.add_header('Cache-Control','no-cache')
req.add_header('Pragma', 'no-cache')
req.add_header('SOAPAction', '')
req.add_header('Authorization', 'Basic %s'%(auth))
return req
https_request = http_request
URL = "https://emea.universal-api.pp.travelport.com/B2BGateway/connect/uAPI/"
https = suds.transport.https.HttpTransport()
opener = urllib2.build_opener(HTTPSudsPreprocessor)
https.urlopener = opener
suds.client.Client(URL, transport = https)
But it is not working.
Traceback (most recent call last):
File "soap.py", line 42, in <module>
suds.client.Client(URL, transport = https)
File "/usr/local/lib/python2.7/site-packages/suds/client.py", line 112, in __init__
self.wsdl = reader.open(url)
File "/usr/local/lib/python2.7/site-packages/suds/reader.py", line 152, in open
d = self.fn(url, self.options)
File "/usr/local/lib/python2.7/site-packages/suds/wsdl.py", line 136, in __init__
d = reader.open(url)
File "/usr/local/lib/python2.7/site-packages/suds/reader.py", line 79, in open
d = self.download(url)
File "/usr/local/lib/python2.7/site-packages/suds/reader.py", line 95, in download
fp = self.options.transport.open(Request(url))
File "/usr/local/lib/python2.7/site-packages/suds/transport/http.py", line 64, in open
raise TransportError(str(e), e.code, e.fp)
suds.transport.TransportError: HTTP Error 500: Dynamic backend host not specified
I'm trying to solve this problem for the past 2 weeks, so if you can, please advise me solution.
I think you can try to download WSDL files in ZIP archive from this url https://support.travelport.com/webhelp/uAPI/uAPI.htm#Getting_Started/Universal_API_Schemas_and_WSDLs.htm
So you will be able to generate your client classes using those WSDL files, because there is no WSDL endpoint on the https://emea.universal-api.pp.travelport.com/B2BGateway/connect/uAPI/
(like ?wsdl or /.wsdl)