I have a Python script that takes advantage of the latest Vimeo API (https://developer.vimeo.com/api/) to upload some videos to my Vimeo account.
Here is what, in a slightly simplified form, the script basically does:
from vimeo import VimeoClient
vimeo = VimeoClient('my_token_here')
uid = vimeo.upload('/path/to/file.mov')
When file.mov is 3MB or less everything works fine and the file is successfully uploaded. However, for larger files I get a timeout error:
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/home/fabio/.virtualenvs/venv/src/vimeo/vimeo/uploads.py", line 79, in __call__
return do_upload()
File "/home/fabio/.virtualenvs/venv/src/vimeo/vimeo/uploads.py", line 70, in do_upload
self.upload_segment(upload_uri, _range, video_data, filetype or 'mp4')
File "/home/fabio/.virtualenvs/venv/src/vimeo/vimeo/uploads.py", line 135, in upload_segment
body=data, headers=request_headers)
File "/home/fabio/.virtualenvs/venv/lib/python2.7/site-packages/tornado/httpclient.py", line 85, in fetch
self._async_client.fetch, request, **kwargs))
File "/home/fabio/.virtualenvs/venv/lib/python2.7/site-packages/tornado/ioloop.py", line 389, in run_sync
return future_cell[0].result()
File "/home/fabio/.virtualenvs/venv/lib/python2.7/site-packages/tornado/concurrent.py", line 131, in result
return super(TracebackFuture, self).result(timeout=timeout)
File "/home/fabio/.virtualenvs/venv/lib/python2.7/site-packages/tornado/concurrent.py", line 65, in result
raise self._exception
HTTPError: HTTP 599: Timeout
This is the vimeo library I am using: https://github.com/vimeo/vimeo.py.
And the Tornado library in my virtual environment is updated to the 3.2.1 version.
Any tips for me?
From the Tornado source, the default request timeout for an HTTPClient which vimeo is using is 20 seconds. It looks like the Vimeo library attempts to upload as much of the video as possible, and then queries the server to see how much was successfully uploaded. It is likely that it is taking over 20 seconds to upload your video and as a result timing out. I'm not convinced they handle this properly though, as you get a timeout error from Tornado, but it seems like they want to support the whole file not being uploaded at once.
You could try modifying the vimeo library code that I linked above to have a much longer timeout by changing the linked line in your local copy to something like:
r = HTTPClient().fetch(upload_uri, method="PUT",
body=data, headers=request_headers,
request_timeout=9999.0)
If that doesn't work you could try raising an issue on their github issues tracker, and someone who actually works on the project might be able to help you further.
Related
I ask this question having run out of ideas - I assumed I was doing something stupid but even if I am I can't figure out what... I'm new to python having drifted away from coding 10 years ago but I was reasonably proficient once :-D I've written a python script to query a google CSE and search for relevant items, and the idea is that it will check a Podio materials list to see if they are already listed there. So at the moment all I want to do is authenticate with Podio and grab items so I can query them in my app.
I'm trying to connect to the Podio API using the prescribed method in the docs (and in questions here, and everywhere else I have looked)
At this point literally all this script is doing is this except for the credentials replaced with the actual strings, just in case I was missing some python variable madness.
import pypodio2
from pypodio2.api import OAuthClient
c = OAuthClient(
"<myappID>",
'<myAPIkey>',
'<myusername>',
'<mypassword>'
)
I'm using pythonanywhere as a sandbox and I am running the script from a bash console, I get this
16:26 ~/mysite $ python apitest.py
Traceback (most recent call last):
File "apitest.py", line 15, in <module>
'Splat_100'
File "/home/Trebuchet/.local/lib/python2.7/site-packages/pypodio2/api.py", line 13, in OAuthClient
api_key, api_secret, domain)
File "/home/Trebuchet/.local/lib/python2.7/site-packages/pypodio2/transport.py", line 41, in __init__
urlencode(body), headers=headers)
File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 2135, in request
cachekey,
File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1796, in _request
conn, request_uri, method, body, headers
File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1701, in _conn_request
conn.connect()
File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1358, in connect
sock.connect((self.host, self.port) + sockaddr[:2])
File "/usr/local/lib/python2.7/dist-packages/httplib2/socks.py", line 496, in connect
self.__negotiatehttp(destpair[0], destpair[1])
File "/usr/local/lib/python2.7/dist-packages/httplib2/socks.py", line 457, in __negotiatehttp
raise HTTPError((statuscode, statusline[2]))
httplib2.socks.HTTPError: (403, 'Forbidden'
The password, key etc are correct. The username is correct. The clientID is correct. So I must be missing something that is obvious to people used to working with APIs. Mustn't I? Thanks :)
Free accounts on PythonAnywhere can only access http(s) sites that are on our whitelist. If the site you're using has a publicly documented API, send a link to it to PythonAnywhere support (support#pythonanywhere.com) and we'll consider add it to the whitelist.
I know just enough devops to be dangerous. I've successfully deployed a VERY simple python flask app to App Engine that basically publishes received post data as a message to PubSub. It is almost identical to Google's sample code to do so. Only difference is it uses a service account I push with the app repository to access PubSub to circumvent this issue.
Works very well so far, but I've started seeing a very small number of errors around starting a new thread in threading.py:
1)
Traceback (most recent call last):
File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 33, in grpc._cython.cygrpc._spawn_callback_async
File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 24, in grpc._cython.cygrpc._spawn_callback_in_thread
File "/usr/lib/python2.7/threading.py", line 736, in start
_start_new_thread(self.__bootstrap, ())
thread.error: can't start new thread
2)
Traceback (most recent call last):
File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 33, in grpc._cython.cygrpc._spawn_callback_async
File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 24, in grpc._cython.cygrpc._spawn_callback_in_thread
3)
Traceback (most recent call last):
File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 33, in grpc._cython.cygrpc._spawn_callback_async
File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 33, in grpc._cython.cygrpc._spawn_callback_async
File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 24, in grpc._cython.cygrpc._spawn_callback_in_thread
File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 24, in grpc._cython.cygrpc._spawn_callback_in_thread
File "/usr/lib/python2.7/threading.py", line 736, in start
File "/usr/lib/python2.7/threading.py", line 736, in start
I have 2 questions, in order of importance:
This is an app that basically needs 100% uptime in order to not lose data (not confident the clients attempt retries if there is an error on my server side). Are these errors internal to how App Engine is managing my app's resources, and not resulting in errors handling actual requests? How can I determine if I ever responded with an HTTP error/didn't successfully handle a request? I don't see any errors in my nginx logs...is that the place I need to look to see if anything failed?
Is there a way I can fix this error?
https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/pubsub/google/cloud/pubsub_v1/publisher/client.py#L143
it looks like publisher.publish(topic_path, data=data) is an async operation, returning a concurrent.futures.Future object
Have you trying calling the Future's result()? https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Future.result
This will block until the future object is successful, fails, or timesout.
You could then forward that result as your HTTP response.
Hopefully, the result object will give you more information about the error.
Ended up changing the methodology a bit. Instead of posting a pubsub message then having dataflow ingest through GCS to BigQuery I decided to stream directly into BQ using the BigQuery python client. Updated dependencies for the python flask app to:
Flask==1.0.2
google-cloud-pubsub==0.39.1
gunicorn==19.9.0
google-cloud-bigquery==1.11.2
and I am no longer seeing any of those exceptions. It's worth noting that I'm still using a service account .json credentials file in the same directory as the app source, and I'm creating the BigQuery client with
bq_client = bigquery.Client.from_service_account_json(BQ_SVC_ACCT_FILE).
For anyone else with similar issues I'd recommend updating your dependencies (esp any Google Cloud client libraries) and create the Client you need from a local service account credentials file. I attempted to use the inherited Compute engine environment credentials (basically the default project compute engine service account) but that was less stable than pushing up an actual credential file and using that locally. However...assess your own security needs before doing the same.
I'm developing a website using the Bottle framework and have been using the Cork library to handle authentication.
The website is nearing completion and I have started to look into deployment. I've looked into using Gunicorn as I had heard good things but the change over has created an unusual bug that I am not sure how to remedy.
Previously, I ran my site using the inbuilt development web server but I tried to use Gunicorn instead, and this seemed to work except trying to login generated a 500 server error and this:
`Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/bottle.py", line 862, in _handle
return route.call(**args)
File "/usr/local/lib/python2.7/dist-packages/bottle.py", line 1740, in wrapper
rv = callback(*a, **ka)
File "/home/shaunf/PycharmProjects/final-project/bottle_app.py", line 75, in login
aaa.login(username, password, success_redirect='/', fail_redirect='/login')
File "/usr/local/lib/python2.7/dist-packages/cork/cork.py", line 130, in login
self._setup_cookie(username)
File "/usr/local/lib/python2.7/dist-packages/cork/cork.py", line 603, in _setup_cookie
session['username'] = username
TypeError: 'NoneType' object does not support item assignment
`
I tried changing back to the development server but that hasn't helped.
I'm not really sure what to try next and any ideas or hints as to what I could trouble would be really appreciated.
I'm getting some weird behavior -- when the application starts up a new instance for the first time, I get a DeadlineExceededError. When I hit refresh in the browser it works just fine And it doesn't matter which page I try. The strange thing is I can see all my debugging code just fine. In fact, I write to the log just prior to calling self.response and it shows up in the console's log. This is pretty hard to troubleshoot, since I'm not having any page load problems in the development environment, and the traceback is a bit opaque to me:
E 2013-09-29 00:10:03.975
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 267, in Handle
for chunk in result:
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/appstats/recording.py", line 1286, in appstats_wsgi_wrapper
end_recording(status)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/appstats/recording.py", line 1410, in end_recording
rec.save()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/appstats/recording.py", line 654, in save
key, len_part, len_full = self._save()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/appstats/recording.py", line 678, in _save
namespace=config.KEY_NAMESPACE)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/memcache/__init__.py", line 1008, in set_multi
namespace=namespace)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/memcache/__init__.py", line 907, in _set_multi_with_policy
status_dict = rpc.get_result()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 612, in get_result
return self.__get_result_hook(self)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/memcache/__init__.py", line 974, in __set_with_policy_hook
rpc.check_success()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 578, in check_success
self.__rpc.CheckSuccess()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_rpc.py", line 133, in CheckSuccess
raise self.exception
DeadlineExceededError
I 2013-09-29 00:10:03.988
This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application.
I'm not sure how to even go about debugging this, since the error seems to be after all my code has already run.
Edit: I should add this:
I 2013-09-29 00:09:06.919
DEBUG: Writing output!
E 2013-09-29 00:10:03.975
You can see there's nearly a full minute between logging "Writing output!" just before self.response is called, and when the error occurs.
Deadlineexceedederror happens in app engine, if any request to a frontend instance does not get a response within 60 seconds. So what is happening in your case must be that when there are no running instance and your app receives a new user request, a new instance is started for processing. This will lead to an overall response time = instance startup time like library loading and initial data access + the time for processing the user request and this causes a deadlineexceeded error. Then when you are accessing your app immediately , there is an already running instance and so response time = the time for processing the user request and you do not get any error.
Please check the suggested approaches for handling deadlineexceedederror including warmup requests, which is like keeping an instance ready before arrival of a live user request.
Since yesterday a working Python gdata program has stopped working after I changed the IP address used.
I receive the following stack trace:
Traceback (most recent call last):
File "C:\prod\googleSite\googleSite2.py", line 23, in
feed = client.GetContentFeed()
File "C:\Python27\lib\site-packages\gdata\sites\client.py", line 155, in get_c
ontent_feed
auth_token=auth_token, **kwargs)
File "C:\Python27\lib\site-packages\gdata\client.py", line 635, in get_feed
**kwargs)
File "C:\Python27\lib\site-packages\gdata\client.py", line 320, in request
RequestError)
gdata.client.RequestError: Server responded with: 500, Internal Error
The code is as follow:
import gdata.sites.client
import gdata.sites.data
client = gdata.sites.client.SitesClient(source='xxx', site='yyy')
client.ssl = True # Force API requests through HTTPS
client.ClientLogin('user#googlemail.com', 'password', client.source);
feed = client.GetContentFeed();
Update:
The issue fixes itself after an hour - is there any kind of commit or logout to avoid this?
Since you're not passing anything in GetContentFeed, it's using CONTENT_FEED_TEMPLATE % (self.domain, self.site) as the URI. I'm not sure if the IP change had an impact on what the self.domain/self.site values should be, but it might be worth checking those out.