I'm using python GData to work with Google Calendar. I'm making really simple requests (e.g. create event), using OAuth authorization.
Usually this works OK, but sometimes I'm receiving lots of 302 redirects, that leads to "Maximum redirects count reached" exception.
If I re-try same request, it's usually works correct.
I can't figure out, why is this happening, looks like it's a random event.
As a walkthrough I wrote a code which retries requests few times, if there is such error, but may be there is an explanation of this behavior or even solutions to evade it?
Answer from Google support forum:
This might happen due to some issues in the Calendar servers and is not an error on your part. The best way to "resolve" this issue is to retry again.
Related
The website i scapped blocked me out by showing 406 Not Acceptable on the browser. It might i mistakenly sent too many requests at once on phython code.
So i put time.sleep(10) for each loop to not make it look like a DDoS attack, and it seems worked out.
My questions are:
How long would it be reasonable to send between each request? Sleep 10 seconds for each loop makes my code running too slow.
How to fix the 406 Not Acceptable error on my browsers? They still block me out, only if i chance my ip address but it's not permanent solution.
Thank you all for your answers and comments. Good day!
Any rate-limit errors are all subject to which website you choose to scrape / interact with. I could set up a website that only allows you to view it once per day, before throwing HTTP errors at your screen. So to answer your first question, there is no definitive answer. You must test for yourself and see what's the fastest speed you can go, without getting blocked.
However, there is a workaround. If you use proxies, then it's almost impossible to detect and stop the requests from executing, and therefore you will not be hit by any HTTP errors. HOWEVER, JUST BECAUSE YOU CAN, DOESN'T MEAN THAT YOU SHOULD- I am a programmer, not a lawyer. I'm sure there's a rule somewhere that says that spamming a page, even after it tells you to stop, is illegal.
Your second question isn't exactly related to programming, but I will answer it anyways- try clearing your cookies or refreshing your IP (try using a VPN or such). Other than changing your IP or cookies, there's not many more ways that a page can fingerprint you (in order to block you).
I wrote script in Python, which send requests to google and make something with results.
I used gogole library, which I found somewhere in Internet :)
When I requested many times ( aprox. 50) Google blocks another requests(503 Error), regardless time.sleep(2) in every iteration.
Above google library uses urllib and BeautifulSoup. Do you know what can make errors or maybe do you know what library is better?
Tomek
I have just began to learn AppEngine with python and in my project I'm building an application in which I need to retrieve all users of my domain.
I used RetrieveAllOrgUsers and RetrieveAllUsers to get users of my domain, but both of them crash when I deploy the application on my AppEngine account. I get the error deadlineexceeded.
Could someone please help me to resolve this issue? I guess maybe I need to use tasks, but I don't know how.
You should spend some time on reading about the limitations of the platform you are using.
Do you understand why/what a DeadelinExceededError means.
That should be you starting point.
Also have a look at the stack trace in the log, and you will see where in you code the error has occurred.
Basically you have a limited amount of time to do things in front end requests. If you exceed that time , then you will get this error (there are other reasons) .
I assume your using the provisioning api, and that could be something that takes some serious time. There are other limits you will need to contend with, such as those around URLFetch as well.
Tasks will more than likely be your solution, but you should try and understand why before embarking on that.
Have a read of https://developers.google.com/appengine/articles/deferred which is a easy path into tasks using deferred.
So many times I have seen this screen:
Server error
The website encountered an error while retrieving http://localhost:8080/. It may be down for maintenance or configured incorrectly.
Here are some suggestions:
Reload this webpage later.
HTTP Error 500 (Internal Server Error):
An unexpected condition was encountered while the server was attempting to fulfill the request.
Obviously, that gives absolutely no useful information. So I have to systematically undo everything I've previously coded, with each step reloading the page to see if I've found what ever it is that app engine doesn't like, and the whole process is just... demoralizing.
I just spent a half-hour digging through my code trying to figure out what was broken. Turned out GAE didn't like a module I was importing -- note that I still have no idea why it doesn't like it, but that's beside the point.
My question is this, is there a way to have report any and all errors? I have Debug set to true, but it very finicky about when it will actually print out the stack trace.
Whatever framework you're using is generating that. There's probably an exception handler that will generate your 500 message. You'll have to find that and adjust it to not catch all exceptions.
Can't help you figure out where though, I'm not sure what framework you're using.
By default - for most frameworks - exceptions are logged rather than output to the user, since there's no point in showing your user a stacktrace of your app's internal details. If you're running the dev_appserver, these show up in the console window; in production these can be accessed via the log viewer.
I have been running a service for some months now using Google's
OAuth2 authentication. Most of the time everything works well, but
there are occasional issues with callbacks coming back empty from Google to me: Something along the lines of 1 out of 15 callbacks arrives at my service completely without parameters in the GET request. Just a naked /my-callback-url request, no parameters at all.
I'm having quite some difficulty explaining this behaviour and neither can I find many references to it when searching the net.
I'm also so far unable to re-create this phenomenon in my own development environment, so my solution ideas have had to be mostly speculation: My first hunch at a quick-n-dirty work around was to re-generate the OAuth request URL and return a 302 redirect response back so Google can have another go. But that sounds like taking the risk of creating an infinite redirect loop if it would turn out that the problem originates from my code. I would very much prefer to understand what's actually going on.
Do any of you have experience of 'empty' OAuth2 callbacks from Google?
And in that case, what would be the most sensible way of handling
them? Or are there a typical error when generating the authentication
URL's that causes this behaviour (I'm using Python &
Requests-OAuthlib)
to handle my OAuth2 interaction).
I suspect that these requests are not redirects back from Google. There are crawlers and other hackers trying to hit every endpoint that they find on the web. So these could be just abusive requests.
If you can correlate the request with an empty parameter with a request that redirected from your server (based on IP address or a cookie you set before redirecting to Google) then we can try to investigate further.