I've a python based SSRS report generation utility that I'm using to generate multiple reports (often 100+). The way it's setup is -
Multiple threads are invoked using threading.Thread and each of them is given a dictionary.
Each thread parses the dictionary and calls rs.exe passing in relevant arguments via python's subprocess.call
Reports get generated with the following caveats -
If there are around 20-30 reports everything works fine without much issues.
If the number of reports go beyond 40-50+ (for reasons unknown to me so far), some of the reports don't get rendered and come back with error
as obtained by subprocess.call non-zero status (Error message from subprocess.call does not point to any real error).
But there is no error in those rs.exe commands, as they get rendered when i run them from windows command prompt.
Additionally when i try to re-run all those failed reports they get rendered. There's no change in the commands or data while they're being re-run.
To work around this, I employed a retry logic for 2 iterations which seems to fix the issue at times. However when the reports go beyond 100/150+
even the retry doesn't work. Now i could extend the retry logic to keep retrying until all the reports are rendered and whatever failures happen
are genuine ones (like RDL not found, corrupted and so on). But before i do any such thing, want to know if there's any limitation on how many
rs.exe can be launched simultaneously or if there's any limitation on python's subproces.call when invoked in a multi-threaded context.
Can someone please share their expertise if they've faced this kind of issue and resolved it?
Thanks.
I suspect the limit you are hitting is not rs.exe itself but the target Report Server. This will use as much physical memory as is available but when that is exhausted, further requests will start to fail. This is described in the SSRS doco:
https://msdn.microsoft.com/en-us/library/ms159206.aspx
To avoid this issue and leave some server resources for other users, I would reduce your thread limit as low as you can stand - ideally to 1.
Related
Perhaps this is a broad question, but I haven't found an answer elsewhere, so here goes.
The Python script I'm writing needs to run constantly (in a perfect world, I recognize this may not be exactly possible) on a deployed device. I've already dedicated time to adding "try...except" statements throughout so that, should an issue arise, the script will recover and continue to work.
The issue is that I'm not sure I can (nor should) handle every single possible exception that may be thrown. As such, I've decided it may be better to allow the script to die and to use systemd to restart it.
The three options:
Making no attempt to handle any exception, and just allowing systemd to restart it whenever it dies.
Meticulously creating handlers for every possible exception to guarantee that, short of loss of power, interpreter bug, or heat death of the universe, the script will always run.
A mix of the two -- making an effort to prevent crashes in some cases while allowing them in others and letting systemd restart the script.
The third choice seems the most reasonable to me. So the question is this: What factors should be considered when optimizing between "crash-proof" code and allowing a crash and restart by systemd?
For some more application specific information: there is a small but noticeable overhead involved with starting the script, the main portion will run between 50 to 100 times per second, it is not "mission critical" in that there will be no death/damage in the event of failure (just some data loss), and I already expect intermittent issues with the network it will be on.
All known exceptional cases should be handled. Any undefined behavior is a potential security issue.
As you suggest, it is also prudent to plan for unknown exceptions. Perhaps there's also a small memory leak that will also cause the application to crash even when it's running correctly. So, it's still prudent to have systemd automatically restart it if it fails, even when all expected failure modes have been handled.
I'm working on a project to learn Python, SQL, Javascript, running servers -- basically getting a grip of full-stack. Right now my basic goal is this:
I want to run a Python script infinitely, which is constantly making API calls to different services, which have different rate limits (e.g. 200/hr, 1000/hr, etc.) and storing the results (ints) in a database (PostgreSQL). I want to store these results over a period of time and then begin working with that data to display fun stuff on the front. I need this to run 24/7. I'm trying to understand the general architecture here, and searching around has proven surprisingly difficult. My basic idea in rough pseudocode is this:
database.connect()
def function1(serviceA):
while(True):
result = makeAPIcallA()
INSERT INTO tableA result;
if(hitRateLimitA):
sleep(limitTimeA)
def function2(serviceB):
//same thing, different limits, etc.
And I would ssh into my server, run python myScript.py &, shut my laptop down, and wait for the data to roll in. Here are my questions:
Does this approach make sense, or should I be doing something completely different?
Is it considered "bad" or dangerous to open a database connection indefinitely like this? If so, how else do I manage the DB?
I considered using a scheduler like cron, but the rate limits are variable. I can't run the script every hour when my limit is hit say, 5min into start time and has a wait time of 60min after that. Even running it on minute intervals seems messy: I need to sleep for persistent rate limit wait times which will keep varying. Am I correct in assuming a scheduler is not the way to go here?
How do I gracefully handle any unexpected potentially fatal errors (namely, logging and restarting)? What about manually killing the script, or editing it?
I'm interested in learning different approaches and best practices here -- any and all advice would be much appreciated!
I actually do exactly what you do for one of my personal applications and I can explain how I do it.
I use Celery instead of cron because it allows for finer adjustments in scheduling and it is Python and not bash, so it's easier to use. I have different tasks (basically a group of API calls and DB updates) to different sites running at different intervals to account for the various different rate limits.
I have the Celery app run as a service so that even if the system restarts it's trivial to restart the app.
I use the logging library in my application extensively because it is difficult to debug something when all you have is one difficult to read stack trace. I have INFO-level and DEBUG-level logs spread throughout my application, and any WARNING-level and above log gets printed to the console AND gets sent to my email.
For exception handling, the majority of what I prepare for are rate limit issues and random connectivity issues. Make sure to surround whatever HTTP request you send to your API endpoints in try-except statements and possibly just implement a retry mechanism.
As far as the DB connection, it shouldn't matter how long your connection is, but you need to make sure to surround your main application loop in a try-except statement and make sure it gracefully fails by closing the connection in the case of an exception. Otherwise you might end up with a lot of ghost connections and your application not being able to reconnect until those connections are gone.
I've got a web.py server that has a problem: it's supposed to take input from the user and process it. Some of this processing takes place in a .pyd that I didn't write and I can't debug or rebuild, and it has some bugs. There are certain inputs that can cause it to throw an Access Violation, which for some reason cannot be caught and causes python.exe to crash, bringing down the server.
This is unacceptable, but also unavoidable, so I need to change the rules a little.
What I'd like to do is to move the unstable functionality, the interface to which is already cleanly contained in a single .py file, out of the server and have the server launch it as a separate process. If it works successfully, it should print the output, which (conveniently enough) is a text string. If not, the server should detect that the separate process crashed, and return HTTP 500. Unfortunately, I don't know how to do multi-process work in Python.
How would I implement this?
Is it possible to submit multiple sequences to the Bio.Blast.NCBIWWW module at the same time? I've tried to create a function that runs my blast and have several of them run using multiprocessing, but I think the NCBI server boots me after a while and the connection stops working.
I don't know what sort of limits NCBI has on their service, but you may want to look into installing BLAST locally and running your queries that way. Biopython has support for local BLAST: http://biopython.org/DIST/docs/tutorial/Tutorial.html#sec96
Here they detail how to properly use it:
http://www.ncbi.nlm.nih.gov/BLAST/Doc/node60.html
Do not launch more than 50 threads.
Wait for the RID of a BLAST before starting the next one.
Flooding the server can lead to many problems and eventually we may be forced to block access from sites which flood the severs with no warning. We strongly suggest you limit your scripts to not send a request until you receive a RID from the server. Alternatively please introduce a "sleep" command in your script which sends request no less then one once per three seconds.
Biopython does the wait for the RID for you, but if you launch several queries you are certainly going to be banned.
I have over 100 web servers instances running a php application using apc and we occasionally (order of once per week across the entire fleet) see a corruption to one of the caches which results in a distinctive error log message.
Once this occurs then the application is dead on that node any transactions routed to it will fail.
I've written a simple wrapper around tail -F which can spot the patter any time it appears in the log file and evaluate a shell command (using bash eval) to react. I have this using the salt-call command from salt-stack to trigger processing a custom module which shuts down the nginx server, warms (refreshes) the cache, and, of course, restarts the web server. (Actually I have two forms of this wrapper, bash and Python).
This is fine and the frequency of events is such that it's unlikely to be an issue. However my boss is, quite reasonably, concerned about a common mode failure pattern ... that the regular expression might appear in too many of these logs at once and take town the entire site.
My first thought would be to wrap my salt-call in a redis check (we already have a Redis infrastructure used for caching and certain other data structures). That would be implemented as an integer, with an expiration. The check would call INCR, check the result, and sleep if more than N returned (or if the Redis server were unreachable). If the result were below the threshold then salt-call would be dispatched and a decrement would be called after the server is back up and running. (Expiration of the Redis key would kill off any stale increments after perhaps a day or even a few hours ... our alerting system will already have notified us of down servers and our response time is more than adequate for such time frames).
However, I was reading about the Saltstack event handling features and wondering if it would be better to use that instead. (Advantage, the nodes don't have redis-cli command tool nor the Python Redis libraries, but, obviously, salt-call is already there with its requisite support). So using something in Salt would minimize the need to add additional packages and dependencies to these systems. (Alternatively I could just write all the Redis handling as a separate PHP command line utility and just have my shell script call that).
Is there a HOWTO for writing simple Saltstack modules? The docs seem to plunge deeply into reference details without any orientation. Even some suggestions about which terms to search on would be helpful (because their use of terms like pillars, grains, minions, and so on seems somewhat opaque).
The main doc for writing a Salt module is here: http://docs.saltstack.com/en/latest/ref/modules/index.html
There are many modules shipped with Salt that might be helpful for inspiration. You can find them here: https://github.com/saltstack/salt/tree/develop/salt/modules
One thing to keep in mind is that the Salt Minion doesn't do anything unless you tell it to do something. So you could create a module that checks for the error pattern you mention, but you'd need to add it to the Salt Scheduler or cron to make sure it gets run frequently.
If you need more help you'll find helpful people on IRC in #salt on freenode.