I need to compare values that i have by executing wmi commands(using python) and values from inside a db.. is it best to compare them without storing in separate files or is storing and then comparing is the only possible way?
can someone pls direct me in the right way.. also, where should i look for, for getting more knowledge regarding this?
If in doubt, go for the simplest solution. In this case, compare them in memory.
If you want to be ultra-reliable (i.e. survive after crashes of your application / power outage) or cache values for long times (i.e. it's a requirement to continue working even when the database is down), you may consider files. Be warned though - anything but an extremely careful implementation (you should have lots of try..except..finallys and at least one call to flock and fsync, storing in files tends to be less reliable. So unless you're interested in consistency research and willing to put up a few weeks, go for a simple Python comparison.
Related
I need to perform searches in quite large files. The search operations need random access (think of binary search), and I will mmap the files for ease of use and performance. The search algorithm takes the page size into account so that whenever I need to access some memory area, I will try to make the most of it. Due to this there are several parameters to tune. I would like to find the parameters which give me the least number of reads from the block device.
I can do this with pen and paper, but the theoretical work carries only so far. The practical environment with a lot happening and different page caches is more complex. There are several processes accessing the files, and certain pages may usually be available in the file system page cache due to other activity on the files. (I assume the OS is aware of these when using mmap.)
In order to see the actual performance of my search algorithms in terms of number of blocks read from the block device, I would need to know the number of page misses occurring during my mmap accesses.
A dream-com-true solution would be one that would tell me which pages of the memory area are in the cache already. A very good solution would be a function which tells me whether a given page is in the real memory. This would both enable me to tune the parameters and possibly even be part of my algorithm ("if this page is in real memory, we'll extract some information out of it, if it isn't then we'll read another page").
The system will run on Linux (3-series kernel), so if there is no OS-agnostic answer, Linux-specific answers are acceptable. The benchmark will be written in python, but if the interfaces exist only in C, then I'll live with that.
Example
Let us have a file with fixed record length records carrying a sorted identifier and some data. We want to extract the data between some starting and ending position (as defined by the identifiers. The trivial solution is to use binary search to find the start position and then return everything until the end is reached.
However, the situation changes somewhat if we need to take cacheing into account. Then direct memory accesses are essentially free but page misses are expensive. A simple solution is to use binary search to find any position within the range. Then the file can be traversed backwards till the start position is reached. Then the file is traversed to the forward direction until the end is reached. This sounds quite stupid, but it ensures that once a single point within the range is found, no extra pages need to be loaded.
So, the essential thing is to find a single position within the range. Binary search is a good candidate, but if we know that, for example, the three last or three first pages of the file are usually in the page cache anyway, we should use that information, as well. If we knew which of the pages are in the cache, the search algorithm could be made much better, but even with a posteriori knowledge whether we hit or missed helps.
(The actual problem is a bit more complicated than that, but maybe this illustrates the need.)
Partial solution:
As JimB tells in his answer, there is no such API in Linux. That leaves us with more generic profiling tools (such as python's cProfile or perf stat in Linux).
The challenge with my code is that I know most of the time will be spent with the memory accesses which end up being cache misses. This is very easy, as they are the only points where the code may block. In the code I have something like b = a[i], and this will either be very fast or very slow depending on i.
Of course, seeing the total number of cache misses during the running time of the process may help with some optimizations, but I would really know if the rest of the system creates a situation where, e.g. first or last pages of the file are most of the time in the cache anyway.
So, I will implement timing of the critical memory accesses (ones that may miss the cache). As almost everything running in the system is I/O-limited (not CPU limited), it is unlikely that a context switch would too often spoil my timing. This is not an ideal solution, but it seems to be the least bad one.
This is really something that needs to be handled outside of your program. The virtual memory layer is handled by the kernel, and the details are not exposed to the process itself. You can profile your program within its process, and estimate what's happening based on the timing of your function calls, but to really see what's going on you needs to use OS specific profiling tools.
Linux has a great tool for this: perf. The perf stat command may be all you need to get an overview of how your program is executing.
Due to several edits, this question might have become a bit incoherent. I apologize.
I'm currently writing a Python server. It will never see more than 4 active users, but I'm a computer science student, so I'm planning for it anyway.
Currently, I'm about to implement a function to save a backup of the current state of all relevant variables into CSV files. Of those I currently have 10, and they will never be really big, but... well, computer science student and so on.
So, I am currently thinking about two things:
When to run a backup?
What kind of backup?
When to run:
I can either run a backup every time a variable changes, which has the advantage of always having the current state in the backup, or something like once every minute, which has the advantage of not rewriting the file hundreds of times per minute if the server gets busy, but will create a lot of useless rewrites of the same data if I don't implement a detection which variables have changed since the last backup.
Directly related to that is the question what kind of backup I should do.
I can either do a full backup of all variables (Which is pointless if I'm running a backup every time a variable changes, but might be good if I'm running a backup every X minutes), or a full backup of a single variable (Which would be better if I'm backing up each time the variables change, but would involve either multiple backup functions or a smart detection of the variable that is currently backed up), or I can try some sort of delta-backup on the files (Which would probably involve reading the current file and rewriting it with the changes, so it's probably pretty stupid, unless there is a trick for this in Python I don't know about).
I cannot use shelves because I want the data to be portable between different programming languages (java, for example, probably cannot open python shelves), and I cannot use MySQL for different reasons, mainly that the machine that will run the Server has no MySQL support and I don't want to use an external MySQL-Server since I want the server to keep running when the internet connection drops.
I am also aware of the fact that there are several ways to do this with preimplemented functions of python and / or other software (sqlite, for example). I am just a big fan of building this stuff myself, not because I like to reinvent the wheel, but because I like to know how the things I use work. I'm building this server partly just for learning python, and although knowing how to use SQLite is something useful, I also enjoy doing the "dirty work" myself.
In my usage scenario of possibly a few requests per day I am tending towards the "backup on change" idea, but that would quickly fall apart if, for some reason, the server gets really, really busy.
So, my question basically boils down to this: Which backup method would be the most useful in this scenario, and have I possibly missed another backup strategy? How do you decide on which strategy to use in your applications?
Please note that I raise this question mostly out of a general curiosity for backup strategies and the thoughts behind them, and not because of problems in this special case.
Use sqlite. You're asking about building persistent storage using csv files, and about how to update the files as things change. What you're asking for is a lightweight, portable relational (as in, table based) database. Sqlite is perfect for this situation.
Python has had sqlite support in the standard library since version 2.5 with the sqlite3 module. Since a sqlite database is implemented as a single file, it's simple to move them across machines, and Java has a number of different ways to interact with sqlite.
I'm all for doing things for the sake of learning, but if you really want to learn about data persistence, I wouldn't marry yourself to the idea of a "csv database". I would start by looking at the wikipedia page for Persistence. What you're thinking about is basically a "System Image" for your data. The Wikipedia article describes some of the same shortcomings of this approach that you've mentioned:
State changes made to a system after its last image was saved are lost
in the case of a system failure or shutdown. Saving an image for every
single change would be too time-consuming for most systems
Rather than trying to update your state wholesale at every change, I think you'd be better off looking at some other form of persistence. For example, some sort of journal could work well. This makes it simple to just append any change to the end of a log-file, or some similar construct.
However, if you end up with many concurrent users, with processes running on multiple threads, you'll run in to concerns of whether or not your changes are atomic, or if they conflict with one another. While operating systems generally have some ways of dealing with locking files for edits, you're opening up a can of worms trying to learn about how that works and interacts with your system. At this point you're back to needing a database.
So sure, play around with a couple different approaches. But as soon as you're looking to just get it working in a clear and consistent manner, go with sqlite.
If your data is in CSV files, why not use a revision control system on those files? E.g. git would be pretty fast and give excellent history. The repository would be wholly contained in the directory where the files reside, so it's pretty easy to handle. You could also replicate that repository to other machines or directories easily.
I have a situation where various analysis programs output large amounts of data, but I may only need to manipulate or access certain parts of the data in a particular Excel workbook.
The numbers might often change as well as newer analyses are run, and I'd like these changes to be reflected in Excel in as automated a manner as possible. Another important consideration is that I'm using Python to process some of the data too, so putting the data somewhere where it's easy for Python and Excel to access would be very beneficial.
I know only a little about databases, but I'm wondering if using one would be a good solution for what my needs - Excel has database interaction capability as far as I'm aware, as does Python. The devil is in the details of course, so I need some help figuring out what system I'd actually set up.
From what I've currently read (in the last hour), here's what I've come up with so far simple plan:
1) Set up an SQLite managed database. Why SQLite? Well, I don't need a database that can manage large volumes of concurrent accesses, but I do need something that is simple to set up, easy to maintain and good enough for use by 3-4 people at most. I can also use the SQLite Administrator to help design the database files.
2 a) Use ODBC/ADO.NET (I have yet to figure out the difference between the two) to help Excel access the database. This is going to be the trickiest part, I think.
2 b) Python already has the built in sqlite3 module, so no worries with the interface there. I can use it to set up the output data into an SQLite managed database as well!
Putting down some concrete questions:
1) Is a server-less database a good solution for managing my data given my access requirements? If not, I'd appreciate alternative suggestions. Suggested reading? Things worth looking at?
2) Excel-SQLite interaction: I could do with some help flushing out the details there...ODBC or ADO.NET? Pointers to some good tutorials? etc.
3) Last, but not least, and definitely of concern: will it be easy enough to teach a non-programmer how to setup spreadsheets using queries to the database (assuming they're willing to put in some time with familiarization, but not very much)?
I think that about covers it for now, thank you for your time!
Although you could certainly use a database to do what you're asking, I'm not sure you really want to add that complexity. I don't see much benefit of adding a database to your mix. ...if you were pulling data from a database as well, then it'd make more sense to add some tables for this & use it.
From what I currently understand of your requirements, since you're using python anyway, you could do your preprocessing in python, then just dump out the processed/augmented values into other csv files for Excel to import. For a more automated solution, you could even write the results directly to the spreadsheets from Python using something like xlwt.
As part of a larger project, I want the ability to take two bodies of text and hand them to a merge algorithm which returns either an auto-merged result (in cases where the changes are not conflicting) or throws an error and (potentially) produces a single text document with the conflicting changes highlighted.
Basically, I just want a programmatic way to do what every source control system on the planet does internally, but I'm having a hard time finding it. There are tons of visual GUIs for doing this sort of thing that dominate my search results, but none of them seem to make easily accessible the core merging algorithm. Does everyone rely on some common and well understood algorithm/library and I just don't know the name so I'm having a hard time searching for it? Is this some just minor tweak on diff and I should be looking for diff libraries instead of merge libraries?
Python libraries would be most helpful, but I can live with the overhead of interfacing with some other library (or command line solution) if I have to; this operation should be relatively infrequent.
You're probably searching for merge algorithms like 3-way merging, which you can find in many open source projects, e.g. in the bazaar VCS (merge3.py source).
Did you check out difflib
http://docs.python.org/library/difflib.html
i am reading a csv file into a list of a list in python. it is around 100mb right now. in a couple of years that file will go to 2-5gigs. i am doing lots of log calculations on the data. the 100mb file is taking the script around 1 minute to do. after the script does a lot of fiddling with the data, it creates URL's that point to google charts and then downloads the charts locally.
can i continue to use python on a 2gig file or should i move the data into a database?
I don't know exactly what you are doing. But a database will just change how the data is stored. and in fact it might take longer since most reasonable databases may have constraints put on columns and additional processing for the checks. In many cases having the whole file local, going through and doing calculations is going to be more efficient than querying and writing it back to the database (subject to disk speeds, network and database contention, etc...). But in some cases the database may speed things up, especially because if you do indexing it is easy to get subsets of the data.
Anyway you mentioned logs, so before you go database crazy I have the following ideas for you to check out. Anyway I'm not sure if you have to keep going through every log since the beginning of time to download charts and you expect it to grow to 2 GB or if eventually you are expecting 2 GB of traffic per day/week.
ARCHIVING -- you can archive old logs, say every few months. Copy the production logs to an archive location and clear the live logs out. This will keep the file size reasonable. If you are wasting time accessing the file to find the small piece you need then this will solve your issue.
You might want to consider converting to Java or C. Especially on loops and calculations you might see a factor of 30 or more speedup. This will probably reduce the time immediately. But over time as data creeps up, some day this will slow down as well. if you have no bound on the amount of data, eventually even hand optimized Assembly by the world's greatest programmer will be too slow. But it might give you 10x the time...
You also may want to think about figuring out the bottleneck (is it disk access, is it cpu time) and based on that figuring out a scheme to do this task in parallel. If it is processing, look into multi-threading (and eventually multiple computers), if it is disk access consider splitting the file among multiple machines...It really depends on your situation. But I suspect archiving might eliminate the need here.
As was suggested, if you are doing the same calculations over and over again, then just store them. Whether you use a database or a file this will give you a huge speedup.
If you are downloading stuff and that is a bottleneck, look into conditional gets using the if modified request. Then only download changed items. If you are just processing new charts then ignore this suggestion.
Oh and if you are sequentially reading a giant log file, looking for a specific place in the log line by line, just make another file storing the last file location you worked with and then do a seek each run.
Before an entire database, you may want to think of SQLite.
Finally a "couple of years" seems like a long time in programmer time. Even if it is just 2, a lot can change. Maybe your department/division will be laid off. Maybe you will have moved on and your boss. Maybe the system will be replaced by something else. Maybe there will no longer be a need for what you are doing. If it was 6 months I'd say fix it. but for a couple of years, in most cases, I'd say just use the solution you have now and once it gets too slow then look to do something else. You could make a comment in the code with your thoughts on the issue and even an e-mail to your boss so he knows it as well. But as long as it works and will continue doing so for a reasonable amount of time, I would consider it "done" for now. No matter what solution you pick, if data grows unbounded you will need to reconsider it. Adding more machines, more disk space, new algorithms/systems/developments. Solving it for a "couple of years" is probably pretty good.
If you need to go through all lines each time you perform the "fiddling" it wouldn't really make much difference, assuming the actual "fiddling" is whats eating your cycles.
Perhaps you could store the results of your calculations somehow, then a database would probably be nice. Also, databases have methods for ensuring data integrity and stuff like that, so a database is often a great place for storing large sets of data (duh! ;)).
I'd only put it into a relational database if:
The data is actually relational and expressing it that way helps shrink the size of the data set by normalizing it.
You can take advantage of triggers and stored procedures to offload some of the calculations that your Python code is performing now.
You can take advantage of queries to only perform calculations on data that's changed, cutting down on the amount of work done by Python.
If neither of those things is true, I don't see much difference between a database and a file. Both ultimately have to be stored on the file system.
If Python has to process all of it, and getting it into memory means loading an entire data set, then there's no difference between a database and a flat file.
2GB of data in memory could mean page swapping and thrashing by your application. I would be careful and get some data before I blamed the problem on the file. Just because you access the data from a database won't solve a paging problem.
If your data's flat, I see less advantage in a database, unless "flat" == "highly denormalized".
I'd recommend some profiling to see what's consuming CPU and memory before I made a change. You're guessing about the root cause right now. Better to get some data so you know where the time is being spent.
I always reach for a database for larger datasets.
A database gives me some stuff for "free"; that is, I don't have to code it.
searching
sorting
indexing
language-independent connections
Something like SQLite might be the answer for you.
Also, you should investigate the "nosql" databases; it sounds like your problem might fit well into one of them.
At 2 gigs, you may start running up against speed issues. I work with model simulations for which it calls hundreds of csv files and it takes about an hour to go through 3 iterations, or about 20 minutes per loop.
This is a matter of personal preference, but I would go with something like PostGreSql because it integrates the speed of python with the capacity of a sql-driven relational database. I encountered the same issue a couple of years ago when my Access db was corrupting itself and crashing on a daily basis. It was either MySQL or PostGres and I chose Postgres because of its python friendliness. Not to say MySQL would not work with Python, because it does, which is why I say its personal preference.
Hope that helps with your decision-making!