Dynamic change in Configuration to reflect in user defined Data structures - python

In my program, I read a "configuration file" and from that I initialize many classes. I need a way so that Dynamic changes in the configuration file, can successfully update all the classes.
What is the best way this can be achived in Python ?
As an example:
The /etc/passwd file consists of
Username:Password:User ID:Group ID:User ID Info:Home directory:Shell
My program Initializes User Defined classes for each user based on the input in /etc/passwd file. If one or more attributes in a user entry changes in the file dynamically, how could this be transparently applied to re-initialze the User Defined Classes ?
PS - The actual program is much complex than the above example. So transparently propagating the configuration chnages to User Defined classes is not possible.

You can watch the changes to the file using e.g. pyinotify (Linux) or watchdog (cross-platform). Once the change is detected, you can update your data structures.
Updating username (or other "derived" information stored elsewhere) is better done by storing (and using) not it but some invariants that never change (e.g. UID), getting the "derived" information on demand with relevant API and using it for display purposes only.

Related

How do you pythonically reuse a complex data structure without copying or re-reading a file?

I'm writing a Discord bot that, among many other things, must engage in question-answer dialogues with users. There may be multiple of the same kind of dialogue ("script") running at once asynchronously.
On program startup, a YAML file is read which contains scripts (not in the programming sense), which in turn contain lists of questions, each of which contain the question text. Both scripts and questions contain accessory attributes such as regex formulas, rejection responses, and display names.
Currently, ChatBotManager contains a list of Script objects which in turn contain Question objects, all of which can error-check the relevant portion of the YAML they are passed on initialization. Since the YAML is user-editable, this initialization is non-trivial, and should throw critical-exceptions when passed bad data.
After all this initialization, I have a very nice hierarchical structure with "has a" rather than "is a" relationships, and each level of data can contain methods relevant to it.
The question is this: which of the below ways of recording user responses is most pythonic, expandable, and readable?
Record user responses in a separate, partially-mirrored datastructure and refer to the Script and contained Question objects as read-only
Record user responses in Question objects themselves, then use some way to ensure that data does not contaminate parallel, identical chatbots. Both kinds of objects can have methods relevant to only their function, and other functions need little knowledge of the contents.
Note: When finally processing the responses, I need access to accessory attributes of each question.
I'm not asking this because I am stumped and can't progress: the first option is clearly within my knowledge. I want to know best practice. The second option embraces decoupling and clearly delineated responsibilities, but I don't know how to implement it properly. Perhaps my Googling for a solution lacked the right terminology?
I don't want to re-read the YAML file on every object's creation, nor run the validation code each time. Reading and validation need only happen on bot startup. I'd prefer to keep validation methods in the object that will store the validated data.
One idea I had is to deep-copy a Script object on each chatbot creation, then discard it on completion. I've read this is an expensive operation, and while my program isn't low on resources, it seems bad practice.
Should each Question be able to generate a Response object that contains the response and the data about the question that is needed later? These could then be compiled in a ScriptResponse object generated by the Script which also contains information about the script.

Shared Memory for Python Classes

So as I was learning about classes in Python I was taught that the class level attributes are shared among all instances of a given class. Something I don't think I've seen any other language before.
As a result I could have multiple instances of say a DB class pulling data DB data and dumping it into a class level attribute. Then any instance that needed any of that data would have access to it without having to go to a cache or saved file to
get it.
Currently I'm debugging an analytics class that grabs DB data via an inefficient means - one I'm currently trying to make much faster. Right now it takes several minutes for the DB data to load. And the format I've chosen for the data with ndarrays and such doesn't want to save to a file via numpy.save (I don't remember the error right now). Every time I make a little tweak the data is lost and I have to wait several minutes for it to reload.
So the thought occurred to me that I could create a simple class for holding that data. A class I wouldn't need to alter that could run under a separate iPython console (I'm using Anaconda, Python 2.7, and Spyder). That way I could link the analytics class to the shared data class in the init of the analytics class. Something like this:
def __init__(self):
self.__shared_data = SharedData()
self.__analytics_data_1 = self.__shared_data['analytics_data_1']
The idea would be that I would then write to self.__analytics_data_1 inside the analytics class methods. It would be automatically get updated to the shard data class. I would have an iPython console open to do nothing more than hold an instance of the shared data class during debug. That way when I have to reload and reinstantiate the analytics class it just gets whatever data I've already captured. Obviously, if there is an issue with the data itself then it will need to be removed manually.
Obviously, I wouldn't want to use the same shared data class for every tool I built. But that's simple to get around. The reason I mention all of this is that I couldn't find a recipe for something like this online. There seem to be plenty of recipes out there so I'm thinking the lack of a recipe might be a sign this is a bad idea.
I have attempted to achieve something similar via Memcache on a PHP project. However, in that project I had a lot of reads and write and it appeared that the code was causing some form of write collision. So data wasn't getting updated. (And Memcache doesn't guarantee the data will even be there.) Preventing this write collision meant a lot of extra code, extra processing time, and ultimately the code got to be too slow to be useful. I thought I might attempt it again with this shared memory in Python as well as using the shared memory for debugging purposes.
Thoughts? Warnings?
If you wanna shared data across your class instances, you should do:
class SharedDataClass(object):
shared_data = SharedData()
These SharedData instance will be shared between your class instances unless you override in in the constructor.
Careful, GIL lock may affect performance!
Sharing data "between consoles" or processes is different story. They're separate processes, and, of course, class attributes are not shared. In this case, you need IPC (it could be filesystem, database, process with open socket to connect to, etc.)

Updating files with a Perforce trigger before submit

I understand that this question has, in essence, already been asked, but that question did not have an unequivocal answer, so please bear with me.
Background: In my company, we use Perforce submission numbers as part of our versioning. Regardless of whether this is a correct method or not, that is how things are. Currently, many developers do separate submissions for code and documentation: first the code and then the documentation to update the client-facing docs with what the new version numbers should be. I would like to streamline this process.
My thoughts are as follows: create a Perforce trigger (which runs on the server side) which scans the submitted documentation files (such as .txt) for a unique term (such as #####PERFORCE##CHANGELIST##NUMBER###ROFL###LOL###WHATEVER#####) and then replaces it with the value of what the change list would be when submitted. I already know how to determine this value. What I cannot figure out, is how or where to update the files.
I have already determined that using the change-content trigger (whether possible or not), which
"fire[s] after changelist creation and file transfer, but prior to committing the submit to the database",
is the way to go. At this point the files need to exist somewhere on the server. How do I determine the (temporary?) location of these files from within, say, a Python script so that I can update or sed to replace the placeholder value with the intended value? The online documentation for Perforce which I have found so far have not been very explicit on whether this is possible or how the mechanics of a submission at this stage would work.
EDIT
Basically what I am looking for is RCS-like functionality, but without the unsightly special character sequences which accompany it. After more digging, what I am asking is the same as this question. However I believe that this must be possible, because the trigger is running on the server side and the files had already been transferred to the server. They must therefore be accessible by the script.
EXAMPLE
Consider the following snippet from a release notes document:
[#####PERFORCE##CHANGELIST##NUMBER###ROFL###LOL###WHATEVER#####] Added a cool new feature. Early retirement is in sight.
[52702] Fixed a really annoying bug. Many lives saved.
[52686] Fixed an annoying bug.
This is what the user submits. I then want the trigger to intercept this file during the submission process (as mentioned, at the change-content stage) and alter it so that what is eventually stored within Perforce looks like this:
[52738] Added a cool new feature. Early retirement is in sight.
[52702] Fixed a really annoying bug. Many lives saved.
[52686] Fixed an annoying bug.
Where 52738 is the final change list number of what the user submitted. (As mentioned, I can already determine this number, so please do dwell on this point.) I.e., what the user sees on the Perforce client console is.
Changelist 52733 renamed 52738.
Submitted change 52738.
Are you trying to replace the content of pending changelist files that were edited on a different client workspace (and different user)?
What type of information are you trying to replace in the documentation files? For example,
is it a date, username like with RCS keyword expansion? http://www.perforce.com/perforce/doc.current/manuals/p4guide/appendix.filetypes.html#DB5-18921
I want to get better clarification on what you are trying to accomplish in case there is another way to do what you want.
Depending on what you are trying to do, you may want to consider shelving ( http://www.perforce.com/perforce/doc.current/manuals/p4guide/chapter.files.html#d0e5537 )
Also, there is an existing Perforce enhancement request I can add your information to,
regarding client side triggers to modify files on the client side prior to submit. If it becomes implemented, you will be notified by email.
99w,
I have also added you to an existing enhancement request for Customizable RCS keywords, along
with the example you provided.
Short of using a post-command trigger to edit the archive content directly and then update the checksum in the database, there is currently not a way to update the file content with the custom-edited final changelist number.
One of the things I learned very early on in programming was to keep out of interrupt level as much as possible, and especially don't do stuff in interrupt that requires resources that can hang the system. I totally get that you want to resolve the internal labeling in sequence, but a better way to do it may be to just set up the edit during the trigger so that a post trigger tool can perform the file modification.
Correct me if I'm looking at this wrong, but there seems a bit of irony, or perhaps recursion, if you are trying to make a file change during the course of submitting a file change. It might be better to have a second change list that is reserved for the log. You always know where that file is, in your local file space. That said, ktext files and $ fields may be able to help.

How to properly manage application configurations

What is the most universal and best application configurations management method? I want to have these properties in order to have "good configuration management":
A list of all available properties and their default values in one
place.
A list of properties which can be changed by an app user, also in one
place.
When I retrieve a specific property, it's value is returned from the
2nd list (user changeable configs) or if it's not there, from the
first list.
So far, what I did was hard coding the 1st list as an object (more specific as a dict), wrote .conf file used by ConfigParser to make an app user to easily change some of the properties (2nd list), and wrote a public method on the config object to retrieve a property by it's name or if it's not there, raise an exception. In the end, one object was responsible for managing all the stuff (parsing file, raising exception, overriding properties etc.) But I was wondering, if there's a built-in library which does more or less the same thing, or even a better way to manage configuration, which takes into account all the KISS, DRY and other principles (I'm not always successful to do that with this method)?
Thanks in advance.
Create a default settings module which contains your desired default settings. Create a second module intended to be used by the the user with a from default_settings import * statement at the top, and instructing the user to write any replacements into this module instead.
Python is rather expressive, so in most cases, if you can expect the user to understand it on any level, you can use a Python module itself as the configuration file.

User location information mapping with lastfm music track history in python

I have two separate python scripts. One is for getting user location
information (which I get from web based geofeed provider.User Gsm is registerd
with that services).Another is for retrieving lastfm user track history.I have already
able to get user location data and user music track information.
Goal is to map those two script in such a way that I could be
able to make relation from those information" Users in certain location are listening certain music in certain time".
Can anybody have nice idea to get this out?Thanks in advance!
Write a third script that import both of the other as modules, and make sure each module's functionality is embodied in a function (as is Python's best practice), not just "floating" as module-level code -- a module's top-level statements should usually be limited to import, from, def, class, and simple assignments of names to constants (possibly initial values for global variables), with all actual logic within functions and classes.
So in your third script, after importing the other two as modules, you have a main function that calls the operating functions of the others to get location and track info, calls a function from standard module datetime (e.g. datetime.datetime.now()) -- or possibly time -- to get the current time, and finally formats all this information in the way you desire and writes it somewhere (where and how do you want to "publish" this info?).
At the end of the script you do
if __name__ == '__main__':
main()
which is the usual Python idiom to ensure the module's main function executes when the file's being used as the main script rather than just being imported from elsewhere.

Categories