This is something that I think would be very useful. Basically, I'd like there to be a way to edit Python source programmatically without requiring human intervention. There are a couple of things I would like to do with this:
Edit the configuration of Python apps that use source modules for configuration.
Set up a "template" so that I can customize a Python source file on the fly. This way, I can set up a "project" system on an open source app I'm working on and allow certain files to be customized.
I could probably write something that can do this myself, but I can see that opening up a lot of "devil's in the details" type issues. Are there any ways to do this currently, or am I just going to have to bite the bullet and implement it myself?
Python's standard library provides pretty good facilities for working with Python source; note the tokenize and parser modules.
Most of these kinds of things can be determined programatically in Python, using modules like sys, os, and the special _file_ identifier which tells you where you are in the filesystem path.
It's important to keep in mind that when a module is first imported it will execute everything in the file-scope, which is important for developing system-dependent behaviors. For example, the os module basically determines what operating system you're using on import and then adjusts its implementation accordingly (by importing another module corresponding to Linux, OSX, Windows, etc.).
There's a lot of power in this feature and something along these lines is probably what you're looking for. :)
[Edit] I've also used socket.gethostname() in some rare, hackish instances. ;)
I had the same issue and I simply opened the file and did some replace: then reload the file in the Python interpreter. This works fine and is easy to do.
Otherwise AFAIK you have to use some conf objects.
Related
Why does Qt seem to need "resource files"? I mean the things you compile with pyside2-rcc resources.qrc -o resources.py, for example.
The documentation seems to say that it has something to do with accessing paths, abstracted in such a way that Python doesn't trip over the differences in a cross-platform application. But IME, Python does fine with Linux/Unix-style pathnames on Windows, especially if you use pathlib or os.path.join() or something.
In the days I used to use DOS, I found that command.com had problems with /, but you could binary-patch command.com to rename cd to something else, and use a cd.exe that worked with /'s fine - it was command.com that had the problem, not the underlying OS.
https://doc.qt.io/qtforpython/overviews/resources.html seems to say it has something to do with not losing files, which seems kinda odd.
So why does Qt need resource files, and can I safely skip them?
TL; DR; We are not required to use qresource, it is only an option.
It is not that Qt requires qresources on a mandatory basis, but rather that it is an option that in the world of Qt provides us. That is, using them or not in general does not bring a benefit or harm.
From a Qt/C++ point of view, resources allow us to embed resources within binaries. And this abstraction possibly avoids having problems with the directory system paths. And since PySide2 is a Qt wrapper it also tries to expose that tool.
And as you point out, there are several python libraries that already handle the paths in a generic way avoiding the known routing problems between OS so you could use it.
Is there a way to add functions I create to the Python standard library on my local machine?
I come from the matlab world where things aren't really efficient and fast but there are looooads of functions at my fingertips without having to import their files. My problem is that, if I make a function in Python and want to use it, then i will need to also remember the module its in. My memory is shite. I understand that Python is structured that way for efficiency but if I'm adding only a handful of functions to the standard library that I consider very important, I'd guess that the impact to the performance is practically negligible.
Python has a namespace called __builtins__ in which you can stick stuff that you want available all the time. You probably shouldn't, but you can. Be careful not to clobber anything. Python won't stop you from using the same name as a built-in function, and if you do that, it'll probably break a lot of things.
# define function to always be available
def fart():
print("poot!")
__builtins__.fart = fart
# make re module always available without import
import re
__builtins__.re = re
Now the question is how to get Python to run that code for you each time you start up the interpreter. The answer is usercustomize.py. Follow these instructions to find out where the correct directory is on your machine, then put a new file called usercustomize.py in that directory that defines all the stuff you want to have in __builtins__.
There's also an environment variable, PYTHONSTARTUP, that you can set to have a Python script run whenever you start the interpreter in interactive mode (i.e. to a command prompt). I can see the benefit of e.g. having your favorite modules available when exploring in the REPL. More details here.
It sounds like you want to create your own packages & modules with tools you plan on using in the future on other projects. If that is the case, you want to look into the packaging your own project documentation:
https://packaging.python.org/tutorials/packaging-projects/
You may also find this useful:
How to install a Python package system-wide on Linux?
How to make my Python module available system wide on Linux?
How can I create a simple system wide python library?
My project has to be extensible, i have a lot of scripts with the same interface that lookup things online. Before i was using __import__ but that does not let me put my 'plugins' on a dedicated directory:
root/
main.py
plugins/
[...]
So my question is: Is there a way to individually import modules from that subdirectory? I'm guessing importlib, but i'm so lost in how the python module loading process works... What i want to do is something like this:
for pluginname in plugins:
plugin = somekindofimport("plugins/{name}".format(name=pluginname))
plugin.unififedinterface()
Also, as a side question, the way am i trying to achieve extensibility is a good way?
I'm on python3.3
Stop thinking in terms of pathnames and start thinking in terms of packages. Read Packages in the tutorial, and if you want more detail see The import system.
But the basic idea is this:
Create a file name plugins/__init__.py. It can be empty; that's enough to turn plugins into a package. Which means you can import modules from that package with:
import plugins.plugin
So, how do you do this dynamically? That's what importlib is for. (You can also use __import__ here, but it's less flexible, and less readable in non-trivial cases, so unless you need pre-3.3 compatibility, don't.)
plugin = importlib.import_module('plugins.{name}'.format(name=pluginname))
It would probably be cleaner to import plugins to get the package, and then use relative imports from within that package, as shown in the examples in the import_module docs.
This also means Python takes care of the .pyc creation and caching, etc.
And it means that you can later expand plugins to be a "namespace package", which can be split across multiple directories like /usr/share/myapp/plugins for stock plugins, /etc/myapp/plugins for site plugins and ~/myapp/plugins for user-specific plugins.
If you really, really want to import from a directory that isn't a package, you can create a module loader and use it, but that's a whole lot of work for no actual benefit. (It's actually not that hard in 3.3 (SourceLoader and friends will do most of the work for you), but you will find almost no examples out there to guide you; instead, you'll find examples of the 2.6-3.2 way, or the 2.0-2.5 way, both of which are hard.) Plus, it means that if someone creates a plugin named, say, gzip, you can end up blocking the stdlib gzip module with the plugin. (That's especially fun if the gzip plugin tries to use the gzip stdlib module, as it likely will…) If the plugin ends up being named plugins.gzip, there's no problem.
Also, as a side question, the way am i trying to achieve extensibility is a good way?
As long as you only want to support 3.3+, yes, I think this is a great solution.
Before 3.3, using a package for plugins was a lot more problematic. People have come up with a variety of different plugin systems—in one case going so far as to dynamically create module objects and execfile into them. If you need to deal with that, I would suggest looking at existing Python apps with plugins (e.g., MusicBrainz Picard) to get different ideas.
Here's the scenario:
I am writing a program that will run inside a Telit HE910 GSM module, which has an on-board python interpreter.
I would like my program to be able to read and store some parameters inside a configuration file onboard the GSM module.
The file may occasionally be transferred into a separate machine for viewing and changing, so it should be both machine-readable and writeable on multiple platforms.
So far, there are multiple solutions for this issue. Here's the kicker though:
The GSM module does not have any python modules for parsing / writing configuration files (so I can't simply use import yaml, import json, import configparser, or even import csv)
The GSM module does not allow the creation of subdirectories. From my limited understanding, this prevents me from simply dumping the contents of say, the PyYAML python module into the GSM module and calling it from my program.
I found a similar question here, but I don't even know where in the GSM module's filesystem I am. import os doesn't seem to work, which is strange (contrary to documentation).
I know I can use a Python file to store some read-only configurations, but I also want to be able to write to the config file (redesigning the system to avoid this is really undesirable).
I think my best bet so far seems to be to write a simple csv parser / writer myself, unless someone has a better idea (or know how to utilise Python modules without any subdirectories).
PS: The documentation below has a list of supported modules. None of the config-related modules seem to be available however.
GSM Module homepage: http://www.telit.com/en/products/umts-hsdpa.php?p_ac=show&p=108
Python interpreter manual : http://www.telit.com/module/infopool/download.php?id=4378
EDIT: I should have mentioned, the configuration file needs to be readable / writeable from a c# .NET application, not another python interpreter on the desktop.
You have _ast (§5.2.36), so it should be possible to reimplement ast.literal_eval(). At that point reading and writing becomes mostly trivial.
I wouldn't recommend it in normal Python usage, but a possible option would be the marshal module mentioned in 5.2.34 of the manual you posted, and description/limitations/warnings here: http://docs.python.org/library/marshal.html
Why don't you just open a txt file and store all info as a text file? I read the Easy Script Manual for your module, there is a posix module which supports methods like open, close, unlink.
My Delphi program uses Python4Delphi, and the Python script uses some standard libs. When deploying my program I don't want an entire Python installation, and it must work with python27.dll. What are the minimal set of necessary files? The document in Python4Delphi is dated and it's not clear to me...
Thanks for your help.
When I did this, I made the list myself, of what I needed for my embedded python application to work.
I remember this worked with python15.dll:
PythonXX.dll should work, without any other external files other than the Visual C++ Runtime DLLs, which require a side-by-side manifest (see the p4d wiki page) to work.
If you want to IMPORT something, then you need to ship it and anything it depends on. That means, either you pick part of the python standard libraries you want, or you pick all of it. There is no way you need all of Python's standard libraries. But I wouldn't want to live without OS and a a few other key ones. BUt the decision is yours.