I'm trying to understand the backend of the request and sys, and os modules.
I've seen or heard somewhere that python modules are pre-written python codes made to be called later in other scripts with the import command.
I was just curious as to where the module source codes are located so I can read them to better understand some things in python.
I think you've got the right mentality. I've become such a better developer just by taking the time to step through the source code.
You can find all of the Python built-in modules here.
You may notice that some of the built-in modules like os and sys are written in C. I’d recommend checking this question out for a better understanding.
In regards to requests, it's a third-party library built around urllib.
Related
I am running a python script that calls local modules (with import), and since few days it became so slow.
I could not find the reason why it is so, neither on other Github posts or Google, that's why I am posting this.
Providing code won't be of much help, but here is the import that poses a problem:
import latplan
where latplan is this library
But again, this import did not pose any problem at all before...
import statements can be very slow in Python because they are allowed to execute arbitrary code as side effects; they don't just scoot for class and def keywords.
Now, as with any performance issue, it's hard to guess what exactly is taking so long without profiling. Luckily there's a builtin, specialised profiler for import time:
python -X importtime -c "import latplant"
I recommend using tuna to visualise the reports.
My project has to be extensible, i have a lot of scripts with the same interface that lookup things online. Before i was using __import__ but that does not let me put my 'plugins' on a dedicated directory:
root/
main.py
plugins/
[...]
So my question is: Is there a way to individually import modules from that subdirectory? I'm guessing importlib, but i'm so lost in how the python module loading process works... What i want to do is something like this:
for pluginname in plugins:
plugin = somekindofimport("plugins/{name}".format(name=pluginname))
plugin.unififedinterface()
Also, as a side question, the way am i trying to achieve extensibility is a good way?
I'm on python3.3
Stop thinking in terms of pathnames and start thinking in terms of packages. Read Packages in the tutorial, and if you want more detail see The import system.
But the basic idea is this:
Create a file name plugins/__init__.py. It can be empty; that's enough to turn plugins into a package. Which means you can import modules from that package with:
import plugins.plugin
So, how do you do this dynamically? That's what importlib is for. (You can also use __import__ here, but it's less flexible, and less readable in non-trivial cases, so unless you need pre-3.3 compatibility, don't.)
plugin = importlib.import_module('plugins.{name}'.format(name=pluginname))
It would probably be cleaner to import plugins to get the package, and then use relative imports from within that package, as shown in the examples in the import_module docs.
This also means Python takes care of the .pyc creation and caching, etc.
And it means that you can later expand plugins to be a "namespace package", which can be split across multiple directories like /usr/share/myapp/plugins for stock plugins, /etc/myapp/plugins for site plugins and ~/myapp/plugins for user-specific plugins.
If you really, really want to import from a directory that isn't a package, you can create a module loader and use it, but that's a whole lot of work for no actual benefit. (It's actually not that hard in 3.3 (SourceLoader and friends will do most of the work for you), but you will find almost no examples out there to guide you; instead, you'll find examples of the 2.6-3.2 way, or the 2.0-2.5 way, both of which are hard.) Plus, it means that if someone creates a plugin named, say, gzip, you can end up blocking the stdlib gzip module with the plugin. (That's especially fun if the gzip plugin tries to use the gzip stdlib module, as it likely will…) If the plugin ends up being named plugins.gzip, there's no problem.
Also, as a side question, the way am i trying to achieve extensibility is a good way?
As long as you only want to support 3.3+, yes, I think this is a great solution.
Before 3.3, using a package for plugins was a lot more problematic. People have come up with a variety of different plugin systems—in one case going so far as to dynamically create module objects and execfile into them. If you need to deal with that, I would suggest looking at existing Python apps with plugins (e.g., MusicBrainz Picard) to get different ideas.
I am confused about the two ways to import modules in IronPython.
On the one hand, the tutorial documentation that comes with IronPython 2.7.4 states that you can do it using the regular import syntax:
import System
from System import Xml
This works as I would expect.
On the other hand, many resources on the internet state that the way to import modules is by using the clr module like so:
import clr
clr.AddReference("System.Xml")
What is the difference between the two methods?
While I was researching this question I stumbled across what I believe to be the answer (this is from trial and error alone so if I'm wrong I'd be happy to be corrected!)
The import statement in Python is more analogous to a using <namespace> statement in C#. you still need to load the relevant .dll assembly. C# does this at compile-time by using references; IronPython by default includes standard CLR references, which is why it is immediately possible to import System.Xml
However, if you want to load a .dll that is not included by default in IronPython, you must use clr.AddReference("myAssembly.dll") and then use the import statement to import the module.
For example:
import clr
clr.AddReferenceToFileAndPath(r"..\lib\umbraco.dll")
import umbraco
The umbraco module is now accessible to IronPython
N.B. The Visual Studio plugin "Python Tools" allows you to add references to a Python project, but the above steps are still necessary to use a reference.
Visual Studio projects support adding references to projects and extensions. Typically they indicate dependencies between projects and are used to provide IntelliSense at design time or linking at compile time. Python Tools for Visual Studio also uses references in a similar fashion, but due to the dynamic nature of Python they are primarly used at design time to provide improved IntelliSense.
Link
I got this script for my program. The program uses python for scripting. Anyway, the script has this line
from common import Struct
Is this part of Python's standard library? because my python seems to be missing it. Maybe it is deprecated?
The script didn't include anything else but that one python file, so i guessed it's not a module made by the script creator.
I would suggest you check for a common.py file and add it to your PYTHONPATH.
if you are using some sort of unix/bsd you could try to do a "locate common.py" and check if it has a Struct somewhere.
Hope this helps
In your script, just add after the line you quoted:
import sys
print sys.modules['common']
And it will show you the path where the module common was imported from.
Based on this you could easily see if it is a Python standard library or not.
Alternatively if you go to any page like https://docs.python.org/3/library/index.html you can see the list of modules included with Python.
In both cases I think you can come to the conclusion that there is no common module shipped with Python, and I would find it to be a very bad name if it existed like that.
This is something that I think would be very useful. Basically, I'd like there to be a way to edit Python source programmatically without requiring human intervention. There are a couple of things I would like to do with this:
Edit the configuration of Python apps that use source modules for configuration.
Set up a "template" so that I can customize a Python source file on the fly. This way, I can set up a "project" system on an open source app I'm working on and allow certain files to be customized.
I could probably write something that can do this myself, but I can see that opening up a lot of "devil's in the details" type issues. Are there any ways to do this currently, or am I just going to have to bite the bullet and implement it myself?
Python's standard library provides pretty good facilities for working with Python source; note the tokenize and parser modules.
Most of these kinds of things can be determined programatically in Python, using modules like sys, os, and the special _file_ identifier which tells you where you are in the filesystem path.
It's important to keep in mind that when a module is first imported it will execute everything in the file-scope, which is important for developing system-dependent behaviors. For example, the os module basically determines what operating system you're using on import and then adjusts its implementation accordingly (by importing another module corresponding to Linux, OSX, Windows, etc.).
There's a lot of power in this feature and something along these lines is probably what you're looking for. :)
[Edit] I've also used socket.gethostname() in some rare, hackish instances. ;)
I had the same issue and I simply opened the file and did some replace: then reload the file in the Python interpreter. This works fine and is easy to do.
Otherwise AFAIK you have to use some conf objects.