So I wanted to check out some implementations of standard libraries. I started with the os library with the code being here on github.
I took one method for example os.listdir() and I have absolutely no idea how it is implemented even after looking at the code ( pardon this noob ). I have following questions:
os.__all__ do not list this method but I think it is definitely a method as print(type(os.listdir)) listed <class 'builtin_function_or_method'> and I searched on google to find all the builtin functions which I found on this doc page and this is not one of them.
There is not such exclusive function named listdir defined in the module. In the code, from my limited understanding, the function is taken from globals() and put into a support_fd set. How this method is being called I do not understand.
I think the main problem I have is how that module is designed and I was not able to find any resources online to explain in simpler terms hence I am asking here for pointers.
EDIT: For those who are asking, I tried the following code in onlinegdb
import os
if "listdir" in os.__all__:
print("Yes")
print(os.listdir())
The result is only main.py, it should also print Yes, maybe the platform onlinegdb is the problem but it clearly shows the output of listdir as main.py.
After having discussion in the comments I see now that this is more of a online python version problem and not an issue with python or the module itself.
I know that's possible from a snippet of code in python to get its AST or its code object via compile().
I was wondering if it is possible to do the opposite: I have the code object(extracted by a .pyc file) and I was looking for the its AST.
No, unfortunately it is not possible to do by conventional methods unless the file that the code object created is still available in the path. There are tools like uncompyle6, which might help though.
I am trying to figure out how to list out all the standard library functions/methods (still learning the difference, I'm a noob). So I get how to import a module and use dir() and help(). These have been a great help and when I try to rewrite a program I learned to write and I get lost I try to use the dir() and help() to spark my memory.
However, for example, I was re-looking through some code and I had .replace() and I was scouring through my imported modules to find it and then found out it's in the standard library and a sub-part of the str() function. Is there something I can type into dir() that will spit out str()?
How can I print out all top level built in function/methods like I would when I do dir(re)?
I've tried re-wording this question in search engines several ways and I can't find anything and the results get muddied with "python list _____".
Thanks so much!
Is there something I can type into dir() that will spit out str()?
dir() on a value of that type.
dir('hello') # string methods
dir(5) # integer methods
dir([]) # list methods
Ah, I never tried typing just "dir()" that listed I could put in dir(__builtins__) and that is what I was looking for. I'll leave it here in case someone else is a noob too and knows where they can start. Just start with dir() and go down the rabbit hole!
Given a method name, how to determine which module(s) in the standard library contain this method?
E.g. If I am told about a method called strip(), but told nothing about how it works or that it is part of str, how would I go and find out which module it belongs to? I obliviously mean using Python itself to find out, not Googling "Python strip" :)
The trouble is, strip is not defined in any module. It is not a part of the standard library at all, but a method on str, which in turn is a built in class. So there isn't really any way of iterating through modules to find it.
You could use modulefinder to determin all the loaded modules then loop though each one and to get a list of methods using inspect.getmembers looping though those to find what you are looking for. I don't thing there is a built-in way to do this.
https://python.readthedocs.org/en/v2.7.2/library/modulefinder.html
https://docs.python.org/2/library/inspect.html
How do you figure out what methods are available within a class?
Example :
I am trying to learn about urllib.request. I found urlopen() in the docs :
http://docs.python.org/3.0/library/urllib.request.html
So I have :
response = urllib.request.urlopen(url)
What does this return? I know it will probably be an object, but what are this objects methods? Eventually I get to the examples (stuck at the bottom of the page far from where the idea was introduced) and discover read(), but I had to look outside the docs to find .decode(), which finally lets you do what the whole purpose of this library is.
Im having this type of problem with much of the docs.python.org pages. Is there a better documentation somewhere else, or am I going about learning this all the wrong way?
The real problem here is that you're using a very old version of the documentation. I have no idea how you found it, but it should be pretty clear from the URL http://docs.python.org/3.0/library/urllib.request.html, and the header on the top of that page ("Python v3.0.1 documentation"), and so on that you're not looking at the documentation for your version.*
If you were looking at the 3.3 documentation, you would have seen this:
For http and https urls, this function returns a http.client.HTTPResponse object which has the following HTTPResponse Objects methods.
The first link takes you to the reference for the exact class, the second to the reference for the abstract type. Which gives you exactly what you were looking for—the read method, and everything else.
In older versions of Python, the term "file-like object" was thrown around loosely. This was always a vague term (sometimes it means "an iterable of lines", "has a read() and/or write() as appropriate", "has a fileno()", …), and became much more so in Python 3 (because you have to distinguish a binary file from a text file). So, over the years, they've phased this out in favor of more specific documentation. But if you're looking at the very earliest Python 3 documentation, you're not getting the benefit.
* Note that if you visit any recent-ish version of the docs, like the 3.3 linked above, there's a pulldown menu in the header that lets you switch to a different version if you've found the wrong one. And, the default will always be 2.7 or the latest stable 3.x, and those are also the most common search results, and the easiest things to link to, so you will usually be on one of those unless you're really trying to make things hard for yourself. If you do find yourself on ancient docs like 3.0 or 2.4 or something, you can often just edit the URL to 3.3 or 2.7, or just 3 or 2; if not, a quick search should work.
The key phrase in the documentation of urlopen is:
This function returns a file-like object with two additional methods from the urllib.response module
The fact that this is a file-like object indicates that all the normal file operation functions (read, close) apply to the returned object also.
The document you linked to says this:
This function returns a file-like object with two additional methods
from the urllib.response module
geturl() — return the URL of the resource retrieved, commonly used to determine if a redirect was followed
info() — return the meta-information of the page, such as headers, in the form of an http.client.HTTPMessage instance (see Quick Reference to HTTP Headers)
Emphasis mine. A "file-like object" is defined as follows:
file object
An object exposing a file-oriented API (with methods such as read() or write()) to an underlying resource. Depending on the way it was created, a file object can mediate access to a real on-disk file or to another type of storage or communication device (for example standard input/output, in-memory buffers, sockets, pipes, etc.). File objects are also called file-like objects or streams.
As #abarnert points out, you're probably just better off using the new docs, which explicitly link to the methods of the returned object.
If you aren't using Ipython already you should be. Then you can just type urllib. and press tab to see all the available options and do the same for whatever it returns. If you type a question mark after the function it will bring up the documentation, two question marks often brings up the source.
For what its worth though I have always found the online documentation to be very helpful. It does say that urllib.request.urlopen is going to return a file like object early in the explanation