We Keep Coding

How to access a docstring from a separate script?

Building a GUI for users to select Python scripts they want to run. Each script has its own docstring explaining inputs and outputs for the script. I want to display that information in the UI once they've highlighted the script, but not selected to run it, and I can't seem to get access to the docstrings from the base program.
ex.
test.py
"""this is a docstring"""
print('hello world')
program.py
index is test.py for this example, but is normally not known because it's whatever the user has selected in the GUI.
# index is test.py
def on_selected(self, index):
script_path = self.tree_view_model.filePath(index)
fparse = ast.parse(''.join(open(script_path)))
self.textBrowser_description.setPlainText(ast.get_docstring(fparse))

Let's the docstring you want to access belongs to the file, file.py.
You can get the docstring by doing the following:
import file
print(file.__doc__)
If you want to get the docstring before you import it then the you could read the file and extract the docstring. Here is an example:
import re
def get_docstring(file)
with open(file, "r") as f:
content = f.read() # read file
quote = content[0] # get type of quote
pattern = re.compile(rf"^{quote}{quote}{quote}[^{quote}]*{quote}{quote}{quote}") # create docstring pattern
return re.findall(pattern, content)[0][3:-3] # return docstring without quotes
print(get_docstring("file.py"))
Note: For this regex to work the docstring will need to be at the very top.

Here's how to get it via importlib. Most of the logic has been put in a function. Note that using importlib does import the script (which causes all its top-level statements to be executed), but the module itself is discarded when the function returns.
If this was the script docstring_test.py in the current directory that I wanted to get the docstring from:
""" this is a multiline
docstring.
"""
print('hello world')
Here's how to do it:
import importlib.util
def get_docstring(script_name, script_path):
spec = importlib.util.spec_from_file_location(script_name, script_path)
foo = importlib.util.module_from_spec(spec)
spec.loader.exec_module(foo)
return foo.__doc__
if __name__ == '__main__':
print(get_docstring('docstring_test', "./docstring_test.py"))
Output:
hello world
this is a multiline
docstring.
Update:
Here's how to do it by letting the ast module in the standard library do the parsing which avoids both importing/executing the script as well as trying to parse it yourself with a regex.
This looks more-or-less equivalent to what's in your question, so it's unclear why what you have isn't working for you.
import ast
def get_docstring(script_path):
with open(script_path, 'r') as file:
tree = ast.parse(file.read())
return ast.get_docstring(tree, clean=False)
if __name__ == '__main__':
print(repr(get_docstring('./docstring_test.py')))
Output:
' this is a multiline\n docstring.\n'

Creating a class object to be used globally in Python

I wrote the module below that will standardize how my logfiles are written as well as easily changing whether events get printed/written to the logfile or not.
FILE: Logging.py
================
import os
import datetime
import io
class Logfile():
def __init__(self,name):
self.logFile = os.getcwd() + r'\.Log\\' + name + '_' + str(datetime.date.today().year) + ('00' + str(datetime.date.today().month))[-2:] + '.log'
self.printLog = False
self.debug = False
# Setup logFile and consolidated Folder
if not os.path.exists(os.path.dirname(self.logFile)):
os.mkdir(os.path.dirname(self.logFile))
#Check if logfile exists.
if not os.path.exists(self.logFile):
with open(self.logFile, 'w') as l:
pass
# Write LogFile Entry
def logEvent(self, eventText, debugOnly): # Function to add an event to the logfile
# If this is marked as debugging only AND debugging is off
if debugOnly == True and self.debug == False:
return
if self.printLog == True:
print(datetime.datetime.strftime(datetime.datetime.now(), '%m/%d/%Y, %I:%M:%S %p, ') + str(eventText))
with open(self.logFile, 'a') as l:
l.seek(0)
l.write(datetime.datetime.strftime(datetime.datetime.now(), '%m/%d/%Y, %I:%M:%S %p, ') + str(eventText) + '\n')
return
This is very handy but, I am having trouble understanding how to make this available to all of my classes. For example, If i import the following module, I am not sure how to use the logfile i created within my main script.
FILE: HelloWorld.py
===================
class HelloWorld():
def __init__(self):
log.logEvent('You have created a HelloWorld Object!', False)
Main Script Here:
import Logging
from HelloWorld import HelloWorld
log = logging.Logfile
hw = HelloWorld()
^^ Will fail because it does not know log is a thing. What is the proper way to handle these sort of situations?

I believe you're trying to do something like this. (and as a side note, you may want to look into using pythons default logging module)
FILE: HelloWorld.py
===================
# import LogFile
from .Logging import LogFile
# create new LogFile instance
log = LogFile(name='log name')
class HelloWorld():
def __init__(self):
# call logEvent method on your LogFile instance
log.logEvent('You have created a HelloWorld Object!', False)
FILE: Main.py
===================
# import HelloWorld
from .HellowWorld import HellowWorld
# create new HellowWorld instance
hw = HellowWorld()
Also, to create a module you will need to add an __init__.py file in that given directory.

This problem is easily solved by using the built-in "Logging" module. In an answer to the broader "how to use a thing(log) within all of my modules" question, I assume the answer to this can be found by reading through the code in the logging module and mimicking that.

python readline module folder name completion

I am trying to implement a folder name completion for one of my programs and i am using the readline module for it. I have written a Completer class (file autocompletion.py):
import os.path
import readline
class Completer(object):
def _match_path(self, text):
text = readline.get_line_buffer()
return DirectoryPathCompleter(text).completions()
def complete(self, text, state):
matches = self._match_path(text)
try:
return matches[state]
except IndexError:
return None
class DirectoryPathCompleter(object):
def __init__(self, text):
self.text = text
def completions(self):
path, rest = os.path.split(self.text)
if rest == '.' or rest == '..':
result = [rest + '/']
else:
result = [i + '/' for i in get_dirs_under(path) if i.startswith(rest)]
return result
def get_dirs_under(path):
if not path:
path = '.'
dirpath, dirnames, filenames = os.walk(path).next()
return dirnames
To try this out here is a little script try_readline.py
#!/usr/bin/env python
from autocompletion import Completer
import readline
readline.parse_and_bind('tab: complete')
readline.set_completer(Completer().complete)
raw_input('Test it! > ')
I can see there, that it works quite good as long as I do not try to complete dirnames that can be completed just partially. When I create for example a directory with the following structure and run the program via shell:
$ mkdir -p test/bla-foo test/bla-bar test/bla-baz test/something test/else
$ cp autocompletion.py try_readline.py test
$ cd test; tree
.
├── autocompletion.py
├── bla-bar
├── bla-baz
├── bla-foo
├── else
├── something
└── try_readline.py
5 directories, 2 files
$ python try_readline.py
and I try to complete bla- via TAB, I get as a result bla-bla-. I would like the completion to stick with bla- and show the alternatives that can be completed.
How can I achieve this?
EDIT:
Ok, I don't know exactly why this is happening. If I alter the try_readline.py script to look like this:
#!/usr/bin/env python
import readline
readline.parse_and_bind('tab: complete')
raw_input('Test it! > ')
So I am not using my own Completer and I run it on the same folder structure as shown above, I can see similar behaviour:
Test it! > bla-b<TAB>
Test it! > bla-bla-
Whereas I would expect:
Test it! > bla-b<TAB>
Test it! > bla-ba<TAB><TAB>
bla-bar/ bla-baz/
EDIT2
Ok I have another approach now, it took so long because the readline documentation is very sparse.. I still don't get exactly why I do have to do it this way but it works.
If I look at the delimiters readline is using, I get:
In [1]: readline.get_completer_delims()
Out[1]: ' \t\n`!##$^&*()=+[{]}\\|;:\'",<>?'
Especially the \\ part looked odd to me, so I did set new delimiters in the try_readline.py
#!/usr/bin/env python
import readline
from autocompletion import Completer
readline.parse_and_bind('tab: complete')
readline.set_completer_delims(' \t\n')
raw_input('Test it! > ')
I am now using the standard readline completion as I don't know exactly how to write a correct Completer... I would like to reuse the standard readline completion function and just filter out any non-directories but I don't know how to extract it from the module.

Python merge .py part-files into one .py file

I am doing browser automation using python + splinter.
my structure is like this
[root]
+--start.py
+--end.py
+--[module1]
| +--mod11area1.py
| +--mod12area2.py
| +--[module1_2]
| | +--mod121area1.py
| +--[module1_3]
| +--mod131area1.py
+--[module2]
+--mod21area1.py
start.py sets the initialization and opening of the browser
and the inner modules.py performs actions per module
this structure would then be merged into one script upon execute by appending the contents in this fasion:
start.py
mod11area1.py
mod12area2.py
mod121area1.py
mod131area1.py
mod21area1.py
end.py
My question is, is there a better way of doing this? I'm quite new to this and just usually create a single script. since my project keeps on expanding I had to employ several other guys to script with me. Hence I came up with this approach.

No, Python has no simple way to merge scripts into one .py file.
But you can fake it, albeit in a fairly limited way.
Heres an example of how you can define multiple modules (each with their own namespace), in a single file.
But has the following limitations.
No package support(although this could be made to work).
No support for modules depending on eachother(a module can't be imported unless its already defined).
Example - 2 modules, each containing a function:
# Fake multiple modules in a single file.
import sys
_globals_init = None # include ourself in namespace
_globals_init = set(globals().keys())
# ------------------------
# ---- begin
__name__ = "test_module_1"
__doc__ = "hello world"
def test2():
print(123)
sys.modules[__name__] = type(sys)(__name__, __doc__)
sys.modules[__name__].__dict__.update(globals())
[globals().__delitem__(k) for k in list(globals().keys()) if k not in _globals_init]
# ---- end ------------
# ---------------------
# ---- begin
__name__ = "some_other"
__doc__ = "testing 123"
def test1():
print(321)
sys.modules[__name__] = type(sys)(__name__, __doc__)
sys.modules[__name__].__dict__.update(globals())
[globals().__delitem__(k) for k in list(globals().keys()) if k not in _globals_init]
# ---- end ------------
# ----------------
# ---- example use
import test_module_1
test_module_1.test2()
import some_other
some_other.test1()
# this will fail (as it should)
test1()
Note, this isn't good practice, if you have this problem, you're probably better off with some alternative solution (such as using https://docs.python.org/3/library/zipimport.html)

See my GitHub project.
There is likely a better way for your needs. I developed this project/hack for programming contests which only allow the contestant to submit a single .py file. This allows one to develop a project with multiple .py files and then combine them into one .py file at the end.
My hack is a decorator #modulize which converts a function into a module. This module can then be imported as usual. Here is an example.
#modulize('my_module')
def my_dummy_function(__name__): # the function takes one parameter __name__
# put module code here
def my_function(s):
print(s, 'bar')
# the function must return locals()
return locals()
# import the module as usual
from my_module import my_function
my_function('foo') # foo bar
I also have a script which can combine a project of many .py files which import each other into one '.py' file.
For example, assume I had the following directory structure and files:
my_dir/
__main__.py
import foo.bar
fb = foo.bar.bar_func(foo.foo_var)
print(fb) # foo bar
foo/
__init__.py
foo_var = 'foo'
bar.py
def bar_func(x):
return x + ' bar'
The combined file will look as follows. The code on the top defines the #modulize decorator.
import sys
from types import ModuleType
class MockModule(ModuleType):
def __init__(self, module_name, module_doc=None):
ModuleType.__init__(self, module_name, module_doc)
if '.' in module_name:
package, module = module_name.rsplit('.', 1)
get_mock_module(package).__path__ = []
setattr(get_mock_module(package), module, self)
def _initialize_(self, module_code):
self.__dict__.update(module_code(self.__name__))
self.__doc__ = module_code.__doc__
def get_mock_module(module_name):
if module_name not in sys.modules:
sys.modules[module_name] = MockModule(module_name)
return sys.modules[module_name]
def modulize(module_name, dependencies=[]):
for d in dependencies: get_mock_module(d)
return get_mock_module(module_name)._initialize_
##===========================================================================##
#modulize('foo')
def _foo(__name__):
##----- Begin foo/__init__.py ------------------------------------------------##
foo_var = 'foo'
##----- End foo/__init__.py --------------------------------------------------##
return locals()
#modulize('foo.bar')
def _bar(__name__):
##----- Begin foo/bar.py -----------------------------------------------------##
def bar_func(x):
return x + ' bar'
##----- End foo/bar.py -------------------------------------------------------##
return locals()
def __main__():
##----- Begin __main__.py ----------------------------------------------------##
import foo.bar
fb = foo.bar.bar_func(foo.foo_var)
print(fb) # foo bar
##----- End __main__.py ------------------------------------------------------##
__main__()

Instead of appending the contents into a single *.py file, why not just import what you need from the code that the other people in your team write?

Python - create object of class from one package in different package

I started using Python few days back and I think I have a very basic question where I am stuck. Maybe I am not doing it correctly in Python so wanted some advice from the experts:
I have a config.cfg & a class test in one package lib as follows:
myProj/lib/pkg1/config.cfg
[api_config]
url = https://someapi.com/v1/
username=sumitk
myProj/lib/pkg1/test.py
class test(object):
def __init__(self, **kwargs):
config = ConfigParser.ConfigParser()
config.read('config.cfg')
print config.get('api_config', 'username')
#just printing here but will be using this as a class variable
def some other foos()..
Now I want to create an object of test in some other module in a different package
myProj/example/useTest.py
from lib.pkg1.test import test
def temp(a, b, c):
var = test()
def main():
temp("","","")
if __name__ == '__main__':
main()
Running useTest.py is giving me error:
...
print config.get('api_config', 'username')
File "C:\Python27\lib\ConfigParser.py", line 607, in get
raise NoSectionError(section)
ConfigParser.NoSectionError: No section: 'api_config'
Now if I place thie useTest.py in the same package it runs perfectly fine:
myProj/lib/pkg1/useTest.py
myProj/lib/pkg1/test.py
myProj/lib/pkg1/config.cfg
I guess there is some very basic package access concept in Python that I am not aware of or is there something I am doing wrong here?

The issue here is that you have a different working directory depending on which module is your main script. You can check the working directory by adding the following lines to the top of each script:
import os
print os.getcwd()
Because you just provide 'config.cfg' as your file name, it will attempt to find that file inside of the working directory.
To fix this, give an absolute path to your config file.
You should be able to figure out the absolute path with the following method since you know that config.cfg and test.py are in the same directory:
# inside of test.py
import os
config_path = os.path.join(os.path.dirname(os.path.abspath(__file__)),
'config.cfg')