We Keep Coding

multiprocessing exits before doing any job

I have inherited a certain parser that is supposed to parse 10 files with ~4m lines each.
The code was written in Python 2, which I updated.
There is a multiprocessing logic which i just don't seem to be able to get to work.
from multiprocessing.pool import ThreadPool
import glob
DATADIR = 'home/my_dir/where/all/my/files/are'
def process_file(filepath):
# read line by line, parse and insert to postgres database.
def process_directory(dirpath):
pattern = f'{dirpath}/*dat' # files have .dat extension.
tp = ThreadPool(10)
for filepath in glob.glob(pattern):
print(filepath)
tp.apply_async(process_file, filepath)
tp.close()
tp.join()
if __name__ == '__main__':
process_directory(DATADIR)
I have gone through a lot of the documentation and some similar questions but it just doesn't seem to work.
With the parser code what happens is that I do get printed on the console all the paths of the file that I need parsed, but then that's it the program doesn't do anything else.

The problem is in the way you're calling apply_async. I made a simple reproducer of your problem, but with a slight tweak to get the result from each call:
from multiprocessing.pool import ThreadPool
def func(f):
print("hey " + f)
return f + "1"
l = ["name", "name2", "name3"]
pool = ThreadPool(3)
out = []
for a in l:
print(a)
out.append(pool.apply_async(func, a))
# Check the response from each `apply_async` call
for a in out:
a.get()
pool.close()
pool.join()
This returns an error:
Traceback (most recent call last):
File "a.py", line 16, in <module>
a.get()
File "/usr/lib64/python3.4/multiprocessing/pool.py", line 599, in get
raise self._value
File "/usr/lib64/python3.4/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
TypeError: func() takes 1 positional argument but 4 were given
It thinks you're passing four positional arguments, instead of one. This is because apply_async wants all the arguments passed in a tuple, like this:
pool.apply_async(func, (a,))
If you put filepath in a tuple when you call apply_async, I think you'll get the behavior you expect.
It's also worth noting that your usecase is well-suited to using pool.map instead of apply_async, which is a little more succinct:
pool.map(process_file, glob.glob(pattern))

Python - Access "import" from within a class

This has been vexing me for a while now. I am trying to create a very simple REST-like interface (without using third-party libraries, which I know are available).
The idea behind it is that I can have a directory, for example mylib, where I can drop in python files, like do_something.py and, by POSTing to http://localhost/do_something the code will spring into life and do something!
I think I have managed to get somewhere near to my goal with the following structure:
The contents of the files are as follows.
example.py
from http.server import HTTPServer
from http.server import BaseHTTPRequestHandler
import json, logging
from mylib import my_module
class MyRequestHandler (BaseHTTPRequestHandler):
# Send JSON responses
# -----------
def send_json(self, json_message, response_code=200):
self.send_response(response_code)
self.send_header('Content-type', 'application/json')
self.end_headers()
self.request.sendall(json.dumps(json_message).encode())
# Get JSON requests
# ----------
def get_json(self):
body = self.rfile.read(int(self.headers.get('Content-Length')))
if (body):
try:
receivedData = json.loads(body.decode())
except:
self.send_json({"Status": "Error", "Message": "Invalid JSON received"}, 400)
receivedData = None
else:
receivedData = None
return receivedData
# POST
# ---------
def do_POST(self):
module_to_call = (self.path).replace('/', '.')[1:]
if module_to_call.endswith('.'): # Remove trailing dot
module_to_call = module_to_call[:-1]
print("Path is: '" + module_to_call + "'")
# invoke function
module_to_call = getattr(self, module_to_call)
response = module_to_call()
self.send_json(response)
# GET
# --------
def do_GET(self):
pass
# -----------------------------------------------------------------------------
# Server startup code
# -------------------
def start_server():
# Begin serving
# -------------
port = 8003
server = HTTPServer(('', port), MyRequestHandler)
print(("Server now running on port {0} ...").format(port))
server.serve_forever()
# -----------------------------------------------------------------------------
# Start the Server
# ----------------
if __name__ == '__main__':
start_server()
my_module.py
def my_module():
print("Hello World!")
return{'Greeting': 'Hello World!'}
When I fire up the server and attempt to POST to http://localhost:8003/my_module, I get the following output:
Server now running on port 8003 ...
Path is: 'my_module'
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 59541)
Traceback (most recent call last):
File "C:\Users\Test\AppData\Local\Programs\Python\Python35-32\lib\socketserver.py", line 313, in _handle_request_noblock
self.process_request(request, client_address)
File "C:\Users\Test\AppData\Local\Programs\Python\Python35-32\lib\socketserver.py", line 341, in process_request
self.finish_request(request, client_address)
File "C:\Users\Test\AppData\Local\Programs\Python\Python35-32\lib\socketserver.py", line 354, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Users\Test\AppData\Local\Programs\Python\Python35-32\lib\socketserver.py", line 681, in __init__
self.handle()
File "C:\Users\Test\AppData\Local\Programs\Python\Python35-32\lib\http\server.py", line 422, in handle
self.handle_one_request()
File "C:\Users\Test\AppData\Local\Programs\Python\Python35-32\lib\http\server.py", line 410, in handle_one_request
method()
File ".\example.py", line 43, in do_POST
module_to_call = getattr(self, module_to_call)
AttributeError: 'MyRequestHandler' object has no attribute 'my_module'
----------------------------------------
This makes perfect sense, since 'MyRequestHandler" does not have an attribute "my_module"! What I can't wrap my head around, is how to fix this?
Should I pass "mylib" into MyRequestHandler? Should I perform the import within the class (but then the functionality would only be available within the class)?
I'm trying to keep things clean and simple, so that even a Python novice (like I seem to be!) can just write a standalone script, drop it into "mylib" and everything "just works". The novice could visit the web address of their script and have it magically run.
Any help or advice would be gratefully received.

Use the __import__() method:
temp = __import__('mylib', globals(), locals(), ['module_to_call'], -1)
response = temp.module_to_call()
I use 2.6 at work, and this is usually used by those using even 2.7 because the importlib module is far more robust in 3. If you are using 3 you can do the following:
from importlib import import_module
temp = import_module('mylib')
but now you have to use getattr to get the function you want to be called
func_to_call = getattr(temp, 'module_to_call')
response = func()
Or you can have a dictionary of functions in another module, but that will require a lot of work as that dictionary grows.

django.core.exceptions ImproperlyConfigured error when running selenium functional test in python

I've had a similar problem to this one before where I received a nearly identical error message, i.e.
C:\Users\David J\optilab\optilab_project>python functional_tests.py
Traceback (most recent call last):
File "functional_tests.py", line 4, in <module>
from optilab.models import Design
File "C:\Users\David J\optilab\optilab_project\optilab\models.py", line 10, in <module>
class Design(models.Model):
File "C:\Users\David J\optilab\optilab_project\optilab\models.py", line 11, in Design
name = models.CharField(max_length=30)
File "C:\Python27\lib\site-packages\django\db\models\fields\__init__.py", line 1072, in __init__
super(CharField, self).__init__(*args, **kwargs)
File "C:\Python27\lib\site-packages\django\db\models\fields\__init__.py", line 166, in __init__
self.db_tablespace = db_tablespace or settings.DEFAULT_INDEX_TABLESPACE
File "C:\Python27\lib\site-packages\django\conf\__init__.py", line 55, in __getattr__
self._setup(name)
File "C:\Python27\lib\site-packages\django\conf\__init__.py", line 41, in _setup
% (desc, ENVIRONMENT_VARIABLE))
django.core.exceptions.ImproperlyConfigured: Requested setting DEFAULT_INDEX_TABLESPACE, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.
But it was being produced because I was calling my test incorrectly from the shell. Now I am attempting to run a functional test, which I have done several times before successfully using the command
"python functional_tests.py"
in my project folder -- but it produces the above error message. Can anyone tell me what I'm doing wrong, or why this may have worked before but not now? thank.
Edit:
Here is functional_tests.py:
from selenium import webdriver
import unittest
from selenium.webdriver.common.keys import Keys
from optilab.models import Design
from automa.api import *
class HomeTest(unittest.TestCase):
def setUp(self):
self.browser = webdriver.Firefox()
self.browser.get('http://localhost:8000/lab')
self.new_design = self.browser.find_element_by_id("new_design")
self.load_design = self.browser.find_element_by_id("load_test")
self.design_name = self.browser.find_element_by_id("design_name")
self.browser.implicitly_wait(3)
def test_latest_designs_displayed_in_table(self):
#
self.browser.implicitly_wait(3)
self.design_table = self.browser.find_element_by_id("design_table")
self.rows = self.browser.find_elements_by_tag_name('tr')
self.assertTrue(
any(row.text == 'My First Design' for row in rows),
"New design did not appear in test table."
)
def test_entering_text_activates_new(self):
# user enters name of design. new becomes clickable.
self.browser.implicitly_wait(3)
self.design_name.send_keys("sample_design")
def test_new_test_can_be_saved_and_retrieved(self):
name = "sample design"
self.design_name.send_keys(name)
self.new_design.click()
self.browser.implicitly_wait(3)
self.saved_design = Design.objects.get(name=design_name)
def test_entering_name_of_previously_saved_test_activates_load(self):
# user enters the name of a previously saved test. load button becomes clickable.
design = Design(name="sample_design")
saved_design = Design.objects.get(id=1).name
self.browser.implicitly_wait(3)
self.design_name.send_keys(saved_design)
self.load_design.click()
def test_entering_name_of_previously_saved_test_but_clicking_new_prompts_overwrite(self):
# user entes name of saved test but clicks new. dialogue box asks if test should be overwritten.
previously_saved_design = Design.objects.get(id=1).name
self.design_name.send_keys(previously_saved_design)
def test_clicking_save_redirects_to_model(self):
pass
def tearDown(self):
self.browser.quit()
if __name__ == '__main__':
unittest.main()

The start of your script should look something like this
import os, sys
sys.path.append(os.getcwd())
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "optilab.settings") # or whatever
import django
django.setup()
from selenium import webdriver
import unittest
from selenium.webdriver.common.keys import Keys
from optilab.models import Design
from automa.api import * # try not to do this

This happens because Django has not been correctly setup before trying to call settings, it has no way to know which project settings should be used without the "DJANGO_SETTINGS_MODULE" variable and the django.setup() call is essential to import models and access the database. (see e4c5's answer for that method)
However, this is not the recommended method or the easiest one. You can call the functional tests just like the unit tests from manage.py test and Django will automagically setup everything for you, it'll even use a tempdb for functional tests. If your functional tests include insertions and updates in the functional tests (and you should) a tempdb comes in handy.
To use this facility though, you must change a few things in your code.
First there is the matter of the filename, the default pattern for manage.py test(doc) is "test*.py", you can change this with -p or --pattern but the usual solution is to rename functional_tests.py to tests.py and place it inside a functional_tests folder. That way you can call just the functional tests by calling manage.py test functional_tests.
Also, instead of importing TestCase from unittest you should look into using TestCase(doc) from django.test or even LiveServerTestCase(doc) which would save you the trouble of running manage.py runserver yourself.

Python custom __init__ not called when imported

I have two files,
main.py
options.py
main.py is my main program. I import options.py via:
from options import Options
whereas "options" is a class I defined in options.py
Options has a custom init method:
def __int__(self, afc="red", awc="orange", asc="gray", apn=1738):
If I try to make an object from within main.py like so:
options = Options("red","green","blue",1738)
Python tells me that there are unexpected arguments. How can I instantiate the object with my custom init method?
here is the respective code of my options.py file:
class Options:
anyBarFailureColor = ""
anyBarWarningColor = ""
anyBarScrubInProgressColor = ""
anyBarPortNumber = 0
def __int__(self, afc="red", awc="orange", asc="gray", apn=1738):
self.anyBarFailureColor = afc
self.anyBarWarningColor = awc
self.anyBarScrubInProgressColor = asc
self.anyBarPortNumber = apn
here is the Python error:
Traceback (most recent call last):
File "/pathToFile/Python/project/main", line 58, in <module>
start()
File "/pathToFile/Python/project/main.py", line 55, in start
options,listOfPools=mapArgs(args)
File "/pathToFile/Python/project/main.py", line 39, in mapArgs
options = Options(anyBarFailureColor,anyBarWarningColor,anyBarScrubInProgressColor,anyBarPortNumber)
TypeError: object() takes no parameters

One should neither type int nor index and wonder why it isn't working. Sigh. Typo, never was a init to begin with. Now everything works.

Unittesting a file by calling it from another file in Python

I am new to unnitest module. I have a file that has unittest in it. The file is something like ...
File1.py
class ABC (unittest.TestCase):
def setUp(self):
# Do some work here
def test_123(self, a,b,c):
# Do some work here
if __name__ == "__main__":
unittest.main()
*Now I am calling this file from another file by passing values to the function "test_123".* But python displays the following error. Could anybody please help!
Traceback (most recent call last):
File "caller_file.py", line 20, in <module>
r = file1.ABC()
File "/usr/lib/python2.7/unittest/case.py", line 191, in __init__
(self.__class__, methodName))
ValueError: no such test method in <class 'file1.ABC'>: runTest

You can run file1.ABC test case like this:
import unittest
import file1
suite = unittest.TestLoader().loadTestsFromTestCase(file1.ABC)
unittest.TextTestRunner(verbosity=2).run(suite)
Also you need to add the self argument to the setUp and test_123 methods and self should be the sole argument.

I run into similar problems with my unittests because missing entries in search path for modules.
I solved it by creating
my_env = os.environ.copy()
if not 'PYTHONPATH' in my_env:
my_env['PYTHONPATH'] = ''
my_env['PYTHONPATH'] += ';' + ';'.join(
[os.path.abspath('.'),
os.path.abspath('..'),
os.path.abspath('..\\..')])
and then the call of the file
_ = subprocess.check_output(filepath, shell=True, env=my_env)
I just added the current path environment because the calling-file is in the same directories. Maybe you have to adjust that.