We Keep Coding

Can't debug unittests in Pycharm

I'm having trouble debugging unit tests in pycharm. I can run them fine with my configuration but when I run the debugger I get the following error output:
Error
Traceback (most recent call last):
File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/case.py", line 59, in testPartExecutor
yield
File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/case.py", line 605, in run
testMethod()
File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 34, in testFailure
raise self._exception
ImportError: Failed to import test module: tests
Traceback (most recent call last):
File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 462, in _find_test_path
package = self._get_module_from_name(name)
File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
__import__(name)
File "/Users/paymahn/solvvy/scheduler/tests/__init__.py", line 2, in
import tests.test_setup_script
File "/Users/paymahn/solvvy/scheduler/tests/test_setup_script.py", line 3, in
import setup
File "/Applications/PyCharm.app/Contents/helpers/pydev/setup.py", line 87, in
data_files.append(('pydevd_attach_to_process', [os.path.join('pydevd_attach_to_process', f) for f in os.listdir('pydevd_attach_to_process') if accept_file(f)]))
FileNotFoundError: [Errno 2] No such file or directory: 'pydevd_attach_to_process'
My directory structure is:
My unittest configuration is:
tests/test_setup_script.py looks like:
import unittest
import os
import setup # the path to this file is scheduler/setup.py. This import may be breaking things
class TestSetupScript(unittest.TestCase):
def test_load_db_connection_valid_yaml_file(self):
file_name = "valid-yaml.yml"
with open(file_name, 'w') as file:
file.write("""
test:
hello world
a = b
""")
yaml = setup.load_yaml_configuration(file_name)
# I want to debug the line above, hence no assertions here
What does pydevd_attach_to_process do and how can I make sure it's found during debugging? Is the problem not actually related to that file/directory being found?

Indeed, the failure occurs because the "import setup" in your code imports the setup module which is part of PyCharm's debugger runtime instead of your own setup module. The easiest fix to this is to rename your setup.py file to something else and to update the imports in your code accordingly.

Pipeline code spanning multiple files in Apache Beam / Dataflow

After a lengthy search, I haven't found an example of a Dataflow / Beam pipeline that spans several files. Beam docs do suggest a file structure (under the section "Multiple File Dependencies"), but the Juliaset example they give has in effect a single code/source file (and the main file that calls it). Based on the Juliaset example, I need a similar file structure:
juliaset/__init__.py
juliaset/juliaset.py # actual code
juliaset/some_conf.py
__init__.py
juliaset_main.py
setup.py
Now I want to import .some_conf from juliaset/juliaset.py, which works when run locally but gives me an error when run on Dataflow
INFO:root:2017-12-15T17:34:09.333Z: JOB_MESSAGE_ERROR: (8cdf3e226105b90a): Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 706, in run
self._load_main_session(self.local_staging_directory)
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 446, in _load_main_session
pickler.load_session(session_file)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/internal/pickler.py", line 247, in load_session
return dill.load_session(file_path)
File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 363, in load_session
module = unpickler.load()
File "/usr/lib/python2.7/pickle.py", line 858, in load
dispatch[key](self)
File "/usr/lib/python2.7/pickle.py", line 1133, in load_reduce
value = func(*args)
File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 767, in _import_module
return getattr(__import__(module, None, None, [obj]), obj)
ImportError: No module named package_name.juliaset.some_conf
A full working example would be very much appreciated!

Can you verify your setup.py containing a structure like:
import setuptools
setuptools.setup(
name='My Project',
version='1.0',
install_requires=[],
packages=setuptools.find_packages(),
)
Import your modules like from juliaset.juliaset import SomeClass
And when you call the Python script, use python -m juliaset_main (without the .py)
Not sure if you already tried this, but just to be sure.

python/django-extensions ImportError: No module named common.types_

I have managed to run a simple script located in a 'scripts' folder using 'runscript' from django-extensions. The technique is described here [1] (https://django-extensions.readthedocs.org/en/latest/runscript.html).
The script will run successfully as follows:
python manage.py runscript mysimplescript --script-args Testing 123
Should I now move the script to a sub-folder, then it appears I need to specify the full path to the sub-folder:
python manage.py runscript scripts.myfolder.mysimplescript --script-args Testing 123
I have written a python script which runs successfully when called directly from the command line within its own project folder. The complex script uses a number of additional scripts located within a plethora of sub-folders.
I now want to call this script within django using django-extensions' 'runscript', which means it's being invoked from the 'mysite/' folder where 'manage.py' is located. The script 'mycomplexscript.py' is not in the 'scripts/' folder but relocated to a separate 'myapps/' project folder, along with the sub-folders and scripts.
I've shown the structure below that includes 'mysite/' and some of the sub-folders/files within 'myapp/', notably the module 'myapp/common/types_':
mysite/
manage.py
mysite/
__init__.py
settings.py
urls.py
wsgi.py
scripts
__init__.py
mysimplescript.py
myapps/
__init__.py
mycomplexscript.py
common/
__init__.py
comparison.py
serialisation/
__init__.py
serialisation.py
xml.py
types_/
__init__.py
enum.py
Django-extensions requires that 'mycomplexscript.py' must incorporate a 'run()' function. Line #6 imports a class from one of the project sub-folders:
def run(*script_args):
import os
import sys
import time
from common.serialisation import XML
When calling the script, the error occurs because the module 'common.types_' cannot be located.
python manage.py runscript myapps.mycomplexscript --script-args Testing 123
/usr/lib/python2.7/site-packages/Django-1.9.2-py2.7.egg/django/core/
management/base.py:265: RemovedInDjango110Warning: OptionParser usage for
Django management commands is deprecated, use ArgumentParser instead
RemovedInDjango110Warning)
Exception while running run() in 'myapps.mycomplexscript'
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/usr/lib/python2.7/site-packages/Django-1.9.2-py2.7.egg/django/core/
management/__init__.py", line 353, in execute_from_command_line
utility.execute()
File "/usr/lib/python2.7/site-packages/Django-1.9.2-py2.7.egg/django/core/
management/__init__.py", line 345, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/lib/python2.7/site-packages/django_extensions-1.6.1-py2.7.egg/
django_extensions/management/email_notifications.py", line 63, in
run_from_argv
super(EmailNotificationCommand, self).run_from_argv(argv)
File "/usr/lib/python2.7/site-packages/Django-1.9.2-py2.7.egg/django/core/
management/base.py", line 348, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/lib/python2.7/site-packages/django_extensions-1.6.1-py2.7.egg/
django_extensions/management/email_notifications.py", line 75, in execute
super(EmailNotificationCommand, self).execute(*args, **options)
File "/usr/lib/python2.7/site-packages/Django-1.9.2-py2.7.egg/django/core/
management/base.py", line 399, in execute
output = self.handle(*args, **options)
File "/usr/lib/python2.7/site-packages/django_extensions-1.6.1-py2.7.egg/
django_extensions/management/utils.py", line 57, in inner
ret = func(self, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/django_extensions-1.6.1-py2.7.egg/
django_extensions/management/commands/runscript.py", line 172, in handle
run_script(mod, *script_args)
File "/usr/lib/python2.7/site-packages/django_extensions-1.6.1-py2.7.egg/
django_extensions/management/commands/runscript.py", line 79, in run_script
mod.run(*script_args)
File "/usr/tester/SwFact/isg_cid-system_test_current/my_site/myapps/
mycomplexscript.py", line 9, in run
from myapps.common.serialisation import XML
File "/usr/tester/SwFact/isg_cid- system_test_current/my_site/myapps/
common/__init__.py", line 7, in <module>
from comparison import Comparison
File "/usr/tester/SwFact/isg_cid-system_test_current/my_site/myapps/
common/comparison.py", line 7, in <module>
from common.types_ import Enum
ImportError: No module named common.types_
The problematic Line #7 in 'comparison.py' is as follows:
from common.types_ import Enum
The error can be fixed if the full path to the root of the project is stated explicitly (+++):
from apps.common.types_ import Enum
This means that I would have to edit all the files and change the paths accordingly, which I'd like to avoid, if possible.
This link [2]:(Python: Referencing another project) describes the referencing another project from the command line, so I realize that this may nothing to do with the fact that I'm trying to do this by using django-extensions' 'runscript'.
Is there a way that I can resolve the path discrepancy, without having to modify all the files? Would setting PYTHONPATH be the answer, as suggested here [3]:(How to use PYTHONPATH)? I checked but 'echo $PYTHONPATH' currently doesn't return anything.
Apologies for the long-winded description.
Thanks in advance.

I tried sys.path.append("my/path") as recommended by link above and [4]:(Python sys.path - appending PYTHONPATH) and it seemed to work, so all the paths included in 'from' are now recognized without having to set the full path, as I described previously here (+++).
myapps/mycomplexscript:
# Procedures
def run(*script_args):
import os
import sys
import time
sys.path.append('/usr/tester/SwFact/isg_cid-system_test_current/
my_site/myapps')
from common.serialisation import XML
from common.task import Subscriber
from test_ import TestHost
from common.filesystem import Directory, File
from common.tables import Column, Condition, CSVTable, Group
from string import Template
from datetime import date, time, datetime, timedelta
from operator import itemgetter
print script_args
So the result when running the script from 'mysite/':
python manage.py runscript myapps.mycomplexscript --script-args Testing 123
/usr/lib/python2.7/site-packages/Django-1.9.2-py2.7.egg/django/core/
management/base.py:265: RemovedInDjango110Warning: OptionParser usage for
Django management commands is deprecated, use ArgumentParser instead
RemovedInDjango110Warning)
('Testing', '123')
It's not ideal to have to hard-code the path this inside the script, so perhaps it can be passed in as one of the arguments. The only problem with this is that imports appear at the top of the file, before any of the command line arguments are processed within the complex script however at least I know it works in principle.

ImportError: No module named simple while running manage.py test in django project

Traceback:
./manage.py test my_app
Traceback (most recent call last):
File "./manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/home/zuber/projects/private-CR/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
utility.execute()
File "/home/zuber/projects/private-CR/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 330, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/zuber/projects/private-CR/env/local/lib/python2.7/site-packages/django/core/management/commands/test.py", line 30, in run_from_argv
super(Command, self).run_from_argv(argv)
File "/home/zuber/projects/private-CR/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 381, in run_from_argv
parser = self.create_parser(argv[0], argv[1])
File "/home/zuber/projects/private-CR/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 354, in create_parser
self.add_arguments(parser)
File "/home/zuber/projects/private-CR/env/local/lib/python2.7/site-packages/django/core/management/commands/test.py", line 52, in add_arguments
test_runner_class = get_runner(settings, self.test_runner)
File "/home/zuber/projects/private-CR/env/local/lib/python2.7/site-packages/django/test/utils.py", line 152, in get_runner
test_module = __import__(test_module_name, {}, {}, force_str(test_path[-1]))
ImportError: No module named simple
I tried to remove init.py from app folder and then I've got "No module named app_name". When I removed init.py from project folder - console said "No module named settings". How to test my app?

So first of all, you need to put your two __init__.pys back where you found them. They are what allow you to import things from that module. Secondly, you should post the code in manage.py so we have a better idea of what is going on, but it looks to me like you had a line in there that looks something like import django.contrib.admin.util or import <something> from django.contrib.admin.util. This module was removed in the release of django that you're using, so you should replace any occurrances of django.contrib.admin.util with django.contrib.admin.utils.

I resolved the problem by removing
TEST_RUNNER = 'django.test.simple.DjangoTestSuiteRunner'
from settings.py
see related question running all tests post django 1.6

For me while using Pycharm the problem was that the PyCharm test runner was not compatible with Django 2.0
The solution was:
replace line 254:EOF with
if VERSION[1] > 1 or VERSION[0] > 1:
return DjangoTeamcityTestRunner(**options).run_tests(test_labels,
extra_tests=extra_tests, **options)
return run_the_old_way(extra_tests, options, test_labels, verbosity)
Instead of:
if VERSION[1] > 1:
return DjangoTeamcityTestRunner(**options).run_tests(test_labels,
extra_tests=extra_tests, **options)
return run_the_old_way(extra_tests, options, test_labels, verbosity)

How can I identify what package an AttributeError: 'module' is referring to?

I am trying to move a Python pyramid app that I'm writing from one server to another. I checked the code out from source control and ran a python setup.py develop to prepare the environment, but when I try to run pserve development.ini I get the following traceback:
2013-02-27 20:38:20,269 INFO [pyramid_scss][MainThread] adding asset path /home/pgrace/repos/Asterisk-WebApps/Cedar-Root/opt/cedar/cedar/assets/scss
Traceback (most recent call last):
File "/home/pgrace/venvs/pyramid/bin/pserve", line 8, in <module>
load_entry_point('pyramid==1.4', 'console_scripts', 'pserve')()
File "/home/pgrace/venvs/pyramid/lib/python2.6/site-packages/pyramid/scripts/pserve.py", line 50, in main
return command.run()
File "/home/pgrace/venvs/pyramid/lib/python2.6/site-packages/pyramid/scripts/pserve.py", line 304, in run
global_conf=vars)
File "/home/pgrace/venvs/pyramid/lib/python2.6/site-packages/pyramid/scripts/pserve.py", line 328, in loadapp
return loadapp(app_spec, name=name, relative_to=relative_to, **kw)
File "/home/pgrace/venvs/pyramid/lib/python2.6/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
return loadobj(APP, uri, name=name, **kw)
File "/home/pgrace/venvs/pyramid/lib/python2.6/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
return context.create()
File "/home/pgrace/venvs/pyramid/lib/python2.6/site-packages/paste/deploy/loadwsgi.py", line 710, in create
return self.object_type.invoke(self)
File "/home/pgrace/venvs/pyramid/lib/python2.6/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke
return fix_call(context.object, context.global_conf, **context.local_conf)
File "/home/pgrace/venvs/pyramid/lib/python2.6/site-packages/paste/deploy/util.py", line 56, in fix_call
val = callable(*args, **kw)
File "/home/pgrace/repos/Asterisk-WebApps/Cedar-Root/opt/cedar/cedar/__init__.py", line 18, in main
config.include("pyramid_scss")
File "/home/pgrace/venvs/pyramid/lib/python2.6/site-packages/pyramid/config/__init__.py", line 773, in include
c(configurator)
File "/home/pgrace/venvs/pyramid/lib/python2.6/site-packages/pyramid_scss/__init__.py", line 88, in includeme
scss.LOAD_PATHS = ','.join([scss.LOAD_PATHS, ','.join(load_paths)])
AttributeError: 'module' object has no attribute 'LOAD_PATHS'
Now, I can tell that the problem originates in pyramid_scss, but as far as I can tell the code it references is fine, it's as if the module class does not contain the LOAD_PATHS directive. I'm trying to figure out which module it's referring to so that I can track down if I'm missing a dependency or something. Does anyone have any ideas how I'd go about identifying where the module reference is pointing to so I can check that code?
EDIT:
The error comes in the includeme definition which from what I've read is automatically included every time a pyramid-specific package is ...instantiated? Maybe? At any rate, it's saying that scss.LOAD_PATHS does not exist, yes, but there's no scss class in either pyramid_scss or pyScss packages, leading me to wonder if there's some other class that's being late-bound or something.
def includeme(config):
load_paths, static_path = _get_import_paths(config.registry.settings)
scss.LOAD_PATHS = ','.join([scss.LOAD_PATHS, ','.join(load_paths)])
scss.STATIC_ROOT = static_path
config.add_renderer('scss', renderer_factory)
The Scss class in pyScss does include a LOAD_PATHS directive but I don't see where the object scss is bound to a Scss class definition. I'm wondering whether there's something missing in the def includeme from above that might be part of the answer, but then it begs the question -- this worked right on the other machine, why break now? There's something else missing that I'm just not picking up on.

From examining the code for the scss package, it looks like the LOAD_PATHS global variable might have been moved from __init__.py to config.py.
Thus, if you want to try to fix the pyramid_scss app, you could change line 88 of pyramid_scss/__init__.py to read:
scss.config.LOAD_PATHS = ','.join([scss.config.LOAD_PATHS, ','.join(load_paths)])

The one-but-last line tells you which module is affected:
scss.LOAD_PATHS = ','.join([scss.LOAD_PATHS, ','.join(load_paths)])
AttributeError: 'module' object has no attribute 'LOAD_PATHS'
So the scss module has no LOAD_PATHS attribute.