We Keep Coding

Importing a function from a sibling folder in python producing strange errors

I'm have a script 'zigbot.py' which was working, but I'm restructuring to dockerize and now I'm questioning my ability to code at all. What I'm trying to do is have a telegram-bot container, and a 'web' container (FLASK), and a nginx container.
Next to zigbot.py I have a folder - 'bot', which has many of my scripts and functions in. Upon trying from bot.somescript import a_function I am getting error after error.
Project structure
zigbot
bot
bot
__init__.py
conversations.py
ct.py
Docerfile
funcs.py
pricedata.py
requirements.txt
util.py
zigbot.py
nginx
somestuff
web
app
migrations
templates
__init__.py
config.py
forms.py
functions.py
models.py
routes.py
signals.py
__init__.py
Dockerfile
requirements.txt
zigweb.py
When running the code below, I get a whole range of strange errors - the one listed below shows ImportError - it can't find 'onboard' in funcs, but its definitely there. Before the restructure, this was working. If I comment out 'onboard' it errors on every function in the list.
from bot.conversations import key_conversation
from bot.crypto_functions import satoshi_to_btc
from bot.funcs import onboard, main_menu, new_message1, new_keyboard, onboarded_keyboard, \
onboarded_message, blank_signal_message, send_signal_format, get_or_create_user, signal_detected_keyboard, log_me
Traceback (most recent call last):
File "zigbot.py", line 15, in <module>
from bot.funcs import onboard, main_menu, new_message1, new_keyboard, onboarded_keyboard, \
File "ZigBot\bot\bot\funcs.py", line 2, in <module>
from zigbot.web.app import db, app
File "ZigBot\bot\zigbot.py", line 15, in <module>
from bot.funcs import onboard, main_menu, new_message1, new_keyboard, onboarded_keyboard, \
ImportError: cannot import name 'onboard'
So I tried adding a .bot.conversation for (relative?) import, but this produces an even weirder error. I've included the code, the error, then the function I'm trying to import below:
from .bot.conversations import key_conversation
from .bot.crypto_functions import satoshi_to_btc
from .bot.funcs import onboard, main_menu, new_message1, new_keyboard, onboarded_keyboard, \
onboarded_message, blank_signal_message, send_signal_format, get_or_create_user, signal_detected_keyboard, log_me
from ..web.app import db
from ..web.app.models import User, Signal
Traceback (most recent call last):
File "zigbot.py", line 13, in <module>
from .bot.conversations import key_conversation
ModuleNotFoundError: No module named '__main__.bot'; '__main__' is not a package
conversations.py
# Initialise conversationHandler states
def key_conversation(cancel):
# Define keyboards
keyboard = [
[InlineKeyboardButton('Previous', callback_data='onboard:key'),
InlineKeyboardButton('Next', callback_data='onboard:alldone')]]
conversation_keyboard = InlineKeyboardMarkup(keyboard)
Finally, I think I'm getting closer to the problem, I go back to the way I think it's supposed to be imported. From bot.funcs import x,y,y... I look in funcs.py and this is how its importing some other code from my Flask web app, but it doesn't like it.
funcs.py
from telegram import InlineKeyboardButton, InlineKeyboardMarkup
from web.app import db, app
from web.app.models import User
Traceback (most recent call last):
File "zigbot.py", line 15, in <module>
from bot.funcs import onboard, main_menu, new_message1, new_keyboard, onboarded_keyboard, \
File "ZigBot\bot\bot\funcs.py", line 2, in <module>
from web.app import db, app
ModuleNotFoundError: No module named 'web'
So I change the import to go up a level since web is two levels above bot - which contains the script I'm importing to. So zigbot>bot>bot>funcs.py trying to import from zigbot>web>app>
from telegram import InlineKeyboardButton, InlineKeyboardMarkup
from ..web.app import db, app
from ..web.app.models import User
import telegram
Traceback (most recent call last):
File "zigbot.py", line 15, in <module>
from bot.funcs import onboard, main_menu, new_message1, new_keyboard, onboarded_keyboard, \
File "\ZigBot\bot\bot\funcs.py", line 2, in <module>
from ..web.app import db, app
ValueError: attempted relative import beyond top-level package
This obviously doesn't work either. Given my structure of having bot and web how do I get this to work? I even tried pulling the scripts out of the second 'bot' folder but I get the same issues. Finally, my weirdest error which might provide a clue to what I'm doing wrong, if I change the import on funcs.py to go one level up, rather than two, I get an even stranger traceback.
from telegram import InlineKeyboardButton, InlineKeyboardMarkup
from .web.app import db, app
from .web.app.models import User
import telegram
Traceback (most recent call last):
File "zigbot.py", line 15, in <module>
from bot.funcs import onboard, main_menu, new_message1, new_keyboard, onboarded_keyboard, \
File "C:\Users\phill\PycharmProjects\ZigBot\bot\bot\funcs.py", line 2, in <module>
from .web.app import db, app
ModuleNotFoundError: No module named 'bot.web'
For clarity I removed some 'C:\Users\phill' from the tracebacks before realising it was pointless to remove that.

I think there's a fundamental flaw in your split if you have library dependencies across the boundary. Assuming that there are no imports of bot-modules in your web container, I would suggest breaking off the shared code into a module named weblib or something (alongside the bot-, nginx- and web-folders).
You still should not try to import across would-be container boundaries however, so just make sure weblib is in your path or installed in your virtual environment or however you want to handle it. Basically, bot and web should run in separate environments and should have weblib as a dependency.
Someone wiser than me might be able to explain it better.

For relative imports to work all the packages and subpackages must be in the sys.path - to achieve this you should run from the top-level directory as in:
C:\Users\phill\zigbot>python -m bot.zigbot
This will make your current working dir (zigbot) available to sys.path and the subpackages will be resolved correctly

How to avoid ImportError in twisted tac file?

I have a twisted tac file (twisted_service.py) with a code:
from twisted.application import service
# application.py file in the same dir
from .application import setup_reactor
class WebsocketService(service.Service):
def startService(self):
service.Service.startService(self)
setup_reactor()
application = service.Application("ws")
ws_service = WebsocketService()
ws_service.setServiceParent(application)
Here is application.py file, which sets up the reactor:
# -*- coding: utf-8 -*-
from twisted.web.server import Site
from twisted.web.static import Data
from twisted.internet import reactor, defer
from autobahn.twisted.resource import WebSocketResource
from autobahn.twisted.websocket import WebSocketServerFactory
from txsni.snimap import SNIMap
from txsni.maputils import Cache
from txsni.snimap import HostDirectoryMap
from twisted.python.filepath import FilePath
from tools.database.async import pg_conn
from tools.database import makedsn
from tools.config import main_db
from tools.modules.external import flask_setup
import tools.config as config
import websockethandlers as wsh
from pytrapd import TrapsListener
PROTOCOLMAP = {
'portcounters': wsh.PortCounters,
'eqcounters': wsh.EquipmentCounters,
'settings': wsh.Settings,
'refresh': wsh.Refresher,
'montraps': wsh.TrapsMonitoring,
'fdbs': wsh.FdbParser,
'portstate': wsh.PortState,
'cable': wsh.CableDiagnostic,
'eqcable': wsh.EquipmentCableDiagnostic,
'igmp': wsh.Igmp,
'ipmac': wsh.IpMac,
'lldp': wsh.LLDPParser,
'alias': wsh.AliasSetup,
'ping': wsh.Ping
}
#defer.inlineCallbacks
def setup_reactor():
flask_setup()
yield pg_conn.connect(makedsn(main_db))
root = Data("", "text/plain")
for key in PROTOCOLMAP:
factory = WebSocketServerFactory("wss://localhost:%s" % config.ws_port)
factory.protocol = PROTOCOLMAP[key]
resource = WebSocketResource(factory)
root.putChild(key, resource)
site = Site(root)
context_factory = SNIMap(
Cache(HostDirectoryMap(FilePath(config.certificates_directory)))
)
reactor.listenSSL(config.ws_port, site, context_factory)
traps_listener = TrapsListener()
traps_listener.listen_traps(config.trap_ip)
traps_listener.listen_messages(config.fifo_file)
if __name__ == '__main__':
setup_reactor()
import sys
from twisted.python import log
log.startLogging(sys.stdout)
reactor.run()
I use twistd -noy twisted_service.py command to run the twisted service. It has been working for twisted 16.3.2 version. After upgrade to any next version I got the error:
Unhandled Error
Traceback (most recent call last):
File "/home/kalombo/.virtualenvs/dev/local/lib/python2.7/site-packages/twisted/application/app.py", line 662, in run
runApp(config)
File "/home/kalombo/.virtualenvs/dev/local/lib/python2.7/site-packages/twisted/scripts/twistd.py", line 25, in runApp
_SomeApplicationRunner(config).run()
File "/home/kalombo/.virtualenvs/dev/local/lib/python2.7/site-packages/twisted/application/app.py", line 380, in run
self.application = self.createOrGetApplication()
File "/home/kalombo/.virtualenvs/dev/local/lib/python2.7/site-packages/twisted/application/app.py", line 445, in createOrGetApplication
application = getApplication(self.config, passphrase)
--- <exception caught here> ---
File "/home/kalombo/.virtualenvs/dev/local/lib/python2.7/site-packages/twisted/application/app.py", line 456, in getApplication
application = service.loadApplication(filename, style, passphrase)
File "/home/kalombo/.virtualenvs/dev/local/lib/python2.7/site-packages/twisted/application/service.py", line 412, in loadApplication
application = sob.loadValueFromFile(filename, 'application')
File "/home/kalombo/.virtualenvs/dev/local/lib/python2.7/site-packages/twisted/persisted/sob.py", line 177, in loadValueFromFile
eval(codeObj, d, d)
File "twisted_service.py", line 3, in <module>
from .application import setup_reactor
exceptions.ImportError: No module named application
How should I run the twisted or import module properly?

What the answer I found nearly yours. like the follows:
import os
import sys
sys.path = [os.path.join(os.getcwd(), '.'), ] + sys.path
just add the current working directory to the sys.path.
But I have not found more better method.... I think this not very good.

I found the answer here http://twistedmatrix.com/pipermail/twisted-python/2016-September/030783.html
It is a new feature in Twisted 16.4.0. In previous versions twistd script automatically added working dir to system paths, from 16.4.0 version i have to added it manually. It can be added something like this in twisted_service.py file:
import os
import sys
TWISTED_DIR = os.path.dirname(os.path.abspath(__file__))
sys.path.append(TWISTED_DIR)

Error trying to import between python files

I have a basic parser app I'm building in Python. I monitors a folder and imports files when they are dropped there. I have a MongoDB that I'm trying to save the imports to. There's almost nothing to it. The problem happens when I try to include one of my class/mongo-document files. I'm sure it's a simple syntax issue I don't understand. I have all my requirements installed, and I'm running this in an virtual env. This is my first python app though, so it's likely something I'm not seeing.
My file structure is
application.py
requirements.txt
__init__.py
-services
parser.py
__init__.py
-models
hl7message.py
__init__.py
Here is application.py
from mongoengine import connect
import os, os.path, time
from services import parser
db = connect('testdb')
dr = 'C:\\Imports\\Processed'
def processimports():
while True:
files = os.listdir(dr)
print(str(len(files)) + ' files found')
for f in files:
msg = open(dr + '\\' + f).read().replace('\n', '\r')
parser.parse_message(msg)
print('waiting')
time.sleep(10)
processimports()
requirements.txt
mongoengine
hl7
parser.py
import hl7
from models import hl7message
def parse_message(message):
m = hl7.parse(str(message))
h = hl7message()
hl7message.py
from utilities import common
from application import db
import mongoengine
class Hl7message(db.Document):
message_type = db.StringField(db_field="m_typ")
created = db.IntField(db_field="cr")
message = db.StringField(db_field="m")
If I don't include the hl7message class in the parser.py it runs fine, but as soon as I include it I get the error, so I'm sure it has something to do with that file. The error message though isn't to helpful. I don't know if I've got myself into some kind of include loop or something.
Sorry, stack trace is below
Traceback (most recent call last):
File "C:/OneDrive/Dev/3/Importer/application.py", line 3, in <module>
from services import parser
File "C:\OneDrive\Dev\3\Importer\services\parser.py", line 2, in <module>
from models import hl7message
File "C:\OneDrive\Dev\3\Importer\models\hl7message.py", line 2, in <module>
from application import db
File "C:\OneDrive\Dev\3\Importer\application.py", line 23, in <module>
processimports()
File "C:\OneDrive\Dev\3\Importer\application.py", line 17, in processimports
parser.parse_message(msg)
AttributeError: module 'services.parser' has no attribute 'parse_message'

This is a circular import issue. Application.py imports parser, which imports h17 which imports h17message, which imports application which runs processimports before the whole code of the parser module has been run.
It seems to me that service modules should not import application. You could create a new module common.py containing the line db = connect('testdb') and import db from common both in application.py and in h17message.

keyring module is not included while packaging with py2exe

I am making an app using python 2.7 on windows and keyring-3.2.1 . In my python code on eclipse, I used
import keyring
keyring.set_password("service","jsonkey",json_res)
json_res= keyring.get_password("service","jsonkey")
is working fine as I am storing json response in keyring. But, when I converted python code into exe by using py2exe, it shows import error keyring while making dist. Please suggest how to include keyring in py2exe.
Traceback (most recent call last):
File "APP.py", line 8, in <module>
File "keyring\__init__.pyc", line 12, in <module>
File "keyring\core.pyc", line 15, in <module>
File "keyring\util\platform_.pyc", line 4, in <module>
File "keyring\util\platform.pyc", line 29, in <module>
AttributeError: 'module' object has no attribute 'system'
platform_.py code is :
from __future__ import absolute_import
import os
import platform
def _data_root_Windows():
try:
root = os.environ['LOCALAPPDATA']
except KeyError:
# Windows XP
root = os.path.join(os.environ['USERPROFILE'], 'Local Settings')
return os.path.join(root, 'Python Keyring')
def _data_root_Linux():
"""
Use freedesktop.org Base Dir Specfication to determine storage
location.
"""
fallback = os.path.expanduser('~/.local/share')
root = os.environ.get('XDG_DATA_HOME', None) or fallback
return os.path.join(root, 'python_keyring')
# by default, use Unix convention
data_root = globals().get('_data_root_' + platform.system(), _data_root_Linux)
platform.py code is:
import os
import sys
# While we support Python 2.4, use a convoluted technique to import
# platform from the stdlib.
# With Python 2.5 or later, just do "from __future__ import absolute_import"
# and "import platform"
exec('__import__("platform", globals=dict())')
platform = sys.modules['platform']
def _data_root_Windows():
try:
root = os.environ['LOCALAPPDATA']
except KeyError:
# Windows XP
root = os.path.join(os.environ['USERPROFILE'], 'Local Settings')
return os.path.join(root, 'Python Keyring')
def _data_root_Linux():
"""
Use freedesktop.org Base Dir Specfication to determine storage
location.
"""
fallback = os.path.expanduser('~/.local/share')
root = os.environ.get('XDG_DATA_HOME', None) or fallback
return os.path.join(root, 'python_keyring')
# by default, use Unix convention
data_root = globals().get('_data_root_' + platform.system(), _data_root_Linux)

The issue you're reporting is due to an environment that contains invalid modules, perhaps from an improper installation of one version of keyring over another. You will want to ensure that you've removed remnants of the older version of keyring. In particular, make sure there's no file called keyring\util\platform.* in your site-packages.
After doing that, however, you'll encounter another problem. Keyring loads its backend modules programmatically, so py2exe won't detect them.
To work around that, you'll want to add a 'packages' declaration to your py2exe options to specifically include the keyring.backends package. I invoked the following setup.py script with Python 2.7 to convert 'app.py' (which imports keyring) to an exe:
from distutils.core import setup
import py2exe
setup(
console=['app.py'],
options=dict(py2exe=dict(
packages='keyring.backends',
)),
)
The resulting app.exe will import and invoke keyring.

No module named v4.proto.omni

I have installed pysnmp-4.x.I am getting following error during running a SNMP program.
I am using pysnmpSE 3.5.2 now but getting same error. I found that pysnmpSE doesn't hav v4 module. I was suggested that following error should resolved if pySNMP SE 3.x is used.
Traceback (most recent call last):
File "C:\Documents and Settings\ggne0622\Desktop\Python\google-python-exercises\babynames\SimpleAgent.py", line 18, in <module>
from twistedsnmp import agent, agentprotocol, bisectoidstore
File "C:\Python27\Lib\site-packages\twistedsnmp\agent.py", line 4, in <module>
from twistedsnmp import datatypes
File "C:\Python27\Lib\site-packages\twistedsnmp\datatypes.py", line 7, in <module>
from twistedsnmp.pysnmpproto import v2c,v1
File "C:\Python27\Lib\site-packages\twistedsnmp\pysnmpproto.py", line 13, in <module>
from pysnmp.v4.proto.omni import v2c,v1, error, rfc1157, rfc1905
ImportError: No module named v4.proto.omni
Code:
#!/usr/bin/env python
from twisted.internet.iocpreactor import reactor
from twisted.internet import error as twisted_error
from twistedsnmp import agent, agentprotocol, bisectoidstore
#from twisted.internet import interfaces
try:
from twistedsnmp import bsdoidstore
except ImportError:
import warnings
warnings.warn( """No BSDDB OID Storage available for testing""" )
bsdoidstore = None
def createAgent( oids ):
ports = [161]+range(20000,25000)
for port in ports:
try:
`agentObject = reactor.IOCPReactor.listenUDP(port,` `agentprotocol.AgentProtocol(snmpVersion = 'v2c',agent = agent.Agent(dataStore =` `bisectoidstore.BisectOIDStore(OIDs = oids,),),),)`
`except twisted_error.CannotListenError:`
`pass`
`else:`
`return agentObject, port`
testingOIDs = {
'.1.3.6.1.2.1.1.1.0': 'Some tool out in the field',
'.1.3.6.1.2.1.1.2.0': '.1.3.6.1.4.1.88.3.1',
'.1.3.6.1.2.1.1.3.0': 558566090,
'.1.3.6.1.2.1.1.4.0': "support#somewhere.ca",
'.1.3.6.1.2.1.1.5.0': "NameOfSystem",
'.1.3.6.1.2.1.1.6.0': "SomeHeadEnd, West Hinterlands, Canada",
}
def main(oids=testingOIDs):
agent, port = createAgent( oids )
if __name__ == "__main__":
reactor.IOCPReactor.callWhenRunning( main )
reactor.IOCPReactor.run()

TwistedSNMP does not seem to be designed to work with PySNMP 4.x. Thus you should either use PySNMP 3.x / PySNMP SE or switch to PySNMP 4.x which has its own Twisted binding.