We Keep Coding

Kedro cannot find run

As a part of upgrading Kedro from 0.16.2 to 0.17.3 in our organization, I've made changes to Kedro related files in our codebase based on Kedro starter pyspark-iris on 0.17.3.
Now I get an error of Error: No such command 'run' on kedro run.
setup.py
from setuptools import find_packages, setup
entry_point = "kedro-project = kedro-package.__main__:main"
# get the dependencies and installs
with open("requirements.txt", "r", encoding="utf-8") as f:
# Make sure we strip all comments and options (e.g "--extra-index-url")
# that arise from a modified pip.conf file that configure global options
# when running kedro build-reqs
requires = []
for line in f:
req = line.split("#", 1)[0].strip()
if req and not req.startswith("--"):
requires.append(req)
setup(
name="kedro-package",
version="0.1",
packages=find_packages(exclude=["tests"]),
entry_points={"console_scripts": [entry_point]},
install_requires=requires,
extras_require={
"docs": [
"sphinx~=3.4.3",
"sphinx_rtd_theme==0.5.1",
"nbsphinx==0.8.1",
"nbstripout==0.3.3",
"recommonmark==0.7.1",
"sphinx-autodoc-typehints==1.11.1",
"sphinx_copybutton==0.3.1",
"jupyter_client>=5.1.0, <6.0",
"tornado>=4.2, <6.0",
"ipykernel~=5.3",
]
},
)
main.py
from pathlib import Path
from kedro.framework.project import configure_project
import logging
from .cli import run
def main():
package_name = str(Path(__file__).resolve().parent.name)
logging.getLogger(__name__).info(f"package name is: {package_name}")
configure_project(package_name=package_name)
run()
if __name__ == "__main__":
main()
and cli.py is at the same level as main.py which are directly inside the package (altered to kedro-package here for anonymity)
This only happens when performing kedro run on the EMR. When we run locally we don't see that error. Rather it errors out because it can't connect to S3, which is expected.
Additionally, I've tried running

Compile a Python project Windows

I have the following directory structure to my python project:
eplusplus/
|
|
----__main__.py
----model/
----exception/
----controller/
----view/
The directories: model, exception, controller and view each one has its
__init__.py. When I run the program at my machine I always use this following command: py -m eplusplus. But when I tried to use py2exe or pytinstaller the the points to: permission denied. For what I found, this is because its a directory I trying to compile, but when I compiled the __main__.py it compiled normally, but when I try to execute it says: Error! No eplusplus module founded!
I have no setup.py file and I don't know how they worked.

After some very intensive research and error and try I succeeded by doing this:
I added an empty __init__.py at the eplusplus folder
Out of the eplusplus folder, I had to write a compilation.py file (the file doesn't necessary must have this) to include all libraries I was using (I will post the file at the end of this answer)
Finally, at the PowerShell, all I have to type was py compilation.py py2exe
Thanks for all that tried to help me!
compilation.py file:
#To compile we need to run: python compilation.py py2exe
from distutils.core import setup
from glob import glob
import os
import py2exe
import pyDOE
VERSION=1.0
includes = [
"sip",
"PyQt5",
"PyQt5.QtCore",
"PyQt5.QtGui",
"PyQt5.QtWidgets",
"scipy.linalg.cython_blas",
"scipy.linalg.cython_lapack",
"pyDOE"
]
platforms = ["C:\\Python34\\Lib\\site-packages\\PyQt5\\plugins" +
"\\platforms\\qwindows.dll"]
dll = ["C:\\windows\\syswow64\\MSVCP100.dll",
"C:\\windows\\syswow64\\MSVCR100.dll"]
media = ["C:\\Users\\GUSTAVO\\EPlusPlus\\media\\title.png",
"C:\\Users\\GUSTAVO\\EPlusPlus\\media\\icon.png"]
documents = ["C:\\Users\\GUSTAVO\\EPlusPlus\\docs\\"+
"documentacaoEPlusPlus.pdf"]
examples = ["C:\\Users\\GUSTAVO\\EPlusPlus\\files\\"+
"\\examples\\baseline2A.idf",
"C:\\Users\\GUSTAVO\\EPlusPlus\\files\\"+
"\\examples\\vectors.csv",
"C:\\Users\\GUSTAVO\\EPlusPlus\\files\\"+
"\\examples\\BRA_SC_Florianopolis.838970_INMET.epw"]
datafiles = [("platforms", platforms),
("", dll),
("media", media),
("docs", documents),
("Examples", examples)]
imageformats = glob("C:\\Python34\\Lib\\site-packages\\PyQt5\\"+
"plugins\\imageformats\\*")
datafiles.append(("imageformats", imageformats))
setup(
name="eplusplus",
version=VERSION,
packages=["eplusplus"],
url="",
license="",
windows=[{"script": "eplusplus/__main__.py"}],
scripts=[],
data_files = datafiles,
options={
"py2exe": {
"includes": includes,
}
}
)

Python shove module not working after py2app build on mac

I am using the shove module for object persistence. It works fine but when I attempt to build the app using py2app, it ceases to work. The following code works normally except after freezing using py2app:
import os
from shove import Shove
if __name__ == '__main__':
home=os.path.expanduser("~")
path = os.path.join(home, 'testdb')
uri = 'file://{0}'.format(path)
print path
print uri
db = Shove(uri)
print db.keys()
db['1'] = 'dog'
db['2'] = 'cat'
db.sync()
db.close()
The setup.py file contains the following:
from setuptools import setup
OPTIONS = dict(
argv_emulation = True,
includes=['future_builtins','concurrent.futures']
)
DATA_FILES = []
setup(
app='highlightdb.py',
data_files=DATA_FILES,
options={'py2app': OPTIONS},
setup_requires=['py2app'],
)
The traceback I am getting is:
highlightdb[68150]: db = Shove(uri)
highlightdb[68150]: File "shove/core.pyc", line 23, in __init__
highlightdb[68150]: File "shove/_imports.pyc", line 46, in store_backend
highlightdb[68150]: KeyError: 'file'
What could the problem be?

I found the problem is that shove functionality is largely implemented using plugins, which are loaded using pkg_resources.iter_entry_points. Unfortunately py2app doesn't support iter_entry_points. Nevertheless I found this, which may be a possible workaround.

Did you restarted Python after installing Shove?
The same KeyError: 'file' happens if you don't restart: you can import the library, but Shove can't load any modules

Scons and external directories

I am working with a scons build system for building shared libraries. Everything builds fine up to this part but now I am having some difficulties.
I cannot get scons to move the output file to a directory outside of the scons folder structure.
The only thing that I've sen on SO about it is this question here
If I try to write a python function to do it, the function runs first before the build finishes so I get some file not found errors.
How can I get scons to move a file to a directory outside of the directory where SConstruct file is defined?
import os
import shutil
Import('env')
android_files = [
'os_android.cpp',
'pic_android.cpp',
'file_access_android.cpp',
'dir_access_android.cpp',
'audio_driver_opensl.cpp',
'file_access_jandroid.cpp',
'dir_access_jandroid.cpp',
'thread_jandroid.cpp',
'audio_driver_jandroid.cpp',
'ifaddrs_android.cpp',
'android_native_app_glue.c',
'java_glue.cpp',
'cpu-features.c',
'java_class_wrapper.cpp'
]
env = env.Clone()
if env['target'] == "profile":
env.Append(CPPFLAGS=['-DPROFILER_ENABLED'])
android_objects=[]
for x in android_files:
android_objects.append( env.SharedObject( x ) )
prog = None
abspath=env.Dir(".").abspath
pp_basein = open(abspath+"/project.properties.template","rb")
pp_baseout = open(abspath+"/java/project.properties","wb")
pp_baseout.write( pp_basein.read() )
refcount=1
name="libpic"+env["SHLIBSUFFIX"]
dir="#bin/"+name
output=dir[1:]
ANDROID_HOME=os.environ.get('ANDROID_HOME')
ant_build=Dir('.').abspath+"/java/"
ANT_TARGET=ant_build+'local.properties'
ANT_SOURCES=ant_build+'build.xml'
ANDROID_HOME=os.environ.get('ANDROID_HOME')
ANT_COMMAND='ant release -Dsdk.dir='+ANDROID_HOME+' -f $SOURCE'
for x in env.android_source_modules:
pp_baseout.write("android.library.reference."+str(refcount)+"="+x+"\n")
refcount+=1
pp_baseout.close()
pp_basein = open(abspath+"/AndroidManifest.xml.template","rb")
pp_baseout = open(abspath+"/java/AndroidManifest.xml","wb")
manifest = pp_basein.read()
manifest = manifest.replace("$$ADD_APPLICATION_CHUNKS$$",env.android_manifest_chunk)
pp_baseout.write( manifest )
for x in env.android_source_files:
shutil.copy(x,abspath+"/java/src/com/android/pic")
for x in env.android_module_libraries:
shutil.copy(x,abspath+"/java/libs")
env.SharedLibrary("#bin/libpic",[android_objects],SHLIBSUFFIX=env["SHLIBSUFFIX"])
env.Command('#platform/android/java/libs/armeabi/libpic_android.so', dir, Copy('platform/android/java/libs/armeabi/libpic_android.so', output))
apk = env.Command(ANT_TARGET, source=ANT_SOURCES, action=ANT_COMMAND)
#env.Install('[install_dir]', apk)
#Cannot get this part to work, tried env.Install() but donno
#if env['target'] == 'release_debug'
#copy 'platform/android/java/bin/Pic-release-unsigned.apk' to templates as 'android_debug.apk'
#else:
#copy 'platform/android/java/bin/Pic-release-unsigned.apk' to templates as 'android_release.apk'

You should consider using the SCons Install Builder. This is the only way to install targets outside of the SCons project hierarchy.

py2exe bundle_files=1 or 2 fails

My application uses QGraphicsPixmapItem, and to make it able to load jpeg files I've placed qjpeg4.dll under 'imageformats' subdirectory in the 'dist' directory.
It works, but only as long as 'bundle_files' option is set to 3.
If I set it to 1 or 2, qt4 (pyqt4) is no longer able to find needed dlls, and so QGraphicsPixmapItems is not visible.
setup.py:
from distutils.core import setup
import py2exe
setup(
options = {'py2exe': {'bundle_files': 1}},
description = "",
name = "name",
windows = ["mainwindow.py"],
zipfile=None,
)

You should be able to convince py2exe to include the dll by using:
setup(
# other options,
data_files=[('imageformats', 'qjpeg4.dll'),
#other options
)
For future reference, data_files should look like this (afaik):
data_files = [ (dir1, [file1, file2, ...]), (dir2, [file3, file4, ...]), ...]
EDIT 1: You could try using a directory structure like this (source):
yourapp.exe
[qt.conf] (optional? see lower down)
plugins/
imageformats/
qjpeg4.dll
And if that doesn't work, here suggests using a qt.conf file that looks like this:
[Paths]
Plugins = <directory containing the imageformats directory>
Which apparently should work fine so long as the core dll QtCore4.dll has been included correctly (as it needs this .dll to interpret your qt.conf file).