I am trying to compile my python program into a singular executable. The libraries I am using are: Pandas, numpy, ConfigParser and cx_oracle. I am also running in an Anacondas environment, using python 3.9. The file structure is as follows:
Code
Calculation
calualtions.py
transformations.py
Db
config_parse.py
db1.py
db2.py
save_to_db.py
ifAdded.py
file_generation.py
I also have a config file, used to set input and output local directories, as well as responsible for getting the database connection string. The script is tested locally, and works as intended, listening to change in a selected directory, making calculations, using some of the database data and records the result in a local directory. The entry point is ifAdded.py at the root of the project.
I tried using PyIstaller, however ran into a problem, after the project is already compiled. The script runs, however suddenly exits with a KeyError, at a spot where the config is processed. My guess is the only reason this could be, is because ConfigParser library is not compiled correctly with the project. I tried passing this as a hidden import option, but it did not help. How can I bundle this project correctly with the config file, being outside the proejct, so it can be adjusted after compilation?
Related
I was trying to debug a modified python script for The Sims 4, running the mod's code in the game context and trying to access the variables using the pydevd-pycharm package and a python debug server, both provided by PyCharm Pro. Although I followed the necessary instructions and settings (described below), I was still unable to successfully debug.
The method I used to perform the debug attempt is as follows:
Inside the PyCharm Pro installation files (version 2022.3.2), I took a copy of the file “pydevd-pycharm.egg” and changed the extension of that file to “.zip” so that I could edit it;
Inside the python installation files (version 3.7.0), I made a copy of the “ctypes” directory and inserted it inside the “pydevd-pycharm.zip” file created in the previous step; this “.zip” file was inserted into the Mods directory, like any other mod;
I configured a python debug server in Pycharm Pro and created a command for The Sims 4 that contained the code to connect to the debug server. The command was as follows:
import sims4.commands
#sims4.commands.Command('start.debug', command_type=sims4.commands.CommandType.Live)
def startdebugging(_conection=None):
import pydevd_pycharm
pydevd_pycharm.settrace('localhost', port=5678, stdoutToServer=True, stderrToServer=True)
I also inserted the command as any mod in the Mods directory and started the python debug server (which was waiting for a connection); I minimized PyCharm Pro and started the game; already inside the game, I started the command start.debug; if everything had gone well, I would be debugging the mod script now.
(If you want to know more about the method I used, the tutorial link is: https://youtu.be/RBnS8m0174U)
In conclusion, i want to know why the method I ran ended up not working and, if possible, suggestions on how I can modify the debug method so that it finally works.
We have a desktop app written in Python that allows for user plugins for customization. There is a folder for the user to add plugins into, and then upon loading of the app, it looks there for plugins. Each plugin has a manifest that lists all the files it needs to work, and these are imported using "importlib". We are also packaging this Python app as a .exe using Pyinstaller, which bundles it with a Python runtime so users who don't already have Python already installed can use it.
We have noticed that the process of looking for and loading plugins makes the initial load of the software very slow, far beyond what it was when we didn't have a plugin system and had all files manually spelled out and imported within the code. We don't know if it's importlib itself or an interaction with Pyinstaller that's creating this slowness, but we have noticed that after the .exe has been launched once and closed, subsequent re-launches without restarting the computer are much faster. However, after restarting the computer, it goes back to being super slow. We suspect that the first load of the plugins by importlib triggers their compilation into .pyc files, which are used from then on. However, once the computer is restarted, although the .pyc files remain, we suspect that the .py files are for some reason re-compiled (despite not having changed).
Is there a way to avoid this, so that the overhead we see on initial plugin load is restricted only the very first time the .exe is run after copying a new plugin to the plugins folder?
A little bit of context:
I have made a password manager in python
I then packed it into a single executable using:
pyinstaller --onefile password_manager.py
Okay first I want to specify that the script works properly when I run it through the password_manager.py file, the executable version also runs properly and when I e.g. create a user and I save the data in a new json file. I'm able to access it on runtime but when I actually close the shell and the program stops running it's like the data has all been lost and or deleted. I'm looking for a way for the required data to remain on my drive like they do when I run password_manager.py on its own.
P.S.
I can view hidden files in my computer too.
P.S.S.
The link to the code is here: code
My file structure when I run it through password_manager.py
Here is the executable showing some random Data I recently added without closing and restarting the program. These information was supposed to be stored in /data/users.json and /data/password (and is stored there when I just use password_manager.py) but when I actually exit the application they are not present in any way when I rerun the executable(neither the directories that were supposed to be created nor the files).
Simple replication:
#mkdir.py
import os
os.mkdir("stack")
shell:
pyinstaller --onefile mkdir.py
I also sadly couldn't find a way to include the executable in this question but you can replicate it with the given command.
When I am adding new functions to a file I can't import them neither if I just run the script in terminal or if I launch ipython and try importing a function there. I have no .pyc files. It looks as if there is some kind of caching going on. I never actually faced such an issue even though have been working with various projects for a while. Why could it happen?
What I see is the following: I launch ipython and the functions that were written long time ago by other programmers can be imported fine. If I comment them out and save the file, they still can be imported without any issues. If I write new functions they can't be imported.
The directory is git directory, I cloned the repo. Then the new branch was created and I switched to it. Python version is 3.7.5, and I am working with virtual environment that I created some time ago which I activated with source activate py37.
I don't know whether its important but I have an empty __init__.py in the folder where script is located.
The code (I don't think its relevant, but still):
import hail as hl
import os
class SeqrDataValidationError(Exception):
pass
# Get public properties of a class
def public_class_props(cls):
return {k: v for k, v in cls.__dict__.items() if k[:1] != '_'}
def hello():
print('hello')
public_class_props is an old function and can be imported, but hello - can't.
assuming that you've successfully cloned the project into your machine, If it is a problem with importing functions or methods from a different .py file, please check the points below.
check your working directory, whether you're in the same directory where the .py file/module with functions/methods exist.
once you import any function/method from a module, even you comment out the function/method and save the .py file, it won't affect the already imported functions as long as you re-import it.
as long as it is a problem of importing from your own .py files, the virtual environment has nothing to do with that.
EDITED:
check this link, it may give you some information about how caching works while importing python modules.
The issue was that PYTHONPATH was set to a wrong folder. We had two folders with the project: old and new, similarly named, with identical project structure (but different file contents) and PYTHONPATH was set to the old project.
I need to pack an application written in Python.
I can't use PyInstaller because I can't have a .exe or anything like that, I need pure python scripts being executed (I have Popen in my code calling some of my scripts and passing parameters);
I need it to work cross-plataform (I KNOW that to do so, I need to generate the package in each system because a package generated on Linux won't work in a Mac and vice-versa);
Since I can't really pack it, my idea is to have a folder called modules and have all the dependencies inside it, then, in my code I would just point the imports to this folder. I would zip the entire project and ship it, the user would just unpack and run it.
The problem is:
How to not only download the package in a specific folder but also install it there? (I can't alter my user's environment, this is a MUST);
How to direct my imports to this local folder? I think I can do something like import modules.numpy for example, but then, numpy have dependencies of its own... how to make sure it will look into my custom folder?
My scenario:
I have a requirements.txt for my project; I have several local files that are used inside the project; I have a Popen that calls one of my file.py; I am using Python3;
I can't use virtualenv because I am using wxPython and there's a conflict between it and virtualenv (no idea why - something related with main thread...)