We Keep Coding

Need to create a hook for the smartsheet-python-sdk and PyInstaller

Ok there is a lot of information here but it's the result of me searching for days for this solution. I've seen so many people ask this question in various forms with no good answer and I think I'm really close to solving it!
I have a fully functioning python script that uses the smartsheet-python-sdk module but when I run it as an exe bundled with Pyinstaller, it reads the smartsheet objects as strings and I cannot access any of the attributes. The module has a sub-folder called "models" with another subfolder in that called "enums". I figured that I would need to create a hook to import these so I attempted to build one and still no luck. The hooks were read while compiling but they didn't work.
For reference here is the smartsheet package structure.
System info
Everything is the most up-to-date current version:
Python 3.7
Pyinstaller 3.6
Smartsheet 2.86
Operating system: Windows 10
Attempts so far
Someone found a solution to the problem in this post but they didn't provide their solution so it wasn't much help. I have tried adding import statements as suggested here
This is the attempted hook I created for the smartsheet.models:
from PyInstaller.utils.hooks import collect_submodules
hiddenimports = collect_submodules('smartsheet.models')
Some possible reason it's not working
I think it has something to do with the information in the module init files but I am not sure how to deal with it. The init from the main module has this statement:
from .smartsheet import Smartsheet, fresh_operation, AbstractUserCalcBackoff # NOQA
and the init from the models sub-module has statements to import the individual models found in the directory:
from __future__ import absolute_import
# import models into model package
from .access_token import AccessToken
from .account import Account
from .alternate_email import AlternateEmail
from .attachment import Attachment
from .auto_number_format import AutoNumberFormat
# This continues for other models
So I think that I need to somehow mimic this model import statement in my hook file but I don't know how to do it.
Code & Error Message:
It creates the main smartsheet object ok as it does not reference any of the sub-module items:
# Creates a smartsheet object for an account with an api access token
ss = smartsheet.Smartsheet(a_token)
But anything that references submodules within this object fails
ss.Sheets.get_sheet(residential_id)
This is the error message I get when I run the program:
ImportError! Could not load api or model class Users
# print statement I added to show the string object that is supposed to be a smartsheet object
<smartsheet.smartsheet.Smartsheet object at 0x00000292D4232408>
Exception type: AttributeError
Exception message: 'str' object has no attribute 'get_sheet'
Stack trace:
File: sum_report.py
Line: 516
Function nameName: main
Message:
File: snow_functions.py
Line: 437
Function nameName: connect_smartsheet
Message:

I was running in to same issue. What I have found is that you have to add all the individual modules you need as hidden imports.
Open you .spec file that was created. If you have previous used pyinstaller script.py, look for the file script.spec in your script directory. Open it and modify the section:
hiddenimports=[
'smartsheet.models'
'smartsheet.sheets',
'smartsheet.search',
'smartsheet.users'
]
Then run pyinstaller script.spec to use the spec file with the hidden imports. Try running your package again. You may need to added additional modules to the hidden imports array if your script fails again (just look at the module referenced in the error.)
I finally got mine working after adding models, sheets, and search.

Does a package 'see' itself in __init__.py?

I have a flask app with the root folder called project_folder.
A code snippet from the __init__.py file of this project_folder package:
#jwt.token_in_blacklist_loader
def check_if_token_in_blacklist(decrypted_token):
jti = decrypted_token['jti']
return project_folder.Model.RevokedTokenModel.is_jti_blacklisted(jti)
from project_folder.Controller.root import root
from project_folder.Controller import auth_controller
from project_folder.Controller import item_controller
Now the interesting thing is, that the project_folder package naturally has other smaller packages itself, which I'm importing to use them (for REST resources in this example). These are the last 3 lines, nothing throws an error so far.
But, if you take a look at the annotated function (in this example it always runs before some kind of JWT Token is being used), I am returning some inner package's function. Now when the logic truly runs this part the code breaks:
PROJECT_ROUTE\project_folder\__init__.py", line 38, in check_if_token_in_blacklist
return project_folder.Model.RevokedTokenModel.is_jti_blacklisted(jti)
NameError: name 'project_folder' is not defined
After thinking about it, it seems understandable. Importing from project_folder does import from the __init__.py file of the package, which is the actual file the interpreter currently is. So removing the package name prefix form the
return project_folder.Model.RevokedTokenModel.is_jti_blacklisted(jti)
to
return Model.RevokedTokenModel.is_jti_blacklisted(jti)
does not throw an error anymore.
The question is: Why is it only a problem inside the callback function and not with the last 3 imports?

This has to do with circular imports in python. Circular import is a form of circular dependency, created at the module import level.
How it works:
When you launch your application, python keeps a register (a kind of table) in which it records all the imported modules. When you call somewhere in your code a module, python will see in its registry if it has already been registered and loads it from there. You can access this registry via sys.module, which is actually a dictionary containing all the modules that have been imported since Python was started.
Example of use:
>>> import sys
>>> print('\n'.join(sys.modules.keys()))
So, since Python is an interpreted language, reading and execution of code is done line by line from top to bottom.
In your code, you put your imports at the bottom of your __init__.py file.
While browsing it, when python arrives at the line return project_folder.Model.RevokedTokenModel.is_jti_blacklisted(jti), it will look if the module exists in its register. Which is clearly not yet the case. That's why he raises an NameError: name 'project_folder' is not defined exception.

Run python script from another python script on linux

I'm using python 2.7 and ubuntu 16.04.
I have a simple REST server on python: file server_run.py in module1 which is importing some scripts from module2.
Now I'm writing an integration test, which is sending POST request to my server and verify that necessary action was taken by my server. Obviously, server should be up and running but I don't want to do it manually, I want to start my server (server_run.py which has also main method) from my test: server_run_test.py file in module3.
So, the task sounds very simple: I need to start one python script from another one, but I spent almost the whole day. I found a couple of solutions here, including:
script_path = "[PATH_TO_MODULE1]/server_run.py"
subprocess.Popen(['python', script_path], cwd=os.path.dirname(script_path))
But my server is not coming up, throwing the error:
Traceback (most recent call last):
File "[PATH_TO_MODULE1]/server_run.py", line 1, in <module>
from configuration.constants import *
File "[PATH_TO_MODULE2]/constants.py", line 1, in <module>
from config import *
ModuleNotFoundError: No module named 'config'
So, it looks like when I'm trying to start my server in subprocess it doesn't see imports anymore.
Do you guys have any idea how can I fix it?

Eventually, the solution was found, 2 steps were taken:
1. In each module I had an empty __init__.py file, it was changed to:
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
__version__ = '${version}'
2. Instead of using the following syntax:
from configuration.constants import *
from configuration.config import *
config and constants were imported as:
from configuration import constants,config
and then we are using reference to them when need to get some constant.
Thanks everyone for looking into it.

Rather than running it using os module try using the import function. You will need to save in the same dictionary or a sub folder or the python installation but this seems to be the way to do it. Like this post suggests.

Load module from python string without executing code

I'm trying to do something similar with: Load module from string in python
Sometimes I read source code provided by the user:
module = imp.new_module('AppModule')
exec_(s, module.__dict__)
sys.modules['AppModule'] = module
return module
And sometimes import from a file
module = __import__(module_name, fromlist=_fromlist)
return module
These modules are loaded to map objects and it'll be used inside a IDE.
My problem is: If there is any method call outside if __name__ == '__main__': this code is being executed and can interact with the IDE.
How can I import modules ignoring methods from being executed?

The process of importing a module requires that its code be executed. The interpreter creates a new namespace and populates it by executing the module's code with the new namespace as the global namespace, after which you can access those values (remember that the def and class statements are executable).
So maybe you will have to educate your users not to write modules that interact with the IDE?

Writing to a file from a python module

I am currently trying to debug an app that I have not written myself. I have narrowed the problem down to a particular method that is imported from a module outside of the current script. I would like to step through this module by writing to a file at each step, however it doesn't seem to work.
I have a version of the app that is running correctly and when I write to a file from within the module, the script runs fine but no file is created. Am I missing something here?
Example
Script that I am debugging
from module import method
example code ...
method(data) --- where error occurs
more code ...
module.py
def method(data):
file = open('filetowrite.txt','w')
file.write('something ....')
file.close()

Do not reinvent the wheel, use standard logging module.
Since you mentioned "I would like to step through this module" that's how you can do it with again standard pdb module.

Your code for file writing looks fine. I would look at the path (the location) where the file is being written to.
Check out:
How to get full path of current file's directory in Python?