I am running a python script that calls local modules (with import), and since few days it became so slow.
I could not find the reason why it is so, neither on other Github posts or Google, that's why I am posting this.
Providing code won't be of much help, but here is the import that poses a problem:
import latplan
where latplan is this library
But again, this import did not pose any problem at all before...
import statements can be very slow in Python because they are allowed to execute arbitrary code as side effects; they don't just scoot for class and def keywords.
Now, as with any performance issue, it's hard to guess what exactly is taking so long without profiling. Luckily there's a builtin, specialised profiler for import time:
python -X importtime -c "import latplant"
I recommend using tuna to visualise the reports.
Related
I have a question regarding the python import part.
Every time I am running my script python has to load up a bunch of import stuff.
Is there a way to skip this part in the way you can do that with jupyter notebook?
Thanks for your help!
I tried using Jupyter notebook but that is actually not my favorite coding environment.
When you run your script, add "python -O". It will perform an optimized build. However it will disable some features like the built-in debug var.
You can always use the "import time" library to measure the time taken by each import and figure out if what you're trying is effective.
Hope this will help you
When I run a Python script in VS Code, it delays execution by more than a second if, for example, Pandas or Numpy are part of the script's import statement.
If only libraries from the Python Standard Library are used in the imports, the script will start immediately.
A second doesn't sound like a lot, but it is to me because I've used Spyder so far, where the same script starts immediately without spending any noticeable time on imports. I was wondering if this is normal in VS Code or if there are configuration parameters to speed up import time.
Edit
A minimal example would be a script with content
import collections
import pandas
print("a string")
which in my opinion should only take a few milliseconds (not noticeable) for complete processing after clicking the "run" button. Without the pandas import, it actually does.
I think this is an important aspect, because slow "import speeds" hinder the unit-testing workflow.
pandas invokes numpy, and both of those are very large packages with many C DLLs. It takes many seconds for them to load. Once the DLLs have been loaded into Windows file cache, it should load quicker until they age out. It's just a fact.
I'm trying to understand the backend of the request and sys, and os modules.
I've seen or heard somewhere that python modules are pre-written python codes made to be called later in other scripts with the import command.
I was just curious as to where the module source codes are located so I can read them to better understand some things in python.
I think you've got the right mentality. I've become such a better developer just by taking the time to step through the source code.
You can find all of the Python built-in modules here.
You may notice that some of the built-in modules like os and sys are written in C. I’d recommend checking this question out for a better understanding.
In regards to requests, it's a third-party library built around urllib.
I apologize in advance if my question is badly formulated, for I don't know if what I need makes any sense.
I'm currently working on a c# project where I need to run several time the same python script from inside the program, but with different arguments each time.
For this, I'm not using IronPython, but the ProcessStartInfo class, for I understood that IronPython has some problem with certain packages I use. But this can change.
My problem is that although the python script is small and fast, it needs to import first a lot of packages, and this takes a lot of time. And therefore, my code is very slow, while 90% of the time is used to import python packages.
I can't work around the problem by running this python script a single time with many arguments.
So is there a way to "open a permanent python console" from c#, where I could import everything once, then run the small script with my first argument, get the result back in c#, then run the script a second time etc .... ? Or any other way to optimize this ?
Thanks for your help,
Astrosias.
I've encountered this issue with two separate modules now, one that I attempted to download myself (Quartz; could probably be the way I installed it, but let's ignore this scenario for now) and another that I installed using pip install (Pandas; let's focus on this one).
I wrote a two-line script that includes just import pandas and print('test'), for testing purposes. When I execute this in the terminal, instead of printing test to confirm the script runs correctly, it prints the docstring for another completely unrelated script:
[hidden]~/Python/$ python3 test.py
Usage: python emailResponse.py [situation] - copy situation response
The second line is a docstring I wrote for a simple fetch script for responding to emails, which is unrelated. What's worse is, if I just envoke Python3 in the terminal, and try import pandas, it'll print that same docstring and take me out of Python3 and back into the terminal shell / bash (sorry if this is not the right verbiage; still learning). The same results happen trying import Quartz as well, but no other modules are impacted (at least, that I'm aware of).
I'm at a complete loss why this might be the case. It was easy enough to avoid using Quartz, but I need Pandas for work purposes and this issue is starting to directly affect my work.
Any idea why this might be the case?