Importing Python module prints docstring of different, unrelated script? - python

I've encountered this issue with two separate modules now, one that I attempted to download myself (Quartz; could probably be the way I installed it, but let's ignore this scenario for now) and another that I installed using pip install (Pandas; let's focus on this one).
I wrote a two-line script that includes just import pandas and print('test'), for testing purposes. When I execute this in the terminal, instead of printing test to confirm the script runs correctly, it prints the docstring for another completely unrelated script:
[hidden]~/Python/$ python3 test.py
Usage: python emailResponse.py [situation] - copy situation response
The second line is a docstring I wrote for a simple fetch script for responding to emails, which is unrelated. What's worse is, if I just envoke Python3 in the terminal, and try import pandas, it'll print that same docstring and take me out of Python3 and back into the terminal shell / bash (sorry if this is not the right verbiage; still learning). The same results happen trying import Quartz as well, but no other modules are impacted (at least, that I'm aware of).
I'm at a complete loss why this might be the case. It was easy enough to avoid using Quartz, but I need Pandas for work purposes and this issue is starting to directly affect my work.
Any idea why this might be the case?

Related

how to make vscode detect / auto reload modules after editing them?

I've seen a few questions asking this, but none of the solutions worked for me.
I am developing a few functions/classes in different modules and have a main.py script that calls everything.
The problem is, when I make a change to a function in another module i.e. module1.py, VSCode does not detect the changes when I call the function in main.py after updating, it's still the older version.
I can get around this by doing something like:
from importlib import reload
reload module1
but this gets old real quick especially when I'm importing specific functions or classes from a module.
Simply re-running the imports at the top of my main.py doesn't actually do anything, I can only do that if I kill the shell and reopen it from the begining, which is not ideal if I am incrementally developing something.
I've read on a few questions that I could include this:
"files.useExperimentalFileWatcher" : true
into my settings.json, but it does not seem to be a known configuration setting in my version, 1.45.1.
This is something Spyder handles by default, and makes it very easy to code incrementally when calling functions and classes from multiple modules in the pkg you are developing.
How can I achieve this in VSCode? To be clear, I don't want to use IPython autoreload magic command.
Much appreciated
FYI here are the other questions I saw, but did not get a working solution out of, amongst others with similar questions/answers :
link1
link2
There is no support for this in VS Code as Python's reload mechanism is not reliable enough to use outside of the REPL, and even then you should be careful. It isn't a perfect solution and can lead to stale code lying about which can easily trip you up (and I know this because I wrote importlib.reload() 😁).

import pytest from Python3.7 instead 2.7

I'm new using python on kubuntu. I'm writing some simple functions and write also tests on pytest to practise test-driven-development (although I know it's wasted on such easy functions, it's just for the sake of practice).
Because I'm beginner, I'm writing the code in an editor and I'm executing it on the terminal, in a next step I'll use an IDE like Thonny. I have installed Python3.7, although Python2.7 seems to be the standard within the system. Nonetheless, the file with the functions works fine. I'm printing some f-strings and it works fine also. The first line of the file is a Shebang which tells the interpreter, to use Python3.7 (#!/usr/bin/env python3.7). However, when I want to execute the tests, I'm writing pytest in console, as indicated by the pytest-introduction. Alas, I get a syntax error, because it seems that pytest is importing Python2.7 which of course doesn't know f-strings.
I verified that pytest is indeed importing Python2.7 by executing the command pytest --version and I was confirmed.
My question is: How can I make pytest to import Python3.7 so the test would pass or at least the Syntaxerror would go away? Replacing the f-string with a normal string makes the test pass, so I'm assuming this is the only problem.
Any help is highly appreciated. Many thanks in advance. I hope, I gave all the relevant informations. If more information is needed, I'll provide that gladly.
You can install pytest with pip3. For most libraries, pip3 defaults to python3.x and pip defaults to python2.x

python power shell issues

Edited to include a specific example
I am learning to use python and work in a windows 10 environment (although on 3 separate computers). I have gotten past the "add python to %PATH% issue" but continue to see odd outcomes when I call simple scripts from powershell compared to running them from the python terminal. For example, I often see that lines of code are duplicated, even simple print statements. A second repeating issue is that some (but not all) modules work fine in the python terminal but can't be found when running a script from powershell, despite using pip install within powershell to install it in the first place.
Rather then seeking help on a specific issue, I'm hoping for some guidance into how powershell and python interact that might help me understand or identify some commonalities in these issues I keep experiencing that are common from within powershell. If thats as simple as redirecting me towards another source even that'd be apprecaited. But my search results always turn up the common issue of powershell not knowing what python is, and needing to add python to the PATH. But as indicated, I have already gotten past this and can get %50 of my python to work from powershell. There must be something else perhaps obvious to others that I'm missing that can help me understand why some things aren't working while others are.
Thanks for taking the time to consider my problem and any advice is greatly apprecaited.
Here is an example from some code I am trying to get working, but in troubleshooting I have taken out a lot of code and am now only running what is shown that defines a dataframe and a print statement. Everything works line by line in python, but when I call the script from powershell, the print statement executes twice.
#!/usr/bin/python3
import pandas as pd
joedata = {'fpr': [0.2,0.4,0.8], 'tpr':[0.9,0.5,0.1]}
joeframe = pd.DataFrame(data=joedata)
print(joeframe)
Concerning modules, you probably have multiple versions of Python installed on your system. Calling pip in the Powershell doesn't mean that it will install for the Python installation which is called by default when you execute Python in the Powershell. So you should try to figure out on which version pip is installing packages and which version is actually used when executing a script.
EDIT: I tested the example that you gave in Powershell and I don't have the problem, it's working fine.
Moreover, for me, without doing any special configuration, in Powershell the commands pip and python refer to the same version of Python. If by default everything is ok, you should consider uninstalling Python and just running the installer again and let it manage the PATH, etc.

Python twisted web server caching and executing outdated code

Background: Working on a web application that allows users to upload python scripts to a server (Twisted web server). The UI provides full CRUD functionality on these python scripts. After uploading a script the user can then select the script and run it on the server and get results back on the UI. Everything works fine...
Problem: ...except when the user edits the python code inline (via the UI) or updates a script by uploading a new script overwriting one which already exists. It seems that twisted caches the code (both old and new) and runs new code sometimes and sometimes runs the old code.
Example: I upload a script hello.py on the server which has a function called run() which does: print 'hello world'. Someone else comes along and uploads another script named hello.py which does: print 'goodbye world'. Then, I go back and execute the run() function on the script 10 times. Half of the times it will say 'hello world' and half of the times it will say 'goodbye world'.
Tried so far: Several different ways to reload the script into memory before executing it, including:
python's builtin reload():
module = __import__('hello')
reload(module)
module.run()
imp module reload():
import imp
module = __import__('hello')
imp.reload(module)
module.run()
twisted.python.rebuild()
from twisted.python.rebuild import rebuild
module = __import__('hello')
rebuild(module)
module.run()
figured that perhaps if we force python to not write bytecode, that would solve the issue: sys.dont_write_bytecode = True
restart twisted server
a number of other things which I can't remember
And the only way to make sure that the most up to date python code executes is to restart twisted server manually. I have been researching for quite some time and have not found any better way of doing it, which works 100% of the time. This leads me to believe that bouncing twisted is the only way.
Question: Is there a better way to accomplish this (i.e. always execute the most recent code) without having to bounce twisted? Perhaps by preventing twisted from caching scripts into memory, or by clearing twisted cache before importing/reloading modules.
I'm fairly new to twisted web server, so it's possible that I may have overlooked obvious way to resolve this issue, or may have a completely wrong way of approaching this. Some insight into solving this issue would be greatly appreciated.
Thanks
T
Twisted doesn't cache Python code in memory. Python's module system works by evaluating source files once and then placing a module object into sys.modules. Future imports of the module do not re-evaluate the source files - they just pull the module object from sys.modules.
What parts of Twisted will do is keep references to objects that it is using. This is just how you write Python programs. If you don't have references to objects, you can't use them. The Twisted Web server can't call the run function unless it has a reference to the module that defines that function.
The trouble with reload is that it re-evaluates the source file defining the module but it can't track down and replace all of the references to the old version of the objects that module defined - for example, your run function. The imp.reload function is essentially the same.
twisted.python.rebuild tries to address this problem but using it correctly takes some care (and more likely than not there are edge cases that it still doesn't handle properly).
Whether any of these code reloading tools will work in your application or not is extremely sensitive to the minute, seemingly irrelevant details of how your application is written.
For example,
import somemodule
reload(somemodule)
somemodule.foo()
can be expected to run the newest version of somemodule.foo. But...
from somemodule import foo
import somemodule
reload(somemodule)
foo()
Can be expected not to run the newest version of somemodule.foo. There are even more subtle rules for using twisted.python.rebuild successfully.
Since your question doesn't include any of the actual code from your application, there's no way to know which of these cases you've run into (resulting in the inability to reliably update your objects to reflect the latest version of their source code).
There aren't any great solutions here. The solution that works the most reliably is to restart the process. This certainly clears out any old code/objects and lets things run with the newest version (though not 100% of the time - for example, timestamp problems on .py and .pyc files can result in an older .pyc file being used instead of a new .py file - but this is pretty rare).
Another approach is to use execfile (or exec) instead of import. This bypasses the entire module system (and therefore its layer of "caching"). It puts the entire burden of managing the lifetime of the objects defined by the source you're loading onto you. It's more work but it also means there are few surprises coming from other levels of the runtime.
And of course it is possible to do this with reload or twisted.python.rebuild if you're willing to go through all of your code for interacting with user modules and carefully audit it for left-over references to old objects. Oh, and any library code you're using that might have been able to get a reference to those objects, too.

Is there a way to see which lines of the code are currently running?

I guess this is kind of a weird question, but let's say you run a code in python that does something computationally expensive, like image processing. Oh I'm running Ubuntu 12.04 by the way. So I'm running a code, and open another terminal and type top to see what's doing what. This is ok as it tells me that python is doing its job, but what if I want to see which line is being run on the code? Is this possible? More importantly is it worth it to get this information? I can post a sample code of some of the processing if necessary
Don't blink, unless your "line of code" is unbelievably slow there is no way for such a thing to be useful. What you probably want is a Python Profiler. I suggest you start looking in http://docs.python.org/2/library/profile.html for info related to profiling your python code.
It usually is very slow but you can trace you code:
python -m trace --count -C . somefile.py ...
More manual but traditional way is logging: you can insert print statements before and after slow operations.
You can find slow places in you code using a profiler.
And you can run your code step by step with a debugger. Just insert import pdb; pdb.set_trace() (or ipdb if you like ipython) before slow operation.
This is the classic use case for a debugger. Have a look at Eclipse with the PyDev plugin, which is an IDE for Python with a useful debugger integration.
For example, a debugger allows you to set breakpoints where the execution will stop in order to let you manually step through the relevant lines of code to see how it goes. At the same time, you can inspect the variables' contents. You will thereby get a better understanding of what is happening, where and why it fails, and so on.
Go and get yourself a debugger!

Categories