all.
I recently started working with Jenkins, in an attempt to replace cronjob with Jenkins pipeline. I have really a bit knowledge of programming jargon. I learned what I learned from questions on stackoverflow. So, if you guys need any more info, I would really appreciate if you use plain English.
So, I installed the lastest version of Jenkins and suggested plugins plus all the plugins that I could find useful to python running.
Afterwards, I searched stackoverflow and other websites to make this work, but all I could do was
#!/usr/bin/env python
from __future__ import print_function
print('Hello World')
And it succeeded.
Currently, Jenkins is running on Ubuntu 16.04, and I am using anaconda3's python (~/anaconda3/bin/python).
When I tried to run a bit more complicated python code (by that I mean import pandas), it gives me import error.
What I have tried so far is
execute python script build: import pandas - import error
execute shell build: import pandas (import pandas added to the code that worked above)
python builder build: import pandas - invalid interpreter error
pipeline job: sh python /path_to_python_file/*.py - import error
All gave errors. Since 'hello world' works, I believe that using anaconda3's python is not an issue. Also, it imported print_function just fine, so I want to know what I should do from here. Change workspace setting? workdirectory setting? code changes?
Thanks.
Since 'hello world' works, I believe that using anaconda3's python is not an issue.
Your assumption is wrong.
There are multiple ways of solving the issue but they all come down to using the correct python interpreter with installed pandas. Usually in ubuntu you'll have at least two interpreters. One for python 2 and one for python 3 and you'll use them in shell by calling either python pth/to/myScript.py or python3 pth/to/myScript.py. python and python3 are in this case just a sort of labels which point to the correct executables, using environmental variable PATH.
By installing anaconda3 you are adding one more interpreter with pandas and plenty of other preinstalled packages. If you want to use it, you need to tell somehow your shell or Jenkins about it. If import pandas gives you an error then you're probably using a different interpreter or a different python environment (but this is out of scope here).
Coming back to your script
Following this stack overflow answer, you'll see that all the line #!/usr/bin/env python does, is to make sure that you're using the first python interpreter on your Ubuntu's environment path. Which almost for sure isn't the one you installed with anaconda3. Most likely it will be the default python 2 distributed with ubuntu. If you want to make sure which interpreter exactly is running your script, instead of 'Hello World' put inside:
#!/usr/bin/env python
import sys
print(sys.executable) # this line will give you the exact path to the interpreter
print(sys.version) # this one will give you the version
Ok, so what to do?
Well, run your script using the correct interpreter. Remove #!/usr/bin/env python from your file and if you have a pipeline, add there:
sh "/home/yourname/anaconda3/bin/python /path_to_python_file/myFile.py"
It will most likely solve the issue. It's also quite flexible in the sense that if you ever want to use this python file on a different machine, you won't have your username hardcoded inside.
Related
module path lost in multiprocessing spawn (ModuleNotFoundError)
The so-called solution of inserting sys-path above the importing of the module does not work for me.
Here is my main.py
import multiprocessing
from testing import customfunction
customfunction(1,2,3)
if __name__ == "__main__":
process = multiprocessing.Process(target=customfunction)
process.start()
process.join()
print("DONE")
The main.py works fine up to process.start()
This means customfunction has been imported properly
Here is my testing.py
import random
def customfunction(size, test, hello):
random.seed(size)
print(random.random())
return random.random()
Both main.py and testing.py are in the same folder. A separate folder with an init.py file did not work as well.
I get this error:
from testing import customfunction
ModuleNotFoundError: No module named 'testing'
I can not wrap my head around why does the process created not retain the system pathing in order to import the file. If i place the multiprocessing creation in customfunction it doesn't work either, the same error occurs.
The link I shared at the top does not work for me as well.
Thank you for taking the time to read. If you believe this is a duplicate of another question, please link it and explain, I am new to python.
EDIT:
I am using Windows 10 as my OS
I have installed Spyder 4.1.4 using Anaconda Navigator, Python 3.7.7.
I installed using a executable package.
I have tested this code on VS Code as well.
I am running this via the two IDEs mentioned(E.G VS Code Powershell console and Spyder's Python Console by clicking run)
I generally currently believe it is an issue specific to my computer, and I'll like to know if its replicable in other Windows Systems and whether or not the linked "solution" in the first line works. With that I may be able to pinpoint my errors
This should work if you invoke multiprocessing.Process(target=customfunction, args=(1,2,3)) instead. I can't think of a reason why this would not work on Linux.
Can you update your question and provide the following information?
What OS are you using?
What version of Python are you running, and how was it installed?
How are you running main.py (e.g. from the command line, from an IDE, etc.)?
Any other details about your system configuration that might help others answer your question?
No multiprocessing print outputs (Spyder)
The solution mentioned here seems to solve my problem, I never expected either VS Code or Spyder's Console to have an issue with multiprocessing, but running the code in an external system terminal works.
Thank you to Melih Elibol for helping me think more clearly about the problem, I am new to python.
I'm running Python code in VS Code (1.28.2, with the Python, and Python Extension Pack extensions) and am using the 'Run Selection/Line in Python Terminal' (Shift+Enter) functionality to just run selected code in the Python terminal.
This has always worked well, but today I'm getting a new line added between each line of code in the terminal, i.e. if I ran:
import heapq
import pickle
the output in the terminal would be:
>>>import heapq
>>>
>>>import pickle
At first, this just seems like an annoyance, but any for loops or functions now come out with an indentation error...so essentially I can't successfully run any code.
I've tried re-installing VS Code as well as installing an older version of VS Code but all give the same problem.
It's so odd because it was working fine and then all of a sudden it changed. The only thing I can think of that has possibly changed is I installed the JSON Tools extension, but I don't believe this would change anything within Python (and I've tried uninstalling this, and not loading it again when reinstalling VS Code from scratch)
Any help would be greatly appreciated!
It's a bug that will be fixed in the 2018.9.1 release of the Python extension.
I have a big C++ module with Python 3 bindings using Boost.Python, that I compile to a .so file using CMake on macOS.
When I try to import it in the REPL, everything seems to work fine:
>>>import myModule
>>>
However, as soon as I run the import statement, the famous rocket icon of Python shows up in the Dock and stays there jumping for some minutes and stops after. Obviously then, I cannot access any of the functions defined in my module, so the import looks fine but doesn't actually do anything.
I tried looking in the Console and saw that whenever I import myModule, I get two launchservicesd[83]: SecTaskLoadEntitlements failed error=22.
It brought me to this and that related questions but I can't find what the exact problem is.
The C++ module is huge so I just can't look at the code and find the problem, thus I'm asking for any hints about at least how to debug that problem.
I can suggest the following steps:
Try to import that module though local python session. So, run interactive python interpreter, and 'import myModule'.
If bad, try to check:
are python version, with which myMoudle was linked with, is similiar to used interpreter
check that build architectires are the same
check that you can load even simple boost.python example module
If ok, check that you have correctly set up module search path in your python code.
When I try to import time I get : No module named time
I have tried other time modules(datetime and timeit) and they work fine. I decided to check my installation and I can't find time.py anywhere. I checked the Lib, Scripts, libs and include folders, but can't find it anywhere.
Anyone know what I can do to fix this? Maybe download the .py and put it in Lib myself?
I am using Python 3.3.5 with PyCharm IDE. Only extra scripts I've installed is EasyInstall and PRAW.
The import does work. When PyCharm said No module named time, I assumed I would get a compiler error and started trying to fix it.
However when I eventually just ran the code it worked fine. I expect PyCharm doesn't detect the time module as it's a dll and not a py as noted by Martijn in the comments. This is on PyCharm Community Edition 4.0.4.
I tried playing with virtualenv and a host of other things, but I eventually went to Preferences -> Build, Execution, Deployment -> Console -> Python Console, and in the "starting script" box, I added two lines:
sys.builtin_module_names.append('sys')
sys.builtin_module_names.append('time')
This got rid of errors I had with both sys and time. Once I did that, I even get autocomplete for both of those modules... weird.
I'm using simplejson to get data from the New York Time API. It works when I run the file through the terminal with the command "python test.py" but not when I run through TextMate using command + R. I'm running the exact same file. Why is this?
I am running Snow Leopard 10.6.4, TextMate 1.5.10, and Python 2.6.4.
Edit: Sorry for forgetting to include this: by "doesn't work," I mean it says "No module named simplejson". I also noticed that this happens for PyMongo as well ("No module named pymongo").
What doesn't work? You should provide more information like error messages and what-not. However, I assume that the version of python is different, and simplejson isn't on your PYTHONPATH when launched from textmate.
Just so you know, simplejson was incorporated into the Python 2.6 distribution's standard library as json. So if you don't feel like wrestling with the import problem, try simply changing all your references to simplejson to json instead.
But, as suggested, this is going to turn out to be a PythonPath issue. Run these lines in the Python interpreter and from TextMate and compare the results.
import sys
print sys.path
To find out where simplejson is installed (if you don't know), do this in the Python interpreter:
import simplejson
print simplejson.__file__
If you want/need to set PYTHONPATH manually for TextMate, you can do that by adding it under Preferences > Advanced > Shell Variables.