I am trying to execute a python script on an Amazon Linux 2 instance. In my user-data section I have a script which copies the python script from an S3 bucket to the instance and executes it like so:
#!/bin/bash
# e - stops the script if there is an error
# x - output every command in /var/log/syslog
set -e -x
# set AWS region
echo "export AWS_DEFAULT_REGION=us-east-1" >> /etc/profile
source /etc/profile
# copy python script from the s3 bucket
aws s3 cp s3://${bucket_name}/ /home/ec2-user --recursive
sudo python3 my_python_script.py
The problem is that the python script doesn't seem to be getting executed at all.
Note: the python script gets copied fine from the bucket
What I am missing here?
UPDATE:
after checking /var/log/cloud-init-output.log it looks like the problem is in the python script, it cannot find the boto3 module:
+ python3 /home/ec2-user/my_python_script.py
Traceback (most recent call last):
File "/home/ec2-user/my_python_script.py", line 1, in <module>
import boto3
ModuleNotFoundError: No module named 'boto3'
Dec 10 15:52:25 cloud-init[3697]: util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [1]
Dec 10 15:52:25 cloud-init[3697]: cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
Dec 10 15:52:25 cloud-init[3697]: util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_scripts_user.pyc'>) failed
The problem is that I do have boto3 module installed. I created a custom AMI image that does have all of the modules installed (I used pip3 to install them) before creating the custom AMI image
UPDATE2
I verified that the image does have boto3 package installed in the python3 library:
[ec2-user#ip-ip ~]$ python3
Python 3.7.9 (default, Aug 27 2020, 21:59:41)
[GCC 7.3.1 20180712 (Red Hat 7.3.1-9)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>>
UPDATE3
The cause of the problem was that I installed the boto3 package for my user only (i.e. pip3 install boto3 --user) and then I created the AMI image. So after adding the bellow line to my user-data script it worked fine
#!/bin/bash
...
sudo pip3 install boto3
sudo python3 my_python_script.py
You can redirect output to a file and read it to see the error: did you have python3, did your instance have credentianl/role to access this bucket, did you script requires any third party, can you try to run the script above as root in local first, the run command should be python3 /home/ec2-user/my_python_script.py?
For anyone who is using cloudinit and terraform, this is how I managed it to work (one part of multipart cloudinit user data)-
part {
content_type = "text/x-shellscript"
filename = "run_src.sh"
content = <<-EOF
#!/bin/bash
cd /root
mkdir tmp
mkdir tmp/origin
mkdir tmp/converted
mkdir tmp/packaged
pip3 install boto3
cd src
python3 main.py
EOF
}
And it works like charm.
Related
I have a python file download.py in which I am importing a module opc
Now, this python file invoked from a shell script build.sh like:
python3 $(pwd)/scripts/download.py -d $dir_name
When I am running the shell script, I am getting the following error:
Traceback (most recent call last):
File "/home/ray/Desktop/repo/image-builder/scripts/download.py", line 14, in <module>
from opc.utils import file_utils, logger
ModuleNotFoundError: No module named 'opc'
This is how shell script is invoked:
sudo sh scripts/src/build.sh
I have a virtual environment active where I have installed this package. And this package is there as well
>pip list
Package Version
--------------------------------- -------
attrs 21.2.0
.
.
opc 0.8.0
Interestingly , when I run the python code directly like
python3 scripts/download.py -d dir_path
it works fine. I don't not get ModuleNotFoundError error.
Also, this works fine as well:
❯ python3
Python 3.8.10 (default, Sep 28 2021, 16:10:42)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import opc.utils.global_vars as g
>>> print(g.LOCAL_HOST)
127.0.0.1
>>>
Directory structure:
/home/ray/Desktop/repo/image-builder/scripts/download.py
/home/ray/Desktop/repo/image-builder/scripts/src/build.sh
and i am running commands from:
/home/ray/Desktop/repo/image-builder
This is likely to be because when you invoke the shell script, the python path is different from when you're running the Python interpreter. If that is the case, there are several ways to address that.
What happens if you manually change into the same directory as download.py and run your python script from there? Something like:
cd scripts
python download.py -d <some-dir>
Does that work, or do you still hit the same problem?
In shell scripts like this I first make sure the script will always be invoking the python app from the exact same place in the file system, so that I don't hit these python path issues. If you know for sure that python download.py -d <some-dir> works from inside the same directory as download.py, then you can add something like this to your shell script (assuming your shell script and download.py are in the same directory):
pushd "${0%/*}" # this changes directory to the directory where the shell script itself lives
# do your stuff here - run your python app, etc.
popd # returns to whatever directory we were originally in
In this way, I can always invoke the shell script from whatever directory I am in and I know it will behave the same way - the current directory is no longer relevant.
I wouldn't use $(pwd), because that will give the current directory, which varies, so your shell script will only ever work properly when invoked from inside the right directory if you do that.
I want to run a python file in the Git bash but it is not executing without adding py before the file name. Python is installed but when I use the shortcut to execute the python file, an error is shown as below-
user#WinPC MINGW64 /d/git_basic/scripts (main)
$ python
Python 3.10.0 (tags/v3.10.0:b494f59, Oct 4 2021, 19:00:18) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()
user#WinPC MINGW64 /d/git_basic/scripts (main)
$ ./all_checks.py
Python was not found; run without arguments to install
from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases.
Please guide me. Thanks
Edit 1
What I want is to run the python file in the terminal without using python or py in the command (py file.py or python file.py). I want to run it like this ./file.py
This is the all_checks.py code-
#! /usr/bin/env python3
import os
import sys
def check_reboot():
# returns true if pc has a pending reboot
return os.path.exists("/run/reboot-required")
def main():
if check_reboot():
print("pending reboot.")
sys.exit(1)
print("No pending reboot.")
sys.exit(0)
main()
Edit 2
I was able to run the file with ./ by just changing the python3 to python in the shebang line.
I was able to run the file with ./ by just changing the python3 to python in the shebang line.
Python 3 Docs - Shebang lines
I was not clear on exactly what you wanted but I have 2 options here.
Just running the file and making the file an executable.
If you just want to run it then use the terminal command, python file-name or python3 file-name if you have any python 2 installations.
You do not run a python file with this command: ./file-name
However, if you want to make it an executable file keep on reading.
If you are trying to make an executable, pyinstaller will do the job for you. If you have pip, then run the command pip install pyinstaller. Then, once you have pyinstaller installed, open terminal or command prompt and locate the directory of the python file using cd. Then once you are in the directory in terminal/command prompt run the command pyinstaller --onefile file-name include -w after --onefile if you do not want the command prompt/terminal/command-line to open up when run.
(Pyinstaller works for all operating systems)
I am completely new to docker (on windows 10 machine). I intend to setup a python development environment as a docker container. And most of the reading that I did involved the use of Dockerfile. I want to do it from scratch instead purely using commands.
What I intend to do is very basic requirement: To have python docker image present with me and that I should be able to install more libraries in that image and commit these updates to that image. But I want to do it completely using commands (not via a Dockerfile).
I am using Docker Desktop on windows 10 machine. I did docker pull python:latest and it pulled the image like so:
C:\Users\MyHomeDirectory>docker pull python:latest
latest: Pulling from library/python
d960726af2be: Pull complete
e8d62473a22d: Pull complete
8962bc0fad55: Pull complete
65d943ee54c1: Pull complete
532f6f723709: Pull complete
1334e0fe2851: Pull complete
062ada600c9e: Pull complete
aec2e3a89371: Pull complete
1ec7c3bcb4b2: Pull complete
Digest: sha256:65367d1d3eb47f62127f007ea1f74d1ce11be988044042ab45d74adc6cfceb21
Status: Downloaded newer image for python:latest
docker.io/library/python:latest
Then I did docker images and it showed that python latest image is present with a size of 886 MB.
C:\Users\Tejas.Khajanchee>docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
python latest 5b3b4504ff1f 47 hours ago 886MB
I am also able to enter the interactive python by doing docker run -it python and it generates the interactive shell:
Python 3.9.5 (default, May 12 2021, 15:26:36)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import gc
>>> import numpy
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'
>>>
>>> import pandas
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pandas'
>>>
>>> import openpyxl
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'openpyxl'
>>>
But as evident, some of the libraries are not installed. But this is where I get stuck. How do I install libraries into the python image and have the image updated. Also, if this shell is the only thing that I am allowed to do till now, what does the 886 MB content represent? Also I want to be able to run scripts using this docker image. When I attempt to do this on a very basic hello world script, the following error comes up:
C:\Users\MyHomeDirectory\Downloads>docker run -it python a.py
docker: Error response from daemon: OCI runtime create failed: container_linux.go:367: starting container process caused: exec: "a.py": executable file not found in $PATH: unknown.
I want to be able to do this purely with commands and not a Dockerfile. Please help.
First, looks you confuse the concept of image & container.
Docker image: read only, used as basis of container
Docker container: overlay a writeable layer upon the read only layer of docker image, all container will use image as basis
Second, for you, you mentioned you want to install numpy in the image, the best way for this is to customized a Dockerfile like next:
Dockerfile:
FROM python
RUN pip install numpy
Then, build a new image with docker build -t newpython .
BUT, you mentioned you don't want to use Dockerfile, then the replacement is next:
Install numpy in a container:
docker run -it python /bin/bash
# pip install numpy
Use docker ps -a to get the container id, e.g: 0a6b4df8e2c2, then commit this container which already have numpy installed to a new image:
docker commit 0a6b4df8e2c2 newpython
Finally, all new container need to run base on newpython image not python image, as only the newpython image has numpy installed:
docker run --rm newpython python -c "import numpy; print(numpy.__version__)"
1.20.3
Additional, for docker run -it python a.py, I think you misunderstand the concept. Container command like python a.py means the command will executed in container, so the a.py should be in container, not in host machine.
Hi My python installation is in different directory and i am using a docker image which is mac based and it is referring shebang line as /user/local/bin/python3 from other folder in shell script .
my python installation path
Python 3.4.3 (default, Oct 14 2015, 20:28:29)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/home/myuser/project', '/usr/lib/python3.4', '/usr/lib/python3.4/plat-x86_64-linux-gnu', '/usr/lib/python3.4/lib-dynload', '/usr/local/lib/python3.4/dist-packages', '/usr/lib/python3/dist-packages']
>>>
so is there a way without changing the shebang line i can redirect or link to my installation of python3 to get out of this error.
is it recommended to install python3 in given path. ?
please advice.
If you can't modify the shebang of the file, and you have access to the Dockerfile that creates your docker image, you can add a command directive to create a symbolic link: ln -s /usr/bin/python3 /usr/local/bin/.
If you don't have access to the Dockerfile. Then you can run the above command from within the running docker instance. That should solve your issue without having to modify the file.
https://docs.docker.com/engine/reference/builder/#cmd
You could set you shebang to "/usr/bin/env python" as usual, then set your path appropriately so that the correct version of python is on the path for your executable. In bash you can set the path on the command line using:
PATH=python/path:$PATH app
I will sometimes ignore the shebang and type python/path/python $(which app) in order to control which python interpreter is running.
When using Python on an interactive shell I'm able to import the cx_Oracle file with no problem. Ex:
me#server~/ $ python
Python 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cx_Oracle
>>>
As you can see, importing works without a hitch. However, when I try to run a Python script doing the same thing, I get an error:
me#server~/ $ sudo script.py
Traceback (most recent call last):
File "/usr/local/bin/script.py", line 19, in <module>
import cx_Oracle
ImportError: No module named "cx_Oracle'
Here is the important section from script.py:
# 16 other lines above here
# Imports
import sys
import cx_Oracle
import psycopg2
...
I'm befuddled here. Other pertinent information is the server I'm running is Ubuntu 14.04.1 LTS (upgraded from 12.04) 64bit. which python and sudo which python both point to the same location. Also, doing this as root via sudo su - gets the same results; import OK from interactive but error from script.
Nothing other than the OS upgrade happened between when this worked and when it stopped working.
Sorry, all. This was a silly on my part. Turns out the script in question was using Python3, and when the server upgraded, Python3 went from being 3.2 version to being 3.4 version.
Once the cx_Oracle module was set up in the 3.4 version, everything worked as expected.
Phil, your final note talking about the shebang was what lead me to discover this, so kudos to you! The reason I didn't mark your response as the answer was because technically it wasn't but led me on the right path.
Cheers!
sudo starts a new bash environment which is then pointing to a different python executable (different installed modules).
You can verify this with which python and sudo which python
EDIT: so if they point to the same executable, then you should look at sys.path to find differences. In both environemnts you can:
python -c "import sys; print('\n'.join(sys.path))"
sudo python -c "import sys; print('\n'.join(sys.path))"
Look for differences. If there are none:
A common error in import situations like this is that python will first look at the local dir. So if you happen to be running python and importing something what is found locally (i.e. cx_Oracle is a subdir of your current location), you will get an import error if you change directories.
Final note: I have assumed here that the shbang of the script.py points to the same executable as which python. That is, that python script.py and script.py return the same error.