I'm new to django and python and am trying to determine the best way to 'play around' with querying (I come from a front-end background and am used to using console.log there, but cannot at the back-end).
If I run python3 manage.py shell, I can then run useful commands in that shell, for example, I can test querying Tracking objects:
from myapp.tracking.models import Tracking
trackings = Tracking.objects.all()
print(trackings)
However, in my setup I apparently need to run a shell inside docker:
sudo docker-compose exec myapp bash
Here I can't run commands like from myapp.tracking.models import Tracking as I get errors like bash: from: command not found. From googling around, I assume that bash is a different type of shell to whatever the python manage.py shell shell uses (Ipython?).
So my question is how can I use the same shell that the python3 manage.py shell command uses? Presumably running something like sudo docker-compose exec myapp SOME_OTHER_SHELL? And if this is not possible, am I correct in assuming that bash is just a different type of shell with a different syntax for importing and querying variables? If so, any useful links to docs would be much appreciated.
Thanks
Related
I would like to execute a circleCN.py python script by using #!/bin/sh.
I have searched the internet, found some suggestions, and none work. I get permission denied, and I changed my chmod of circleCN.py with a+x, I get invalid syntax, or I get another error. I have tried . /path/circleCN.py, exec /path/circleCN.py, python -c "/path/circleCN.py" and none work.
I have also tried to change #!/bin/sh to #!/usr/bin/env python and had no success. I would like to keep #!/bin/sh though.
I am very new to shell programming. I am also new to Linux.
When OpenFOAM users write Allrun or Allclean scripts, they use the #!/bin/sh shebang. I am working on an automation script and I would like to conform to the standard.
Why do you need to execute python script under a bash shebang?
You can invoke python script inside the shell script like this
#!/bin/sh
python /path_to_file/circleCN.py
I am using dryscrape in a python script. The python script is called in a bash script, which is run by cron. For those who may not be aware, dryscrape is a headless browser (use QtWebkit in the background - so requires an xsession).
Here are the main points concerning the issue I'm having
When I run the python script from the command line, it works
When I run the bash script from the command line, it works too
I figured out that this may have something to do with different environments between my command prompt and when the cron job is running, so I modified my bash script to source my .profile as follows:
#/bin/bash
. /full/path/to/my/home/directory/.profile
python script_to_run.py
This is what my cronjob crontab entry looks like:
0,55 14-22 * * 1-5 /path/to/script.sh >> $(date "+/path/to/logs/\%Y\%m\%d.mydownload.log" )
By the way, I know that the job is being run (I can see entries in /var/log/syslog, and the script also writes to a log file - which is where I get the error message below):
In all cases, I got the following error message:
Could not connect to X server. Try calling dryscrape.start_xvfb()
before creating a session
I have installed the prerequisites, on my machine (obviously - since it runs at the command line). At the moment, I have run out of ideas.
What is causing the script to run fine at the console, and then fail when run by cron?
[[Relevant Details]]
OS: Linux 16.0.4 LTS
bash: version 4.3.46(1)
cron user: myself (i.e. same user at the command prompt)
dryscrape: version 1.0.1
The solution to this was to call the dryscrape.start_xvfb() method before starting the dryscrape session.
Cron user does not have display, so you cannot run any command which requires a display.
You need to modify the python script to do not use any type of display (check carefully, because some python commands, even though they do not open any display , they internally check for this variable).
The best way to test is to ssh into the machine without Display, and check if you can run it from there without erros.
I'm trying to run a python script from bamboo. I created a script task and wrote inline "python myFile.py". Should I be listing the full path for python?
I changed the working directory to the location of myFile.py so that is not a problem. Is there anything else I need to do within the configuration plan to properly run this script? It isn't running but I know it should be running because the script works fine from terminal on my local machine. Thanks
I run a lot of python tasks from bamboo, so it is possible. Using the Script task is generally painless...
You should be able to use your script task to run the commands directly and have stdout written to the logs. Since this is true, you can run:
'which python' -- Output the path of which python that is being ran.
'pip list' -- Output a list of which modules are installed with pip.
You should verify that the output from the above commands matches the output when ran from the server. I'm guessing they won't match up and once that is addressed, everything will work fine.
If not, comment back and we can look at a few other things.
For the future, there are a handful of different ways you can package things with python which could assist with this problem (e.g. automatically installing missing modules, etc).
You can also use the Script Task directly with an inline Python script to run your myFile.py:
/usr/bin/python <<EOF
print "Hello, World!"
EOF
Check this page for a more complex example:
https://www.langhornweb.com/display/BAT/Run+Python+script+as+a+Bamboo+task?desktop=true¯oName=seo-metadata
I'm just getting started using imposm to help get openstreetmap data into a postgis database. All the docs point to making all commands via Terminal. This is fine for one off imports but I plan to have many many imports of varying bounding boxes and would like to script the loading of the data in the database.
Currently I use:
imposm --overwrite-cache --read --write -d postgis_test --user postgres -p "" /Users/Me/MapnikTest/osmXML.osm
Which works fine from the command line but as osmXML.osm is being created many times I would like somehow to import this at the point of creation.
Putting the same thing in a python script as:
os.system("imposm --overwrite-cache --read --write -d postgis_test --user postgres -p "" /Users/Ali\ Mac\ Pro/Desktop/MapnikTest/osmXML.osm")
just returns:
/bin/sh: imposm: command not found
Solving this would be the final step to automate the acquisition of data to render small maps on demand but I'm falling at the final hurdle!
** Edit full path to imposm solved the first problem but imputing the password for the postgres user happens when prompted. Is there a way to send the password in the same single line command? (maybe this needs to be a new post?, happy if someone points me in the right direction)**
This is probably because os.system() is calling /bin/sh which uses a different shell environment from the one you use when working on the command line.
To work around this, in your script, get the full path to the imposm script and then use that in your command. Use can use some code like this to find the executable.
Or you can fix your shell definitions so that /bin/sh has the proper PATH defined, but that depends greatly on your setup...
Solved with the help of further research and the comments from #Eli Rose (many thanks): find out what path to imposm (or whichever command you are trying to make) with
which <command>
Then include the path in the python shell command. Using a module from subprocess you can even see the full terminal output.
import subprocess
from subprocess import *
print Popen("/usr/local/bin/imposm --overwrite-cache --read --write --connection postgis://<database user>:<password>#<host>/<database> /path/to/data.osm", stdout=PIPE, shell=True).stdout.read()
The
--connection postgis://<database user>:<password>#<host>/<database>
means you can make the command in a single line and not have to worry about entering the database user password in a following command.
This question: Automatically import models on Django shell launch has answers explaining how to import models upon start by using shell_plus, but no answer about how to run code in general.
But is there an easy way to just run a python script?
python manage.py shell [or shell_plus] --run=script.py
Would just run the script as if you'd typed the whole thing in as the shell started.
I realize that you can import things in the shell, but then they're stuck within a namespace.
I would think ipython should have a way to run a script, and then import its locals() into the toplevel namespace. In that case you could just do %magic script.py and we'd be down to just one step, which would be good.
Changing the way you start the shell should be fine - the main goal is to just be able to create a file that's run on startup of the shell.
You can create your own custom command just like shell_plus has done: see the source of the shell_plus command to see how. In that code you can specify and run the file that needs to be executed before starting the shell. Also useful is Django's documentation on creating custom commands.
You can try to use environment variable PYTHONSTARTUP.
Also try django-extensions: django-extensions
See django-extensions/management/commands/shell_plus.py command.
From source code of this command I see that it respects PYTHONSTARTUP env variable.
shell_plus uses a limited form of IPython which doesn't process its startup & configuration, which defeats most normal attempts to run things at django+ipython shell startup. You can switch it to use the full version which will solve most problems.
Modify django_extensions/management/commands/shell_plus.py
remove:
embed(user_ns=imported_objects)
and replace it with:
from IPython import start_ipython
start_ipython(argv=[], user_ns=imported_objects)
Then your python code in the startup directories will be loaded.
Not sure if there's a flag you can use, but if you have ipython installed it should be as simple as:
ipython
Then when you're in the prompt:
run script.py
Then:
run manage.py shell
It seems that the easiest way is to run
cat myscript.py | awx-manage shell
For reference, see https://github.com/ansible/awx-operator/blob/7d2d1b3c5e3766966bfec0f9f58037f654b93b59/roles/installer/tasks/initialize_django.yml#L21-L24