In this project we are asked to will implement value iteration and Q-learning, and test our agents first on Gridworld (from class), then apply them to a simulated robot controller (Crawler) and Pacman. The instructions are to download a zip folder and edit the valueIterationAgents.py and qlearningAgents.py which I have completed, but when I run commands like:
python gridworld.py -m
python gridworld.py -a value -i 100 -k 10
python pacman.py -p PacmanQAgent -x 2000 -n 2010 -l smallGrid
on bash terminal I get this error
python: can't open file 'Pacman.py':[Errno 1] Operation not allowed
I fixed it by moving my file from downloads to desktop
I'm trying to flush to stdout the output of a bioinformatic software written on Python code (Ete-Toolkit software). I tried the command (stdbuf) detailed on Force flushing of output to a file while bash script is still running but does not work because I have seen that stdbuf command it's only possible to execute from shell and not from bash(How to use stdbuf on a bash function).
Moreover from Python I discovered the following function that maybe could be interesting:
import sys
sys.stdout.flush()
But I don't know how can I implement inside the next bash script attached below.
The purpose is that if I only use the options -o and -e in the bash script (as you can see) the output is printed to logs_40markers in a not continuos manner which does not permits me to see the error. I can do it working directly from shell but my internet connection is not stable and practically each night there are a power outage and I have to restart again the command that will takes minimun one week.
#!/bin/bash
#$ -N tree
#$ -o logs_40markers
#$ -e logs_40markers
#$ -q all.q#compute-0-3
#$ -l mf=100G
stdbuf -oL
module load apps/etetoolkit-3.1.2
export QT_QPA_PLATFORM='offscreen'
ete3 build -w mafft_default-none-none-none -m sptree_fasttree_all -o provaflush --cogs coglist_species_filtered.txt -a multifasta_speciesunique.fa --clearall --cpu 40
&> logs_40markers
Thanks on advance if someone can give me some guide/advice,
Have a nice day,
Thank you,
Maggi
one informatician colleague of me solved the problem using the PYTHONUNBUFFERED command.
#!/bin/bash
#$ -N tree
#$ -o logs_40markers
#$ -e logs_40markers
#$ -q all.q#compute-0-3
#$ -l mf=100G
module load apps/etetoolkit-3.1.2
export QT_QPA_PLATFORM='offscreen'
export PYTHONUNBUFFERED="TRUE"
ete3 build -w mafft_default-none-none-none -m sptree_fasttree_all -o provaflush --cogs coglist_species_filtered.txt -a multifasta_speciesunique.fa --clearall --cpu 40 --v 4
To check the current situation of the process,
type in the shell:
tail output.log file -f (means follow)
I hope that someone could find this solution helpful
I want to process a set of HDR files with an Executable (e.g., falsecolor2.exe) file on google colab.
The source file is here: https://github.com/mostaphaRoudsari/honeybee/blob/master/resources/falsecolor2.exe?raw=true
sample HDR files:
http://www.anyhere.com/gward/hdrenc/pages/originals.html
The executable takes an HDR file with some arguments and generates a new HDR file.
On my local machine and drive, the following code works OK:
import os
os.system(r'D:\falsecolor2.exe -i D:\test.hdr -s 250.0 -n 10 -mask 0.1 -l lux -m 179 -lp EN -z > D:\test#fc.hdr')
I am not sure how to create a similar process in colab; after mounting the gdrive, the following code generates a 0 byte not-working HDR in my gdrive and returns error code 32256.
import os
os.system('/content/drive/My\ Drive/falsecolor2.exe -i /content/drive/My\ Drive/MOSELEY\ IMAGES/test.hdr -s 250.0 -n 10 -mask 0.1 -l lux -m 179 -lp EN -z > /content/drive/My\ Drive/test#fc.hdr')
I read some threads on shell and linux executables but could not replicate any of them successfully.
You can install Radiance in Google Colab like this:
# Download the Linux compiled version of radiance from Github (e.g. 5.3, latest official release at the moment):
!wget -O radiance.zip https://github.com/LBNL-ETA/Radiance/releases/download/012cb178/Radiance_012cb178_Linux.zip
# Unzip it
!unzip radiance.zip
# Extract the tar.gz to /usr/local/radiance
!tar -xvf radiance-5.3.012cb17835-Linux.tar.gz --strip-components=1 -C /
# Add /usr/local/radiance/bin to the PATH environment variable
path = %env PATH
%env PATH=/usr/local/radiance/bin:$path
# Set the RAYPATH environment variable to /usr/local/radiance/lib
%env RAYPATH=/usr/local/radiance/lib
I ran !lsb_release -a to find out the Linux distribution in Google Colab and it said that it was Ubuntu 18.04. Unfortunately, Radiance does not seem to be available for that version, but only for 16.04. That is why getting it from Github seems to be the next simplest solution. See radiance in Ubuntu Packages:
Exact hits
Package radiance
xenial (16.04LTS) (graphics): Lighting Simulation and Rendering System [universe]
4R1+20120125-1.1: amd64 arm64 armhf i386 powerpc ppc64el s390x
Then I tried to run your falsecolor command using one of the sample images you linked and found that the -lp and -z options are not available:
# Get a sample HDR file
!wget -O input.hdr http://www.anyhere.com/gward/hdrenc/pages/img/Apartment_float_o15C.hdr
# Try original command
!falsecolor -i input.hdr -s 250.0 -n 10 -mask 0.1 -l lux -m 179 -lp EN -z > output.hdr
# Output:
# bad option "-lp"
# Remove option -lp
!falsecolor -i input.hdr -s 250.0 -n 10 -mask 0.1 -l lux -m 179 -z > output.hdr
# Output:
# bad option "-z"
If you remove those options the command runs successfully:
# Remove option -z
!falsecolor -i input.hdr -s 250.0 -n 10 -mask 0.1 -l lux -m 179 > output.hdr
# List output file
!ls -lh output.hdr
# Output:
# -rw-r--r-- 1 root root 4.8M Mar 31 02:57 output.hdr
See a demo in this colab.
I have been struggling trying to get multiple instances of a python script to run on SLURM. In my login node I have installed python3.6 and I have a python script "my_script.py" which takes a text file as input to read in run parameters. I can run this script on the login node using
python3.6 my_script.py input1.txt
Furthermore, I can submit a script submit.sh to run the job:
#!/bin/bash
#
#SBATCH --job-name=hostname_sleep_sample
#SBATCH --output=output1.txt
#SBATCH --cpus-per-task=1
#
#SBATCH --mem=2G
python3.6 my_script.py input1.txt
This runs fine and executes as expected. However, if I submit the following script:
#!/bin/bash
#
#SBATCH --job-name=hostname_sleep_sample
#SBATCH --output=output2.txt
#SBATCH --cpus-per-task=1
#
#SBATCH --mem=2G
python3.6 my_script.py input2.txt
while the first is running I get the following error message in output2.txt:
/var/spool/slurmd/job00130/slurm_script: line 9: python3.6: command not
found
I found that I have this same issue when I try to submit a job as an array. For example, when I submit the following with sbatch:
!/bin/bash
#
#SBATCH --job-name=hostname_sleep_sample
#SBATCH --output=out_%j.txt
#SBATCH --array=1-10
#SBATCH --cpus-per-task=1
#
#SBATCH --mem=2G
echo PWD $PWD
cd $SLURM_SUBMIT_DIR
python3.6 my_script.py input_$SLURM_ARRAY_TASK_ID.txt
~
I find that only out_1.txt shows that the job ran. All of the output files for tasks 2-10 show the same error message:
/var/spool/slurmd/job00130/slurm_script: line 9: python3.6: command not
I am running all of these scripts in an HPC cluster that I created using the Compute Engine API in the Google Cloud Platform. I used the following tutorial to set up the SLURM cluster:
https://codelabs.developers.google.com/codelabs/hpc-slurm-on-gcp/#0
Why is SLURM unable to run multiple python3.6 jobs at the same time and how can I get my array submission to work? I have spent days going through SLURM FAQs and other stack questions but I have not found out a way to resolve this issue or a suitable explanation of whats causing the issue in the first place.
Thank you
I found out what I was doing wrong. I had created a cluster with two compute nodes, compute1 and compute2. At some point when I was trying to get things to work I had submitted a job to compute1 with the following commands:
# Install Python 3.6
sudo yum -y install python36
# Install python-setuptools which will bring in easy_install
sudo yum -y install python36-setuptools
# Install pip using easy_install
sudo easy_install-3.6 pip
from the following post:
How do I install python 3 on google cloud console?
This had installed python3.6 on compute1 and that is why my jobs would run on compute1. However, I didn't think this script had run successfully I never submitted it to compute2, and therefore the jobs sent to compute2 could not call python3.6. For some reason I thought Slurm was using python3.6 from the login node since I had sourced a path to it in my sbatch submission.
After installing python3.6 on cluster2 I was then able to import all of my locally installed python libraries based on the following link by including
import sys
import os
sys.path.append(os.getcwd())
at the beginning of my python script.
How to import a local python module when using the sbatch command in SLURM
I'm trying to create a python source package, but it fails when creating hard links for files.
$ python setup.py sdist
running sdist
running check
reading manifest template 'MANIFEST.in'
writing manifest file 'MANIFEST'
making hard links in foo-0.1...
hard linking README.txt -> foo-0.1
error: Operation not permitted
I've tried running the command with sudo, but it produces the same error.
This also produces the same error:
ln foo bar
I'm using vbox to run a virtual instance of ubuntu, which is probably where the problem comes from. Is there a way round using hard links when creating source distributions?
System information:
Ubuntu server 11.04;
VirtualBox 4.14;
osx 10.6.6;
python 2.7.1;
Same issue. I am using vagrant, my host OS is Windows while the Gust OS is Ubuntu. I am not a vim fan, so #simo's answer does not help me much because I really rely on virtual box shared folders to sync changes made by sublime editor to the Ubuntu virtual machine.
Thanks to Fabian Kochem, he found a quick and dirty workaround: post
# if you are not using vagrant, just delete os.link directly,
# The hard link only saves a little disk space, so you should not care
if os.environ.get('USER','') == 'vagrant':
del os.link
I ran into the same issues.
I was able to get it working by moving the python sources from the virtual box shared folder to my debian home folder. No error on sdist anymore.
I hope it helps.
It is unclear from your question what step is failing. Might be the hard linking right before the error. You can try strace to see what system call is failing. That should give a better picture of the problem at least.
This python bug report looks like they're not going to fix this until distutils2. Someone did supply a patch that might be useful to you. You might also be able to mount a directory over NFS and build there. I believe that NFS allows hard linking.
Looks like this was fixed in Python version 2.7.9 - https://hg.python.org/cpython/raw-file/v2.7.9/Misc/NEWS
Issue #8876: distutils now falls back to copying files when hard linking
doesn't work. This allows use with special filesystems such as VirtualBox
shared folders
This is the way I reached a working uwsgi(Ubuntu 14.04, default Python 2.7.6) with Python-2.7.10.
Steps
Before continuing, you must compile new Python with --enable-shared:
$ ./configure --enabled-shared
$ sudo make altinstall
Context: Ubuntu 14.04 with Python 2.7.6 with uwsgi and uwsgi-python-plugin installed with apt-get
Problem: I have a virtualenv for my all with compiled Python-2.7.10
# Previously installed Python-2.7.10 as altinstall
$ python2.7
Python 2.7.10 (default, Nov 25 2015, 11:21:38)
$ source ~/env/bin/activate
$ python
Python 2.7.10 (default, Nov 25 2015, 11:21:38)
Preparing stuff:
$ cd /tmp/
$ git clone https://github.com/unbit/uwsgi.git
$ cd uwsgi
$ make PROFILE=nolang
# On /tmp/uwsgi
$ PYTHON=python ./uwsgi --build-plugin "plugins/python python27"
On ini file:
[uwsgi]
plugins = python27
Results on:
** Starting uWSGI 1.9.17.1-debian (64bit) on [Thu Nov 26 12:56:42 2015] ***
compiled with version: 4.8.2 on 23 March 2014 17:15:32
os: Linux-3.19.0-33-generic #38~14.04.1-Ubuntu SMP Fri Nov 6 18:17:28 UTC 2015
nodename: maquinote
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 12
current working directory: /etc/uwsgi/apps-enabled
detected binary path: /usr/bin/uwsgi-core
your processes number limit is 257565
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: enabled
uwsgi socket 0 bound to UNIX address /var/run/uwsgi/app/pypi-server/socket fd 3
Python version: 2.7.10 (default, Nov 26 2015, 11:44:40) [GCC 4.8.4]
None of the above answers solved my problem. However, I was running the following command in a vagrant shared folder on Centos 6:
python setup.py bdist_bdrpm
And ended up with the error:
ln: creating hard link `xxx': Operation not permitted
error: Bad exit status from /var/tmp/rpm-tmp.S9pTDl (%install)
It turns out that it's a bash file that eventually executes the hard links:
cat /usr/lib/rpm/redhat/brp-python-hardlink
#!/bin/sh
# If using normal root, avoid changing anything.
if [ -z "$RPM_BUILD_ROOT" -o "$RPM_BUILD_ROOT" = "/" ]; then
exit 0
fi
# Hardlink identical *.pyc and *.pyo, originally from PLD's rpm-build-macros
# Modified to use sha1sum instead of cmp to avoid a diffutils dependency.
find "$RPM_BUILD_ROOT" -type f -name "*.pyc" | while read pyc ; do
pyo="$(echo $pyc | sed -e 's/.pyc$/.pyo/')"
if [ -f "$pyo" ] ; then
csha="$(sha1sum -b $pyc | cut -d' ' -f 1)" && \
osha="$(sha1sum -b $pyo | cut -d' ' -f 1)" && \
if [ "$csha" = "$osha" ] ; then
ln -f "$pyc" "$pyo"
fi
fi
done
Therefore you should be able to replace the hard link ln -f "$pyc" "$pyo" with a copy command cp "$pyc" "$pyo" in the above shell script.