I created a Cassandra database in DataStax Astra. I'm able to connect to it in Python (using cassandra-driver module, and the secure_connect_bundle). I wrote a few api in my Python application to query the database.
I read that I can upload csv to it using dsbulk. I am able to run the following command in Terminal and it works.
dsbulk load -url data.csv -k foo_keyspace -t foo_table \
-b "secure-connect-afterpay.zip" -u username -p password -header true
Then I try to run this same line in Python using subprocess:
ret = subprocess.run(
['dsbulk', 'load', '-url', 'data.csv', '-k', 'foo_keyspace', '-t', 'foo_table',
'-b', 'secure-connect-afterpay.zip', '-u', 'username', '-p', 'password',
'-header', 'true'],
capture_output=True
)
But I got FileNotFoundError: [Errno 2] No such file or directory: 'dsbulk': 'dsbulk'. Why is dsbulk not recognized if I run it from Python?
A related question, it's probably not best practice to rely on subprocess. Are there better ways to upload batch data to Cassandra?
I think it has to do with the way path is handled by subprocess. Try specifying the command as an absolute path, or relative like "./dsbulk" or "bin/dsbulk".
Alternatively, if you add the bin directory from the DS Bulk package to your PATH environment variable, it will work as you have it.
Related
I'm struggling getting data from an older postgres database to an newer one.
I have already tried a few things without success. For more information, see Useing ANSI driver to connect to a postgreSQL DB with python psycopg2
The last idea is now to use pg_dump and pg_restore.
from subprocess import PIPE, Popen
# destination handling has to be adjusted.
def dump_table(host_name, database_name, user_name, database_password, table_name):
command = 'pg_dump -h {0} -d {1} -U {2} -p 5432 -t public.{3} -Fc -f /tmp/table.dmp'.format(host_name, database_name, user_name, table_name)
p = Popen(command, shell=True, stdin=PIPE, universal_newlines=True)
return p.communicate('{}n'.format(database_password))
After some research, I came across the above approach. Unfortunately, I haven't really done anything with the shell yet. As I currently understand it, the postgresDB is addressed locally so that the table.dmp is stored there in the /tmp/ directory.
However, I do not have direct access to this directory in order to download files from it. Do I have a possibility to receive the file created here directly in Python in order to process it directly on the target server? Because the target server is the one I have access to.
I created a custom command in django and I want to use a docker-compose command in it. I use a subprocess as follow:
class Command(BaseCommand):
def handle(self, *args, **options):
data = open_file()
os.environ['DATA'] = data
command_name = ["docker-compose ", "-f", "docker-compose.admin.yml", "up"]
popen = subprocess.Popen(command_name, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
universal_newlines=True)
return popen
when I do it I get a FileNotFoundError:
FileNotFoundError: [Errno 2] No such file or directory: 'docker-compose ': 'docker-compose
Is it even possible to use docker-compose inside of a command ?
It feels like I am missing something.
Thank you !
I see 2 possible things.
Your docker compose should run in background, so, you should add -d option at the end of the command: docker-compose -f docker-compose.admin.yml up -d
Best practice is start docker compose in background and you can take output with popen executing docker-compose -f docker-compose.admin.yml logs
You can also run docker-compose services and get interactive output, defining stdin_open: true in your yml file.
You could check if your current directory is where docker-compose.admin.yml exists, printing os.getcwd() and comparing it to docker-compose.admin.yml path.
I have generated a python script that opens a deployment config_file.yaml, modifies some parameters and saves it again, using pyyaml. This python script will be executed in the master node of a Kubernetes cluster.
Once is generated the new file, my intention is to execute
kubectl apply -f config_file.yaml
in the python script to apply the modifications to the deployment.
I have been reading how to do it using kubernetes python client, but it seems it is not prepare to execute kubectl apply.
So the other option is to create a bash script and execute it from python script.
Bash scripts:
#!/bin/bash
sudo kubectl apply -f config_file.yaml
I give it permissions
chmod +x shell_scipt.sh
Python script:
import subprocess
subprocess.call(['./shell_script.sh'])
But an error appears:
File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
raise child_exception
OSError: [Errno 13] Permission denied
I don't know how to resolve this error, I have tested givin permissions to the bash script, but nothing worked.
I do not know anything about Kubernetes but I think I might help.
I am basically suggesting that you run the command directly from Python script, not having Python running a bash script which runs a command.
import os
command = 'kubectl apply -f config_file.yaml'
password = 'yourpassword'
p = os.system('echo %s|sudo -S %s' % (passs, command))
If I understand correctly you are using python to dynamically modify static yaml files. If this is the case I would recommend using helm which if perfect for making static yaml file dynamic :-)
How are you running python script ?
I think you are running python script with a non sudo user. Try to run python script as sudo user this way your subprocess will have access to that file.
if this fixes your issue let me know.
I have two Raspberry Pi's. I am trying to transfer files from one Pi to the other using scp. I am trying to do this through Python because the program that will be transferring files is a python file.
below is the shell script I have for the SCP part (Blurred out the pass and IP):
#!/bin/sh
sshpass -p ######## scp test.txt pi#IP:/home/pi
and below is the Python Script that launches that Shell script.
import subprocess
subprocess.call(['./ssh.sh'])
print("DONE")
For some reason the python script doesnt kick back any errors and hits the print line but the file is not transferred. When i run the scp command outside of python the file transfers just fine. Am I doing something incorrect here?
****EDIT****
I cant even get Subprocess to work with this which is why i ended up using na shell script. Here is my attempt with Subprocess:
import subprocess
subprocess.call("sshpass -p ######## scp test.txt pi#IP:/home/pi")
print"DONE"
Again I get no errors, but the file is not transferred
****EDIT #2****
So I found out that because sshpass is being used, scp isnt prompting me to add the IP to known hosts, as a result the file simply isnt trnasferred at all. I need a way to add this acceptance into the script IE I ge the following if I launch the command without sshpass:
The authenticity of host 'IP (IP)' can't be established.
ECDSA key fingerprint is 13:91:24:8e:6f:21:98:1f:5b:3a:c8:42:7a:88:e9:91.
Are you sure you want to continue connecting (yes/no)?
I want to communicate to pass "yes\n" to this prompt as well as the password afterwards. Is this possible?
For the first query
You can use 'subprocess.popen' to get output(STDOUT) and error(STDERR) for the executed command.
import subprocess
cmd = 'sshpass -p ****** scp dinesh.txt root#256.219.210.135:/root'
p = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
print "Output is ",out
print "Error is ",err
If you execute above code with wrong password, the you will get below output:
[root#centos /]# python code.py
Output is
Error is Permission denied, please try again.
In this case, if the file is successfully transferred, then there is no output.
If you execute command like 'ls -l' then output will be printed.
For your second query (****EDIT #2****)
Options are :
Password less SSH. Check this.
Pexpect
I found a much easier way of tackling all of this
sshpass -p ###### scp -o StrictHostKeyChecking=no test.txt pi#IP:/home/pi
The -o switch allows me to auto store the IP into known hosts thus I do not need to communicate with the shell at all. The interaction from Python to Shell works with that addition; Doing this solely through subprocess also works.
If you don't mind to try other approaches it worth to use SCPClient from scp import.
I have a custom django management command in one django project and I would like to call it from another django project. Both are using virtualenv. I also need to check the output of the command.
The command I want to call looks something like this:
/home/path/to/bin/python /home/path/to/manage.py my-command --input-csv=/tmp/w2qGM1 --input-type=csv
Where /tmp/w2qGM1 is a temp csv file I've already created.
I've tried
subprocess.call
subprocess.call(CMD, shell=True, env={
'PYTHONPATH': os.pathsep.join(sys.path)})
subprocess.check_output
output = subprocess.check_output(
shlex.split(CMD),
shell=True,
stderr=subprocess.STDOUT,
env={'PYTHONPATH': os.pathsep.join(sys.path)}
)
subprocess.Popen
args = shlex.split(CMD)
output = subprocess.Popen(args).wait()
returns
Unknown command: 'my-command'
Type 'manage.py help' for usage.
I've tried adding cd /path/to/project/with/management/command/ && source bin/activate to pickup that projects django settings.
I should also mention I have all the correct paths in my PYTHONPATH and the command works when run from the command line and programmatically.
Both projects have to use djagno 1.7.