I'm trying to bring on my local server (XAMPP), a script that's working on my VPS server (Linux CentOS7).
On XAMPP, I call the Python script wit PHP, something like:
$hotel = array("Name"=>$_POST["NAME"]
,..
);
$param = escapeshellcmd(base64_encode(json_encode($hotel)));
$result = shell_exec('python C:\xampp\htdocs\bounce.py $param');
$obj = json_decode($result);
The Python script is something like:
#! /Users/<user>/AppData/Local/Programs/Python/Python37/python.exe
import sys
import json
import base64
content = json.loads(base64.b64decode(sys.argv[1]))
print(json.dumps(content))
The returned JSON string is NULL
This is the Apache error:
[php:warn] [pid 11176:tid 1884] [client ::1:55182] PHP Warning: Attempt to read property "Name" on null in C:\\xampp\\htdocs\\hotel_results.php on line 46, referer: http://localhost/
Updated Django with no results
If the problem is caused by not reading the "$param" variable due to single quotes, replacing the line of
$result = shell_exec('python C:\xampp\htdocs\bounce.py $param');
with
$result = shell_exec("python C:\\xampp\\htdocs\\bounce.py $param");
could help. worth a try.
Related
I'm trying to make this work on XAMPP. It works fine on a Linux server/virtual machine.
I need to send an associative array to a python script, and I do it with:
<?php
echo "Test simple call/answer process<br>";
$age = array("Peter"=>"35", "Ben"=>"37", "Joe"=>"43");
$param = base64_encode(json_encode($age));
$command = escapeshellcmd('python python_array_answer.py $param');
$output = shell_exec($command);
$obj = json_decode($output);
print_r($obj);
?>
The Python script is:
#! /Users/.../AppData/Local/Programs/Python/Python37/python.exe
import sys
import json
import base64
hotel_data = json.loads(base64.b64decode(sys.argv[1]))
print(json.dumps(hotel_data))
I get a NULL array back, with this error:
Traceback (most recent call last):
File "python_array_answer.py", line 6, in <module>
hotel_data = json.loads(base64.b64decode(sys.argv[1]))
File "C:\Users\mghiglia\AppData\Local\Programs\Python\Python37\lib\base64.py", line 87, in b64decode
return binascii.a2b_base64(s)
binascii.Error: Invalid base64-encoded string: number of data characters (5) cannot be 1 more than a multiple of 4
Any ideas? thank.
Assuming the rest of the code is okay, you're using single quotes in this line
$command = escapeshellcmd('python python_array_answer.py $param');
where you should either use double quotes, so php known to replace $param with the value of variable $param, or concatenate the two like this...
$command = escapeshellcmd('python python_array_answer.py '.$param);
I'm facing quite simple problem, yet still not able to figure out what in particular causes that, and - more importantly - how to solve it.
I'm currently on Linux, and I would like to run an python app (script) from a Java application. The script is on PATH, thus I really would like to utilise that (avoid using absolute path to the script), if possible.
However, all I tried resulted in various forms of "File does not exist".
Sample
As a demonstration, I've tried to run one python3-based app (meld)
and one binary-built app (ls) for comparison:
$ which ls
/usr/bin/ls
$ file /usr/bin/ls
/usr/bin/ls: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=[...], for GNU/Linux 3.2.0, stripped
$ which meld
/usr/bin/meld
$ file /usr/bin/meld
/usr/bin/meld: Python script, UTF-8 Unicode text executable
$ head -n1 /usr/bin/meld
#!/usr/bin/python3
Nextly, I've created simple Java main, which tries several ways how to start theese:
package cz.martlin.processes;
import java.io.File;
import java.io.IOException;
import java.lang.ProcessBuilder.Redirect;
import java.util.List;
import java.util.stream.Collectors;
public class ProcessPlaying {
public static void main(String[] args) {
List<List<String>> commands = List.of(
List.of("ls"),
List.of("/usr/bin/ls"),
List.of("meld"),
List.of("/usr/bin/meld"),
List.of("python3", "/usr/bin/meld"),
List.of("/usr/bin/python3", "/usr/bin/meld"),
List.of("sh", "-c", "meld"),
List.of("sh", "-c", "/usr/bin/meld"),
List.of("sh", "-c", "python3 /usr/bin/meld")
);
for (List<String> command : commands) {
run(command);
}
}
private static void run(List<String> command) {
System.out.println("Running: " + command);
//String executable = command.get(0);
//boolean exists = new File(executable).exists();
//System.out.println("Exists the " + executable + " ? " + exists);
try {
ProcessBuilder pb = new ProcessBuilder(command);
pb.redirectError(Redirect.INHERIT);
Process proc = pb.start();
// Process proc = Runtime.getRuntime().exec(command.stream().collect(Collectors.joining(" ")));
int code = proc.waitFor();
System.out.println("OK, return code: " + code);
} catch (IOException e) {
System.out.println("Failed to start: " + e.toString());
} catch (InterruptedException e) {
System.out.println("Failed to await: " + e.toString());
}
System.out.println();
}
}
Here are the results:
Running: [ls]
OK, return code: 0
Running: [/usr/bin/ls]
OK, return code: 0
Running: [meld]
Failed to start: java.io.IOException: Cannot run program "meld": error=2, Directory or file doesn't exist
Running: [/usr/bin/meld]
Failed to start: java.io.IOException: Cannot run program "/usr/bin/meld": error=2, Directory or file doesn't exist
Running: [python3, /usr/bin/meld]
python3: can't open file '/usr/bin/meld': [Errno 2] No such file or directory
OK, return code: 2
Running: [/usr/bin/python3, /usr/bin/meld]
/usr/bin/python3: can't open file '/usr/bin/meld': [Errno 2] No such file or directory
OK, return code: 2
Running: [sh, -c, meld]
sh: line 1: meld: command not found
OK, return code: 127
Running: [sh, -c, /usr/bin/meld]
sh: line 1: /usr/bin/meld: Directory or file doesn't exist
OK, return code: 127
Running: [sh, -c, python3 /usr/bin/meld]
python3: can't open file '/usr/bin/meld': [Errno 2] No such file or directory
OK, return code: 2
To sum it up:
all of the commands tried works when executed directly from the shell (either sh or bash)
executing the binary works either by the full path (/usr/bin/ls) or by just the name (ls)
executing the python script the same way doesn't work neither way
when trying to run the python3 interpreter and populate the script path as an argument to the python, now the python yields the script file doesn't exist
trying to populate it as a command to the brand new shell didn't help either
I've tried to use the Runtime#exec (based on this comment: https://stackoverflow.com/a/36783743/3797793) to start the process (both the exec(String) and exec(String[] forms), but no sucess (none of the listed commands did actually execute).
Thus, my question is/are:
What do I understand wrong?
How to start the Python script from Java?
Would some small (ba)sh script wrapper do the job?
Since it's python3-based, Jython wouldn't help here (because the latest one is 2.7.*), would it?
Further requirements:
As mentioned, I would like to be able to avoid using full path to the Python script
Also, I would like to have platform independant solution (at least Linux and Windows compatible)
Thanks in advance
I call my python script with java and also pass variables. My The script alone in a PyCharm project works so far, only if I call it via Java (InteliJ) nothing happens.
The program should pass values to my script via Java so that the script changes and saves values in a Word Document (docx).
If I use the version of my script in the Intelij folder, it has problems importing docx.
I think the problem is that in the PyCharm project the docx data is directly in the project. This is not the case with Intelij and it should access the system data from docx.
I have already completely reinstalled lxml and docx but to no avail.
How do I have to change my program structure or my script so that it works.
Java:
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
public class runPython {
public static void main(String[] args) throws IOException {
try {
ProcessBuilder builder = new ProcessBuilder("python", "C://Users//Notebook//IdeaProjects//bhTool_bridge//scripts//main.py", "Annemarie Brekow");
Process process = builder.start();
}
catch (Exception e){
e.printStackTrace();
}
}
}
Python:
from docx import Document
import sys
import os
document = Document('C:/Users/Notebook/IdeaProjects/bhTool_bridge/template/Muster_Rechnung.docx')
workingdirectory=os.getcwd()
def find_replace(paragraph_keyword, draft_keyword, paragraph):
if paragraph_keyword in paragraph.text:
# print("found")
paragraph.text = paragraph.text.replace(paragraph_keyword, draft_keyword)
for paragraph in document.paragraphs:
find_replace("$Kunden-Name", "Annemarie Brekow", paragraph)
print(paragraph.text)
document.save("C:/Users/Notebook/IdeaProjects/bhTool_bridge/template/Muster_Rechnung.docx")
EDIT:
Errormsg
Traceback (most recent call last):
File "C:\Users\Notebook\IdeaProjects\bhTool_bridge\scripts\main.py", line 2, in
from docx import Document
ModuleNotFoundError: No module named 'docx'
I am trying to execute dataflow jar through airflow script. For it i am using DataFlowJavaOperator. In the param jar,i am passing the path of the executable jar file present in the local system.But when i try to run this job i get error as
{gcp_dataflow_hook.py:108} INFO - Start waiting for DataFlow process to complete.
[2017-09-12 16:59:38,225] {models.py:1417} ERROR - DataFlow failed with return code 1
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1374, in run
result = task_copy.execute(context=context)
File "/usr/lib/python2.7/site-packages/airflow/contrib/operators/dataflow_operator.py", line 116, in execute
hook.start_java_dataflow(self.task_id, dataflow_options, self.jar)
File "/usr/lib/python2.7/site-packages/airflow/contrib/hooks/gcp_dataflow_hook.py", line 146, in start_java_dataflow
task_id, variables, dataflow, name, ["java", "-jar"])
File "/usr/lib/python2.7/site-packages/airflow/contrib/hooks/gcp_dataflow_hook.py", line 138, in _start_dataflow
_Dataflow(cmd).wait_for_done()
File "/usr/lib/python2.7/site-packages/airflow/contrib/hooks/gcp_dataflow_hook.py", line 119, in wait_for_done
self._proc.returncode))
Exception: DataFlow failed with return code 1`
My airflow script is :
from airflow.contrib.operators.dataflow_operator import DataFlowJavaOperator
from airflow.contrib.hooks.gcs_hook import GoogleCloudStorageHook
from airflow.models import BaseOperator
from airflow.utils.decorators import apply_defaults
from datetime import datetime, timedelta
default_args = {
'owner': 'airflow',
'start_date': datetime(2017, 03, 16),
'email': [<EmailID>],
'dataflow_default_options': {
'project': '<ProjectId>',
# 'zone': 'europe-west1-d', (i am not sure what should i pass here)
'stagingLocation': 'gs://spark_3/staging/'
}
}
dag = DAG('Dataflow',schedule_interval=timedelta(minutes=2),
default_args=default_args)
dataflow1 = DataFlowJavaOperator(
task_id='dataflow_example',
jar ='/root/airflow_scripts/csvwriter.jar',
gcp_conn_id = 'GCP_smoke',
dag=dag)
I am not sure what mistake i am making ,Can anybody please help me to get out of this
Note :I am creating this jar while selecting option as Runnable JAR file by packaging all the external dependencies.
The problem was with the jar that I was using. Before using the jar, Make sure that the jar is executing as expected.
Example:
If your jar was dataflow_job1.jar, Execute the jar using
java -jar dataflow_job_1.jar --parameters_if_any
Once your jar runs successfully, Proceed with using the jar in Airflow DataflowJavaOperator jar.
Furthermore,
If you encounter errors related to Coders, you may have to make your own coder to execute the code.
For instance, I had a problem with TableRow class as it didnot have a default coder and thus i had to make this up:
TableRowCoder :
public class TableRowCoder extends Coder<TableRow> {
private static final long serialVersionUID = 1L;
private static final Coder<TableRow> tableRow = TableRowJsonCoder.of();
#Override
public void encode(TableRow value, OutputStream outStream) throws CoderException, IOException {
tableRow.encode(value, outStream);
}
#Override
public TableRow decode(InputStream inStream) throws CoderException, IOException {
return new TableRow().set("F1", tableRow.decode(inStream));
}
#Override
public List<? extends Coder<?>> getCoderArguments() {
// TODO Auto-generated method stub
return null;
}
#Override
public void verifyDeterministic() throws org.apache.beam.sdk.coders.Coder.NonDeterministicException {
}
}
Then Register this coder in your code using
pipeline.getCoderRegistry().registerCoderForClass(TableRow.class, new TableRowCoder())
If there are still errors(which are not related to coders) Navigate to:
*.jar\META-INF\services\FileSystemRegistrar
and add any dependencies that may occur.
For example there might be a staging error as:
Unable to find registrar for gs
i had to add the following line to make it work.
org.apache.beam.sdk.extensions.gcp.storage.GcsFileSystemRegistrar
I want to use syslog-ng to receive netgear log, and use python script process.
But syslog-ng didn't run the python script.
syslog-ng.config
#version:3.2
options {
flush_lines (0);
time_reopen (10);
log_fifo_size (1000);
long_hostnames (off);
use_dns (no);
use_fqdn (no);
create_dirs (no);
keep_hostname (yes);
};
source s_sys {
udp(ip(0.0.0.0) port(514));
};
destination d_python{
program("/usr/local/bin/python /opt/syslog.py");
#program("/bin/echo 'haha' >> /tmp/test");
};
log { source(s_sys); destination(d_python);};
and python script like this
#!/usr/local/bin/python
#coding:utf8
import os
import sys
import datetime
f = open('/var/log/pnet.log', 'a')
f.write('\nstart\n')
f.write('args\n')
f.write('%s\n' % sys.argv)
if not sys.stdin.isatty():
f.write('stdin\n')
f.write('%s\n' % date.date.now().isoformat() )
tty_read = sys.stdin.read()
f.write("'''\n")
f.write(tty_read)
f.write("\n'''\n")
f.write('end\n')
f.close()
The script is already 777
Even I change my config to use 'echo' directly pipe into a file, didn't write a word too...
So...why?
silly question, but do you have incoming logs? If you use a simple file destination instead of the program, do you receive logs? If not, the problem is not in the program destination.
Also, try changing the flush_lines (0); option to 1 to see if it helps.
Regards,
Robert Fekete
I could show you my code for reference:
my syslog-ng.conf :
source s_test{
file("/home/test/in.log" follow-freq(1) flags(no-parse));
};
destination d_test{
program ("/home/test/out.py" flush_lines(1) flags(no_multi_line));
};
log {
source(s_test);
destination(d_test);
flags(flow-control);
};
my out.py:
#!/usr/bin/env python
# encoding: utf-8
import sys
while True:
line = sys.stdin.readline()
file = open("/home/test/out_from_python.log","a")
file.write(line)
file.close()
when you type echo "something" >> /home/test/in.log , there would be a new log lie in /home/test/out_from_python.log