I'm trying to get into writing multi-threaded scripts in Python for processing large lists of files. In the past, I've used utilities like GNU Parallel, but my needs are starting to exceed what that program can provide.
I've written a simple demo in Java which demonstrates what I'd like to do here:
public class ExecutorTest {
private ExecutorService executorService = Executors.newFixedThreadPool(8);
public static void main(String[] args) {
ExecutorTest t = new ExecutorTest();
t.start();
}
public ExecutorTest() {
}
public void start() {
File f = new File(".");
for (File file : f.listFiles()) {
final String fullPath = file.getAbsolutePath();
this.executorService.submit(new Runnable() {
#Override
public void run() {
// sleep for up to 100 milliseconds
try { Thread.sleep(new Double(Math.random() * 100.0).longValue()); } catch (InterruptedException e) {}
System.out.println(fullPath);
}
});
}
this.executorService.shutdown();
try {
this.executorService.awaitTermination(Long.MAX_VALUE, TimeUnit.SECONDS);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
I'd like to do the same thing in Twisted but I'm having a difficult time understanding how all the working pieces come together so I can start a bounded multi-threaded worker queue, then wait for it to finish.
How do you write something like this in Twisted? I've done a lot of work in Python so I'm not learning Python, I'm just trying to figure out how to use Twisted.
Related
I want Nodejs to run a Python script in a new terminal.
I have a python script named test.py which simply does print('Hello World')
However, when I enter localhost:8080/python_run/ in my browser, I do not know if the script is run.
I can know the output using python.stdout.on('data', (data) => {console.log(`stdout: ${data}`);})
but I want to be able to monitor a large complex python script running which would give many print statements throughout.
I want it to display in a separate terminal instead of seeing only the std.out in the nodejs terminal.
I also wish to only implement these using vanilla nodejs.
The spawn child process of nodejs does not seem to let me run in a separate terminal.
How can I trigger a long complex and probably resource intensive Python script to run with the nodejs server as long as it receives a request (GET request of the url in this case)? If possible I'd also want the Python script to communicate back to the nodejs server once the Python script has finished running, to trigger some change of display of a website.
My current code
const {parse} = require('querystring');
const http = require('http');
const path = require('path');
const fs = require('fs');
var url = require("url");
const {spawn} = require('child_process');
const spawn1 = require('child_process').exec;
const requestHandler = (req, res) => {
fs.readFile("index_noexp.html",(err, data) => {
if(!err) {
res.writeHead(200, {'Content-Type': 'text/html'});
res.end(data);
}
});
if (/python_run/i.test(req.url)) {
console.log('Now running the dangerous python script');
try {
const { spawn } = require('node:child_process');
var python = spawn('python', ['/Users/damian/nodetest/test.py']);
// var python = spawn1('python', [__dirname + '/test.py']);
python.stdout.on('data', (data) => {
console.log(`stdout: ${data}`);
})
} catch (error) {
console.error(error);
} finally {
console.log('Not sure what happened');
}
}
const server = http.createServer(requestHandler);
server.listen(8080);
I'm building an Electron-Python application. I'm using print statements (python-side) to distinguish different "phases" of the pipeline and I'd like to show these up on the UI, but I'm struggling to find an approach that works. Basically, I can't understand how to send data to the browser console every time that the "data" event is emitted from the child process.
Among other things, I've tried exposing ipcRenderer through contextBridge and messing with webContents.send, but I haven't been able to get anything through any of these methods, and being a beginner to Electron doesn't help.
Here's the handler:
// retrieve video data
ipcMain.handle('get-games', async (event, arg) => {
const pythonProcess = spawn('python', ["./backend/predict_games.py", arg]);
// sets pythonProcess as a global variable
global.childProcess = pythonProcess;
return new Promise((resolve, reject) => {
let result = "";
pythonProcess.stdout.on('data', async (data) => {
data = String(data);
if (data.startsWith("{"))
result = JSON.parse(data);
});
pythonProcess.on('close', () => {
resolve(result);
})
pythonProcess.on('error', (err) => {
reject(err);
});
})
});
Does anyone have any clue? If available, I'm up to go with a completely different approach.
I want to make a socket between Java client
`
import java.util.*;
import java.net.*;
import java.io.*;
public class Main
{
public static void client()
{
final int port = 2003;
final String host = "localhost";
try
{
Socket soc = new Socket(host, port);
if(soc.isBound()){
System.out.println("socket binded");
}else if(soc.isConnected()){
Scanner in = new Scanner(soc.getInputStream());
System.out.println("enter msg");
String x = in.nextLine();
PrintStream p = new PrintStream(soc.getOutputStream());
p.println("#client" + x);
} else if(soc.isClosed()){
System.out.println("Closed tunnel");
System.exit(0);
soc.close();
}
}
catch (Exception e)
{
}
}
public static void main(String[] args)
{
client();
}
}
`
Its having problem allowing me to use the scanner to send msg to Python server{I dunno how to make this}
PS : am new and dunno how to make this snippets formatted
I have a problem where a page that I can open in Firefox, Chrome, or even Java fails to open in Python using urllib2:
import urllib2
sock = urllib2.urlopen('http://www.example.com')
li = sock.read()
sock.close()
print li
This code fails (for the particular company web page I am trying to load). The page is actually an interface to a complex back-end server, and the response we are getting is just a couple lines of (incorrect) text. At first we thought there was some bot filtering going on, but we did get the page to load using Java:
package com.ebay.marketing.rtm.components.impl.selector;
import com.sun.jersey.api.client.Client;
import com.sun.jersey.api.client.ClientResponse;
import com.sun.jersey.api.client.WebResource;
public class RtmApiCheck {
private static Client mClient;
private void initClient() {
Client client = mClient;
if (client == null) {
synchronized (this) {
client = mClient;
if (client == null) {
mClient = client = Client.create();
}
}
}
}
public static void main(String[] args) {
RtmApiCheck check = new RtmApiCheck();
try {
check.initClient();
for(int i=0;i<100;i++) {
WebResource wr = mClient.resource("http://www.example.com");
ClientResponse result = wr.get(ClientResponse.class);
String strResult = result.getEntity(String.class);
System.out.println(strResult);
}
}
catch(Exception e) {
e.printStackTrace();
}
}
}
What is going on with Python that causes this code to fail? Is there an alternative way to load the page that might work?
I am trying to host a local server (using Node.js) on a Raspberry Pi. The Pi has an ADC (MCP3008) connected to it, and I have a Python script that continuously samples the ADC and prints the current value. I want to have the Node server run the Python script, and whenever it sees a print statement, to just do a console.log(current value) for the time being. I am new to Node and web development in general, so it may be something simple that I'm missing so that Node will continuously receive data from the Python script. I'm trying to use Socket.io at the moment, as that seems to make sense as the method for Node to see changes from the Python script, but maybe this isn't the best way to do it. The basic webpage is from a tutorial I found (http://www.jaredwolff.com/blog/raspberry-pi-getting-interactive-with-your-server-using-websockets/). The code I am currently using is here:
var app = require('http').createServer(handler)
, io = require('socket.io').listen(app)
, url= require('url')
, fs = require('fs')
, gpio = require('onoff').Gpio
, PythonShell = require('python-shell');
app.listen(5000);
function handler (req, res) {
var path = url.parse(req.url).pathname;
if (path == '/') {
index = fs.readFile(__dirname+'/public/index.html',
function(error,data) {
if (error) {
res.writeHead(500);
return res.end("Error: unable to load index.html");
}
res.writeHead(200,{'Content-Type': 'text/html'});
res.end(data);
});
} else if( /\.(js)$/.test(path) ) {
index = fs.readFile(__dirname+'/public'+path,
function(error,data) {
if (error) {
res.writeHead(500);
return res.end("Error: unable to load " + path);
}
res.writeHead(200,{'Content-Type': 'text/plain'});
res.end(data);
});
} else {
res.writeHead(404);
res.end("Error: 404 - File not found.");
}
}
// Python
var pyshell = new PythonShell('mcp3008.py');
pyshell.run('mcp3008.py', function (err, results) {
if (err) throw err;
console.log('Results: %j', results);
});
io.sockets.on('connection', function (socket) {
pyshell.on('message', function (message) {
console.log(message);
});
});
Thank you for any hints or help that you can provide!
As jfriend00 recommended, I looked into node.js solutions. I had previously tried this, using several mcp3008 packages available on npm, but none of them successfully installed on my Raspberry Pi (model B). However, I ended up rewriting the one located here (https://github.com/fiskeben/mcp3008.js) as a separate .js file, included it with my code (along with some work from the npm spi library), and put it into a loop to read the ADC pin. That's working for now, and should be good enough for my current needs, but it still seems like a more processor-intensive solution than it should be. Thanks for your feedback!