I'm running Caffe using python on AWS.
The scripts uses the GPU, uploads an existing model, and checks the output of it per image URL.
On the first few tries the script ran well. Than it stuck on a few different phases at each time.
Using 'top' I could see that ksoftirqd/0 gets about 93% of the CPU when the process is stuck.
I don't think there is a bug in my script, because originally it ran well no the server. When I reboot the server, sometimes it helps. But later we get the same problem.
Killing all python process on the server doesn't help. Only rebootting it.
Any ideas what I can do here?
It seems like you are experiencing a very high network load.
What exactly are you trying to download from URLs?
Are there any other processes running at the same time on the machine?
It is difficult to diagnose your problem without the specifics of your script.
Related
We are using Xvnc in order to execute UI tests. The tests are written in Python and utilizes Jenkins Pipeline structure.
Everything worked fine until we ported our slaves to containers instead of VMs. The test session times are identical, however, loading the Xvnc wrapper takes around 2 minutes to turn on and shut off.
I tried looking around the web and didn’t found any solution.
What I’m thinking is that the plug-in implementation does cause hiccups, as when I’m trying to run the startup command using shell script stage, it works quickly.
Anyone faces it?
Over the past few weeks I've been coding a program which runs a reddit bot locally from my machine. I've perfected it such that it does not reply to the same comment which it has replied to before, it runs quite efficiently, and in my opinion is complete.
Now, I'm looking for a way to get the program to run on a schedule. I currently have the code in google colab, and I don't know how to use google colab for this functionality.
The program does not require any local storage, it's one code file, and does not require much memory, so I wanted to ask if anyone has a resource which has an detailed tutorial accessible for beginners which I could use to host this code
Note: The code requires an installation of PRAW, in google colab I simply do !pip install PRAW if that means anything differently for what I need to do, what should I do differently?
Thank you in advance.
Google Collab is not designed for this kind of things, and most likely it cannot be used to run your app on schedule.
Probably the easiest solution is some kind of Continuous Integration tool that lets you run code remotely.
Step 1 would be to host your code to some remote code repository like GitHub. Since it most likely won't have to be interactive switching from Collab notebook to simple Python script will make your config much easier later on.
Step 2 would be connecting that repo to some CI tool. One I am familiar with that lets you run pipelines on schedule is CircleCI, with this tutorial here showing very simplistic configuration for running Python scripts from a pipeline.
I have a python script which I am able to run it locally from my machine and works fine. I am using spyder for coding. This python script needs to run daily at a certain time so I created a basic task under task scheduler.
The way I am doing this is in the action script I am putting my pythonw.exe and I am adding my python script as a action argument.
The thing is task scheduler triggers the task and script is not producing any outputs. I am reading a files from the shared drive in my native file.
I also tried to create the batch file and run the batch file but same issue.
Not sure what is wrong.
Can anyone help me with this?
I created the batch file and put that under the task sechduler and it worked fine. It looks like I had an issues with the access permissions as I have two accounts on my laptop.
Thanks Everyone for the help.
I've got a following setup:
VB.NET Web-Service is running and it needs to regularly call Python script with machine learning model to predict some stuff. To do this my Web-Service generates a file with input for Python and runs Python script as a subprocess. The script makes predictions and returns them, as standard output, back to Web-Service.
The problem is, that the script requires a few seconds to import all the machine learning libraries and load saved model from drive. It's much more than doing actual prediction. During this time Web-Service is blocked by running subprocess. I have to reduce this time drastically.
What I need is a solution to either:
1. Improve libraries and model loading time.
2. Communicate Python script with VB.NET Web-Service and run Python all the time with imports and ML model already loaded.
Not sure I understood the question but here are some things I can think of.
If this is a network thing you should have python compress the code before sending it over the web.
Or if you're able to, use multithreading while reading the file from the web.
Maybe you should upload some more code so we can help you better.
I've found what I needed.
I've used web.py to convert the Python script into a Web-Service and now both VB.NET and Python Web-Services can communicate. Python is running all the time, so there is no delay for loading libraries and data each time a calculation have to be done.
I have written a python program with "csv" file as backend. Actually the algorithm which is involved performs long and deep numerical computations.So I am not able to run it on my local computer.Last time,when I ran it,it took more than 2 hrs but didn't fully execute.Can anyone tell me Is there any way of executing the program with csv file as backend on some servers?
Or is there any other way to do it?
Also,I don't want to use spark for the purpose.