port conflict between Hadoop and python - python

I am installing Hadoop 2.5.0 on a Ubuntu 12.04 cluster, 64-bit. At the end of the instructions I type $ jps on the master node and do not get a NameNode. I checked the Hadoop logs and found:
BindException error stating :9000 is already in use.
$ netstat -a -t --numeric-ports -p | grep :9000 returns that python is listening on this port. It appears I need to move python 2.7 to another port. How do I move python?
Followed the command below, the pid=2346.
$ ps -p 2346
PID TTY TIME CMD
2346 ? 01:28:13 python
Tried second command:
$ ps -lp 2346
F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD
4 S 0 2346 1 0 80 0 - 332027 poll_s ? 01:28:30 python
more detail:
$ ps -Cp 2346
PID TTY STAT TIME COMMAND
2346 ? Ssl 88:34 /usr/lib/cmf/agent/build/env/bin/python /usr/lib/cmf/agent/src/cmf/agent.py --package_dir /usr/lib/cmf
It appears a failed Cloudera Hadoop distribution installation has not been removed. It installed python 2.7 automatically. Not sure what else is automatically running. Will attempt to uninstall python 2.7.

To be clear a program written in python is using port 9000 and not python2.7 itself.
You need to track down this program and then work out how to config it to listen on a different port. You could use this command to get the full details of the process listening on port 9000
netstat -a -t --numeric-ports -p | grep :9000 | awk '{print $7}' | sed -e 's/\/.*//' | xargs echo ps -lp

It appears Cloudera installed python 2.7. This was removed / replace with python 3.2.
The $jps command on Hadoop now returns the expected results including NameNode.

Related

Finding on which port a given server is running within python program

I am developing a python 3.11 program which will run on a few different servers and needs to connect to the local Redis server. On each machine the latter might run on a different port, sometimes the default 6379 but not always.
On the commandline I can issue the following command which on both my Linux and MacOS servers works well:
(base) bob#Roberts-Mac-mini ~ % sudo lsof -n -i -P | grep LISTEN | grep IPv4 | grep redis
redis-ser 60014 bob 8u IPv4 0x84cd01f56bf0ee21 0t0 TCP *:9001 (LISTEN)
What's the better way to get the running port using python functions/libraries?
What if you run your commands within a py script using the os library:
import os
cmd = 'ls -l' <-- change the command you want to run
os.system(cmd)
or else you could also use subprocess library as well:
import subprocess
print(subprocess.check_output(['ls', '-l']))

How to fix 8000 port is already in use django runserver problem? [duplicate]

Restarting the Django server displays the following error:
this port is already running....
This problem occurs specifically on Ubuntu and not other operating systems. How can I free up the port to restart the server?
A more simple solution just type sudo fuser -k 8000/tcp.
This should kill all the processes associated with port 8000.
EDIT:
For osx users you can use sudo lsof -t -i tcp:8000 | xargs kill -9
netstat -ntlp
It will show something like this.
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:8000 0.0.0.0:* LISTEN 6599/python
tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN -
tcp 0 0 192.168.124.1:53 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN -
tcp6 0 0 :::3306 :::* LISTEN
So now just close the port in which Django/python running already by killing the process associated with it.
kill -9 PID
in my case
kill -9 6599
Now run your Django app.
ps aux | grep -i manage
after that you will see all process
ubuntu#ip-10-154-22-113:~/django-apps/projectname$ ps aux | grep -i manage
ubuntu 3439 0.0 2.3 40228 14064 pts/0 T 06:47 0:00 python manage.py runserver project name
ubuntu 3440 1.4 9.7 200996 59324 pts/0 Tl 06:47 2:52 /usr/bin/python manage.py runserver project name
ubuntu 4581 0.0 0.1 7988 892 pts/0 S+ 10:02 0:00 grep --color=auto -i manage
kill -9 process id
e.d kill -9 3440
`enter code here`after that :
python manage.py runserver project name
By default, the runserver command starts the development server on the internal IP at port 8000.
If you want to change the server’s port, pass it as a command-line argument. For instance, this command starts the server on port 8080:
python manage.py runserver 8080
lsof -t -i tcp:8000 | xargs kill -9
Sorry for comment in an old post but It may help people
Just type this on your terminal
killall -9 python3
It will kill all python3 running on your machine and it will free your all port. Greatly help me when to work in Django project.
We don't use this command { sudo lsof -t -i tcp:8000 | xargs kill -9 } Because it's close all tabs...You should use to
ps -ef | grep python
kill -9 process_id
ps -ef | grep python (show all process with id)
kill -9 11633
(11633 is a process id to :- /bin/python manage.py runserver)
>> ps aux | grep manage
ubuntu 3438 127.0.0 2.3 40256 14064 pts/0 T 06:47 0:00 python manage.py runserver
>> kill -9 3438
Type 'fg' as command after that Ctrl-C.
Command:
Fg will show which is running on background. After that Ctrl-C will stop it.
fg
ctl-c
In terminal, Type ps aux | grep runserver
Hit Enter
Use PID among the result execute kill -9 <PID>
For an instance, If result of step 1 is as follow
root 1041 0.0 0.1 266912 34580 pts/3 S+ 11:31 0:01 python3 manage.py runserver 0.0.0.0:3030
root 1696 4.5 0.1 126128 40708 ? S Feb14 925:43 /usr/local/bin/python manage.py runserver 0.0.0.0:8000
1041 and 1696 are PIDs. We need to choose whichever process we want to kill among them.
This is an expansion on Mounir's answer. I've added a bash script that covers this for you. Just run ./scripts/runserver.sh instead of ./manage.py runserver and it'll work exactly the same way.
#!/bin/bash
pid=$(ps aux | grep "./manage.py runserver" | grep -v grep | head -1 | xargs | cut -f2 -d" ")
if [[ -n "$pid" ]]; then
kill $pid
fi
fuser -k 8000/tcp
./manage.py runserver
For me, this happens because my API request in Postman is being intercepted by a debugger breakpoint in my app... leaving the request hanging. If I cancel the request in Postman before killing my app's server, the error does not happen in the first place.
--> So try cancelling any open requests you are making in other programs.
On macOS, I have been using sudo lsof -t -i tcp:8000 | xargs kill -9 when I forget to cancel the open http request in order to solve error = That port is already in use. This also, complete closes my Postman app, which is why my first solution is better.
Click the arrow in the screenshot and find the bash with already running Django server. You were getting the message because your server was already running and you tried to start the server again.
Dont use CTRL + Z to stop server, use CTRL + C to stop the server, I had also had the same problem in my linux (fedora) , I used to stop the server using CTRL + Z and again I used to kill the server using sudo fuser -k 8000/tcp command, which worked fine. But later when I started using CTRL + C , I didnot get that port running issue anymore.
if you have face this problem in mac you just need to open activity monitor and force quite python then try again
In case You are using the VSC's screen terminal, The error might be due to the fact that you already runserver in some other shell.
Just click on the dropbox on the left of the + sign in the header of the terminal of VSC and select some other shell and check if the server is already running there. Quit that server and you are ready to launch a another server.
I was trying all the solutions but they were not working i suggest you to keep press the power button or if your battery is removeable then remove it all the process will be killed and your local host will be reset

Start a remote script from a Mac OS X machine via SSH command

I am trying to start a python script on my VM from my local Mac OS
I did
ssh -i /key/path/id_rsa root#111.11.1.0 "sleep 5s; cd /root/Server;pkill -f server.py;./server.py;"
Result
It's SSH in and it quickly runs those commands and it quickly logging me out. I was expecting it to stay open in SSH session.
My script is NOT running ...
ps -aux | grep python
root 901 0.0 0.2 553164 18584 ? Ssl Jan19 20:37 /usr/bin/pytho -Es /usr/sbin/tuned -l -P
root 15444 0.0 0.0 112648 976 pts/0 S+ 19:16 0:00 grep --color=auto python
If I do this it works
ssh -i /key/path/id_rsa root#111.11.1.0 "sleep 5s; cd /root/Server"
Then
./server.py;
Then, it works.
Am I missing anything?
You might need to state the shell starting your script i.e /bin/bash server.py:
ssh -i /key/path/id_rsa root#111.11.1.0 "sleep 5s; cd /root/Server; pkill -f server.py; /bin/bash ./server.py;"
If you would like to start the script and leave it running even after you end your ssh session you could use nohup. Notice that you need to put the process in the background and redirect stdin, stdout and stderr to completly detach from the remote process:
-i /key/path/id_rsa root#111.11.1.0 "sleep 5s; cd /root/Server; nohup /bin/bash ./server.py < /dev/null > std.out 2> std.err &"
It seems like the reason that your ssh command returns imediately is because somehow the call to pkill -f server.py will also terminate the actual ssh session, since it also contains the server.py in the commandline.
I don't have my regular MacBook Pro here to test with, but I think that adding another semicolon and ending the command line with /bin/bash might do it.

Identifying local processes and associated ports with python

Does python or any linux/unix languages have a library which allows it to find ports available on the localhost based on processes name?
I would like to identify all associated ports with the service name ssh.
lsof, netstat ps or grep don't seem the most efficient options.
This got me what I needed, thank you.
ss -l -p -n | grep ",pid=`pgrep ssh`," | grep ::: | cut -d : -f 4

How to kill Django runserver sub processes from a bash script?

I'm working on a Django website where I have various compilation programs that need to run (Compass/Sass, coffeescript, hamlpy), so I made this shell script for convenience:
#!/bin/bash
SITE=/home/dev/sites/rmx
echo "RMX using siteroot=$SITE"
$SITE/rmx/manage.py runserver &
PIDS[0]=$!
compass watch $SITE/media/compass/ &
PIDS[1]=$!
coffee -o $SITE/media/js -cw $SITE/media/coffee &
PIDS[2]=$!
hamlpy-watcher $SITE/templates/hamlpy $SITE/templates/templates &
PIDS[3]=$!
trap "echo PIDS: ${PIDS[*]} && kill ${PIDS[*]}" SIGINT
wait
Everything except for the Django server shuts down nicely on a ctrl+c because the PID of the server process isn't the PID of the python manage.py runserver command. Which means everytime I stop the script, I have to find the running process PID and shut it down.
Here's an example:
$> ./compile.sh
RMX using siteroot....
...
[ctrl+c]
PIDS: 29725 29726 29728 29729
$> ps -A | grep python
29732 pts/2 00:00:00 python
The first PID, 29725, is the initial python manage.py runserver call, but 29732 is the actual dev server process.
edit Looks like this is due to Django's auto-reload feature which can be disabled with the --noreload flag. Since I'd like to keep the auto reload feature, the question now becomes how to kill the child processes from the bash script. I would think killing the initial python runserver command would do it...
SOLVED
Thanks to this SO question, I've changed my script to this:
#!/bin/bash
SITE=/home/dev/sites/rmx
echo "RMX using siteroot=$SITE"
$SITE/rmx/manage.py runserver &
compass watch $SITE/media/compass/ &
coffee -o $SITE/media/js -cw $SITE/media/coffee &
hamlpy-watcher $SITE/templates/hamlpy $SITE/templates/templates &
trap "kill -TERM -$$" SIGINT
wait
PIDs preceded with the dash operate on the PID group with the kill command, and the $$ references the PID of the bash script itself.
Thanks for the help, me!
No problem, self, and hey -- you're awesome.
You can execute this to kill or process and servers, you set PORT number:
$ netstat -tulpn | grep PORT | awk '{print $7}' | cut -d/ -f 1 | xargs kill
OR
$ sudo lsof -i tcp:PORT
$ sudo lsof -i tcp:PORT|awk '{print $2}'|cut -d/ -f 1|xargs kill

Categories