We Keep Coding

How to get more concurrent requests from scrappy to accomplish a job faster?

I wrote scrapper that I need to test proxies. The functionality works well, but I want to make the job run faster. I've made a chain of experiments increasing CONCURRENT_REQUESTS and quickly reaching saturation - can't get less than 0.06s for one item(proxy + url tuple). It might be that I'm doing something wrong, and there are other settings?
proxy+url pairs
job duration(min)
CONCURRENT_REQUESTS
time per pair(seconds)
1000
2
16
0.12
5000
6
100
0.072
5000
6.5
1500
0.078
5000
5
5500
0.06
5000
6
40000
0.072
$ scrapy version
Scrapy 2.7.0
$ python -V
Python 3.10.8
(
$ uname -a
Darwin <host>.local 21.6.0 Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64 x86_64 i386 Darwin

Python script not working when launched at startup of RPi

RPi 1B with V1 cam.
Python script takes a picture when pushbutton hooked to gpio is pressed. Then picture is emailed via Mutt.
All working fine when doing step by step.
But not doing as intended when lauched automatically at startup.
import subprocess
from datetime import datetime
from gpiozero import Button
button = Button(17)
while True:
button.wait_for_press()
time = datetime.now()
filename = "capture-%04d%02d%02d-%02d%02d%02d.jpg" % (time.year, time.month, time.day, time.hour, time.minute, time.second)
subprocess.call("raspistill -t 500 -o %s" % filename, shell=True)
subprocess.call("echo "" | mutt -s 'Someone at the door' -i messageBody.txt myname#mailprovider.com -a %s" % filename, shell=True)
All working fine when typing :
$ python raspicam.py
I get a nice email within seconds with picture attached to it.
Next logical step is to get this script to be launched at startup:
$ nano launcher.sh
#!/bin/sh
# launcher.sh
cd /
cd home/pi
python doorbell02.py
cd /
$ chmod 755 launcher.sh
$ sh launcher.sh
Then get it to be launched at startup via cron :
$ mkdir logs
$ sudo crontab -e
add: #reboot sh /home/pi/launcher.sh >/home/pi/logs/cronlog 2>&1
At next reboot all working fine except sending mail via with mutt.
$ ps aux shows that my python script and script launcher belongs to "root"... is it where trouble comes from ?
root 475 0.0 0.0 0 0 ? S 16:51 0:00 [cifsd]
root 500 0.0 0.6 7932 2300 ? Ss 16:51 0:00 /usr/sbin/cron -f
root 502 0.0 0.6 9452 2384 ? S 16:51 0:00 /usr/sbin/CRON -f
root 506 0.0 0.3 1924 1148 ? Ss 16:51 0:00 /bin/sh -c sh /home/pi/launcher.sh >/home/pi/logs/cronlog 2>&1
root 511 0.0 0.2 1924 1108 ? S 16:51 0:00 sh /home/pi/launcher.sh
root 513 1.5 2.5 34348 9728 ? Sl 16:51 4:25 python doorbell02.py
I am also unable to get pdb to work alongside with my script to get some log or debug info...
Some hints would be very appreciated
Thank you very much for your time

Try using absolute paths in your code.
It helped me in my case.

Do Processes take up memory after it has finished running?

I am write an operational script which starts processes at specific times of the day. The script will be running constantly, as such I am afraid that the creation and starting of processes will take up memory.
I write a short script to see what happens to the Process after it has finished running the function.
# t.py
import time
import multiprocessing as mp
def f(t=0):
time.sleep(t)
a = mp.Process(target=f, args=(10, ))
a.start()
print(a.pid)
time.sleep(10000)
$ python3 t.py
32263
I then run the following to see what processes are running
$ ps auxf | grep python t.py
user 32262 3.0 0.1 21360 11200 pts/0 S+ 11:37 0:00 | \_ python3 t.py
user 32263 0.0 0.0 21360 7704 pts/0 S+ 11:37 0:00 | \_ python3 t.py
This shows that a is still present despite having finished running. I found that the pid 32263 only disappears if i include a.join() in the script. But this would block me from starting and creating other processes.
Is there any way to kill a for good? a.kill() and a.terminate() does not seem to do the job.
If I were to neglect the fact that a still appears in ps auxf, and I were to keep on creating and starting new processes, would that take up memory?

Hot reloading python process for code reload

Any way to hot reload python modules for a running python process? In usual cases we could run kill -HUP <pid> for some of the servers like squid, nginx,gunicorn. My running processes are
root 6 0.6 0.9 178404 39116 ? S 14:21 0:00 python3 ./src/app.py --config ./conf/config.yml
root 7 0.0 1.0 501552 43404 ? Sl 14:21 0:00 python3 ./src/app.py --config ./conf/config.yml
root 8 0.0 1.0 501808 43540 ? Sl 14:21 0:00 python3 ./src/app.py --config ./conf/config.yml

Is the question about reloading a Sanic app? If yes, then there is a hot reload built into the server.
app.run(debug=True)
Or if you want the reload without debugging
app.run(auto_reload=True)
See docs
Or, if this is a question in general, checkout aoiklivereload

CPU usage difference between ps aux and -ef

[user#centos-vm-02 ~]$ ps aux|grep python
user 4182 0.0 0.0 9228 1080 ? Ss 02:00 0:00 /bin/sh -c cd data/trandata && /usr/local/bin/python2.7 main.py >> /dev/null 2>&1
user 4190 0.1 0.1 341108 10740 ? Sl 02:00 0:52 /usr/local/bin/python2.7 main.py
user 4205 166 1.6 1175176 129312 ? Sl 02:00 901:39 /usr/local/bin/python2.7 main.py
user 10049 0.1 0.1 435856 10712 ? Sl 10:21 0:04 /usr/local/bin/python2.7 main.py
user 10051 71.1 2.5 948248 207628 ? Sl 10:21 28:42 /usr/local/bin/python2.7 main.py
user 10052 51.9 1.9 948380 154688 ? Sl 10:21 20:57 /usr/local/bin/python2.7 main.py
user 10053 85.9 0.9 815104 76652 ? Sl 10:21 34:41 /usr/local/bin/python2.7 main.py
user 11166 0.0 0.0 103240 864 pts/1 S+ 11:01 0:00 grep python
[user#centos-vm-02 ~]$ ps -ef|grep python
user 4182 4174 0 02:00 ? 00:00:00 /bin/sh -c cd /data/trandata && /usr/local/bin/python2.7 main.py >> /dev/null 2>&1
user 4190 4182 0 02:00 ? 00:00:52 /usr/local/bin/python2.7 main.py
user 4205 4190 99 02:00 ? 15:01:46 /usr/local/bin/python2.7 main.py
user 10049 1 0 10:21 ? 00:00:04 /usr/local/bin/python2.7 main.py
user 10051 10049 71 10:21 ? 00:28:47 /usr/local/bin/python2.7 main.py
user 10052 10049 51 10:21 ? 00:21:01 /usr/local/bin/python2.7 main.py
user 10053 10049 85 10:21 ? 00:34:45 /usr/local/bin/python2.7 main.py
user 11168 10904 0 11:01 pts/1 00:00:00 grep python
As we see, I launch a python process that it would spwan multiprocess, and inside the processes, multithreads are started, and inside the threads, multithreads are started.
Process tree like this:
main_process
--sub_process
----thread1
------sub_thread
------sub_thread
------sub_thread
------sub_thread
----thread2
----thread3
--sub_process
----......
Inside the picture, the pid-4205 shows different CPU usage in ps aux and ps -ef, one is 166, and the other is 99, 166 was also shown in top -c.
And I assure that the pid-4205 is one of the sub processes, which means it could not use more than 100% of CPU with GIL in python.
So that's my question, why ps -ef and ps aux show difference.

It's just a sampling artifact. Say a factory produces one car per hour. If you get there right before a car is made and leave right after a car is made, you can see two cars made in a span of time just over an hour, resulting in you thinking the factory is operating at near double capacity.
Update: Let me try to clarify the example. Say a factory produces one car per hour, on the hour. It is incapable of producing more than one car per hour. If you watch the factory from 7:59 to 9:01, you will see two cars produced (one at 8:00 and one at 9:00) in just over one hours (62 minutes). So you would estimate the factory produces about two cars per hour, nearly double its actual production. That is what happened here. It's a sampling artifact caused by top checking the CPU counters at just the wrong time.