I created a simple python app on Heroku to launch scrapyd. The scrapyd service starts, but it launches on port 6800. Heroku requires you to bind it the $PORT variable, and I was able to run the heroku app locally. The logs from the process are included below. I looked at a package scrapy-heroku, but wasn't able to install it due to errors. The code in app.py of this package seems to provide some clues as to how it can be done. How can I implement this as a python command to start scrapyd on the port provided by Heroku?
Procfile:
web: scrapyd
Heroku Logs:
2022-01-24T05:17:27.058721+00:00 app[web.1]: 2022-01-24T05:17:27+0000 [twisted.scripts._twistd_unix.UnixAppLogger#info] twistd 21.7.0 (/app/.heroku/python/bin/python 3.10.2) starting up.
2022-01-24T05:17:27.058786+00:00 app[web.1]: 2022-01-24T05:17:27+0000 [twisted.scripts._twistd_unix.UnixAppLogger#info] reactor class: twisted.internet.epollreactor.EPollReactor.
2022-01-24T05:17:27.059190+00:00 app[web.1]: 2022-01-24T05:17:27+0000 [-] Site starting on 6800
2022-01-24T05:17:27.059301+00:00 app[web.1]: 2022-01-24T05:17:27+0000 [twisted.web.server.Site#info] Starting factory <twisted.web.server.Site object at 0x7f1706e3eaa0>
2022-01-24T05:17:27.059649+00:00 app[web.1]: 2022-01-24T05:17:27+0000 [Launcher] Scrapyd 1.3.0 started: max_proc=32, runner='scrapyd.runner'
2022-01-24T05:18:25.204305+00:00 heroku[web.1]: Error R10 (Boot timeout) -> Web process failed to bind to $PORT within 60 seconds of launch
2022-01-24T05:18:25.231596+00:00 heroku[web.1]: Stopping process with SIGKILL
2022-01-24T05:18:25.402503+00:00 heroku[web.1]: Process exited with status 137
You just need to read the PORT environment variable and write it into your scrapyd config file. You can check out this code that does the same.
# init.py
import os
import io
PORT = os.environ['PORT']
with io.open("scrapyd.conf", 'r+', encoding='utf-8') as f:
f.read()
f.write(u'\nhttp_port = %s\n' % PORT)
Source: https://github.com/scrapy/scrapyd/issues/367#issuecomment-591446036
Related
I'm running Django web app on a docker container where I use Nginx with uwsgi.
Overall the web works just fine it fails only on specific callback endpoints during the social app (Google, Facebook) registration.
Below is the command I use to run the uswgi
uwsgi --socket :8080 --master --strict --enable-threads --vacuum --single-interpreter --need-app --die-on-term --module config.wsgi
Below is the endpoint where it fails (Django allauth lib)
accounts/google/login/callback/?state=..........
Below is the error message:
!! uWSGI process 27 got Segmentation Fault !!!
...
upstream prematurely closed connection while reading response header from upstream, client: ...
...
DAMN ! worker 1 (pid: 27) died :( trying respawn ...
Respawned uWSGI worker 1 (new pid: 28)
Just FYI.. this works without any issues in the local docker container but it fails when I use the GCP container. Also, this used to work fine on GCP as well so probably something happened after recent dependency updates.
Environment:
Python: 3.9.16
Django: 3.2.3
allauth: 0.44.0 (Django authentication library)
Nginx: nginx/1.23.3
uWSGI: 2.0.20
I'm trying to deploy the following python app on an heroku server:
https://github.com/PX4/flight_review
The main server script is called ./serve.py, you will find it under:
https://github.com/PX4/flight_review/blob/master/serve.py
It's a tornado web server, but I m not sure on how to configure it, on my local machine I respond to the url http://localhost:5006/
In added a Procfile as follow:
web: ./serve.py --host 0.0.0.0:5006
and also tried with and without other arguments for the port, and also added a PORT variable into the env variable, but I constantly get the following error:
Error R10 (Boot timeout) -> Web process failed to bind to $PORT within 60 seconds of launch
Any idea on what could go wrong?
Thanks in advance!
Following the documentation about Procfiles:
You have to just use the $PORT environment variable, which is set by Heroku when trying to start your dyno.
In your case that would seem to be:
web: ./serve.py --host 0.0.0.0:$PORT
I readed a lot of questions here in stackoverflow, but I cannot solve my problem.
I have a simples python script (Only a print('Hello World')) and I'm sending it to Heroku, my files are:
Procfile: (I believe the problem is here)
web: python index.py
heroku logs --tail
2020-05-06T16:06:38.000000+00:00 app[api]: Build started by user <myemail>
2020-05-06T16:07:00.948412+00:00 heroku[web.1]: State changed from crashed to starting
2020-05-06T16:07:00.572714+00:00 app[api]: Release v11 created by user <myemail>
2020-05-06T16:07:00.572714+00:00 app[api]: Deploy 4053de7f by user <myemail>
2020-05-06T16:07:07.099124+00:00 heroku[web.1]: State changed from starting to crashed
2020-05-06T16:07:07.103524+00:00 heroku[web.1]: State changed from crashed to starting
2020-05-06T16:07:07.002972+00:00 app[web.1]: Hello World
2020-05-06T16:07:08.000000+00:00 app[api]: Build succeeded
2020-05-06T16:07:13.797877+00:00 heroku[web.1]: State changed from starting to crashed
2020-05-06T16:07:13.619779+00:00 app[web.1]: Hello World
heroku ts
Free dyno hours quota remaining this month: 550h 0m (100%)
For more information on dyno sleeping and how to upgrade, see:
https://devcenter.heroku.com/articles/dyno-sleeping
=== web (Free): python index.py (1)
web.1: crashed 2020/05/06 13:07:13 -0300 (~ 4m ago)
How we can see, the "Hello World" in log shows that the script runs when I send it to server. But after it, the dyno crashes
in case of Web dyno the application must bind to the port provided by Heroku and defined in the $PORT env variable, where it will receive the incoming requests.
If you need to HTTP traffic try to create a worker node
For example to create a simple Flask app which can receive requests:
from flask import Flask
import os
import logging
try:
app = Flask(__name__)
except Exception as e:
logging.exception("Error at startup")
#app.route('/test')
def test():
logging.info('/test')
return "test Ok"
if __name__ == '__main__':
app.run(debug=False, port=int(os.environ.get("PORT", 5000)), host='0.0.0.0')
Port is set using $PORT on Heroku while locally defaults to 5000
I have developed VK bot on language python and deploy to Heroku
after the command heroku ps:scale web=1, the script runs and works few seconds and after few seconds script stops proccess with log
Error R10 (Boot timeout) -> Web process failed to bind to $PORT within
60 seconds of launch on Python Process exited with status 137
That is my Procfile:
web: python bot.py
I changed the Procfile to: web: python bot.py -p $PORT, but It did not help
I decided to use python's subprocess where I'm invoking a particular command line program. It was working before but after starting my ec2 instance (from being shutdown), this is showing up in the command line:
[INFO] Handling signal: ttou
Here are the programs I've used.
nginx and gunicorn for my webserver, flask for my api, python 2.7.
Here's the "now problematic" line of code that used to work.
query = subprocess.check_output(couch_cmd, shell=True)
this is the value for couch_cmd:
couch_cmd = "/opt/couchbase/bin/cbq --script=\"select * from `user` where email='" + email + "'\""
This used to work before but after I stopped and started my ec2 instance, this keeps on appearing in the logs when I call my api.
[2017-10-18 02:13:39 +0000] [3324] [INFO] Handling signal: ttou
Note: I've also used the command above and executed it in the python shell. And it's working! I've already changed the appropriate config in nginx to point to my dns and also used proxy_pass. I think my nginx config is fine cause the request gets routed to my webserver. I think there's something in gunicorn that messes up the invocation of subprocess.check_output(). Please help!
I've read this article and realized my mistake:
https://www.computerhope.com/unix/signals.htm
The TTIN and TTOU signals are sent to a process when it attempts to
read or write respectively from the tty while in the background.
Typically, this signal can be received only by processes under job
control; daemons do not have controlling terminals and should never
receive this signal.
I've realized that I was starting gunicorn in the background. This must have prompted gunicorn to intercept the output of check_output() when I tried to log its result to the terminal, thereby intercepting the output and making my request timeout. I was using the output to in my response payload.
How I started gunicorn:
gunicorn app:app -b 127.0.0.1:8000 &
Solution:
gunicorn app:app -b 127.0.0.1:8000