We Keep Coding

function serving deployment failed

Here, I'm attaching actual error showed. im using mlrun with docker. specifically mlrun 1.2.0.
--------------------------------------------------------------------------
RunError Traceback (most recent call last)
<ipython-input-20-aab97e08b914> in <module>
1 serving_fn.with_code(body=" ") # adds the serving wrapper, not required with MLRun >= 1.0.3
----> 2 project.deploy_function(serving_fn)
/opt/conda/lib/python3.8/site-packages/mlrun/projects/project.py in deploy_function(self, function, dashboard, models, env, tag, verbose, builder_env, mock)
2307 :param mock: deploy mock server vs a real Nuclio function (for local simulations)
2308 """
-> 2309 return deploy_function(
2310 function,
2311 dashboard=dashboard,
/opt/conda/lib/python3.8/site-packages/mlrun/projects/operations.py in deploy_function(function, dashboard, models, env, tag, verbose, builder_env, project_object, mock)
344 )
345
--> 346 address = function.deploy(
347 dashboard=dashboard, tag=tag, verbose=verbose, builder_env=builder_env
348 )
/opt/conda/lib/python3.8/site-packages/mlrun/runtimes/serving.py in deploy(self, dashboard, project, tag, verbose, auth_info, builder_env)
621 logger.info(f"deploy root function {self.metadata.name} ...")
622
--> 623 return super().deploy(
624 dashboard, project, tag, verbose, auth_info, builder_env=builder_env
625 )
/opt/conda/lib/python3.8/site-packages/mlrun/runtimes/function.py in deploy(self, dashboard, project, tag, verbose, auth_info, builder_env)
550 self.status = data["data"].get("status")
551 self._update_credentials_from_remote_build(data["data"])
--> 552 self._wait_for_function_deployment(db, verbose=verbose)
553
554 # NOTE: on older mlrun versions & nuclio versions, function are exposed via NodePort
/opt/conda/lib/python3.8/site-packages/mlrun/runtimes/function.py in _wait_for_function_deployment(self, db, verbose)
620 if state != "ready":
621 logger.error("Nuclio function failed to deploy", function_state=state)
--> 622 raise RunError(f"function {self.metadata.name} deployment failed")
623
624 #min_nuclio_versions("1.5.20", "1.6.10")
RunError: function serving deployment failed
I don't have any idea what is the reason behind this error. as I'm new bee here. so someone pls help me to resolve this error.

I see two steps, how to solve the issue:
1. Relevant installation
The MLRun Community Edition in desktop docker has to be install under relevant HOST_IP (not with localhost or 127.0.0.1, but with stable IP address, see ipconfig) and with relevant SHARED_DIR. See relevant command line (from OS windows):
set HOST_IP=192.168.0.150
set SHARED_DIR=c:\Apps\mlrun-data
set TAG=1.2.0
mkdir %SHARED_DIR%
docker-compose -f "c:\Apps\Desktop Docker Tools\compose.with-jupyter.yaml" up
BTW: YAML file see https://docs.mlrun.org/en/latest/install/local-docker.html
2. Access to the port
In case of call serving_fn.invoke you have to open relevant port (from deploy_function) on your IP address (based on setting of HOST_IP, see the first point).
Typically this port can be blocked based on your firewall policy or your local antivirus. It means, you have to open access to this port before invoke call.
BTW: You can see focus on the issue https://github.com/mlrun/mlrun/issues/2102

pysipp - Trying to use it with existing sipp conf file

Background
I have an existing sipp conf file that I launch like so:
sipp mysipdomain.net -sf ./testcall.conf -m 1 -s 12345 -i 10.1.1.1:5060
This runs just fine. It simulates a call in our test labs. But now I need to expand this test to make it a part of a larger test script where not only do I launch the sipp test, but I prove (via sip trace) that it's hitting the right boxes.
I decided to wrap this sipp call in python. I just found https://github.com/SIPp/pysipp and am trying to see if I can write this entire test in python. To start, i tried to run the same sipp test using pysipp.
Problem / Question
I'm currently getting an error that says:
lab2:/tmp/jj/sipp_tests# python mvv_numeric.py
No handlers could be found for logger "pysipp"
Traceback (most recent call last):
File "mvv_numeric.py", line 6, in <module>
uac()
File "/usr/lib/python2.7/site-packages/pysipp-0.1.alpha-py2.7.egg/pysipp/agent.py", line 71, in __call__
raise_exc=raise_exc, **kwargs
File "/usr/lib/python2.7/site-packages/pluggy-0.3.1-py2.7.egg/pluggy.py", line 724, in __call__
return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
File "/usr/lib/python2.7/site-packages/pluggy-0.3.1-py2.7.egg/pluggy.py", line 338, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "/usr/lib/python2.7/site-packages/pluggy-0.3.1-py2.7.egg/pluggy.py", line 333, in <lambda>
_MultiCall(methods, kwargs, hook.spec_opts).execute()
File "/usr/lib/python2.7/site-packages/pluggy-0.3.1-py2.7.egg/pluggy.py", line 596, in execute
res = hook_impl.function(*args)
File "/usr/lib/python2.7/site-packages/pysipp-0.1.alpha-py2.7.egg/pysipp/__init__.py", line 250, in pysipp_run_protocol
finalize(cmds2procs, raise_exc=raise_exc)
File "/usr/lib/python2.7/site-packages/pysipp-0.1.alpha-py2.7.egg/pysipp/__init__.py", line 228, in finalize
raise SIPpFailure(msg)
pysipp.SIPpFailure: Some agents failed
'uac' with exit code 255 -> Command or syntax error: check stderr output
Code
Here's what the py script looks like:
1 import pysipp
2 uac = pysipp.client(destaddr=('mysipdomain.net', 5060))
3 uac.uri_username = '12345'
4 uac.auth_password = ''
5 uac.scen_file = './numeric.xml'
6 uac()
And the original sipp "testcall.conf" has been renamed to "numeric.xml" and looks like this: (I'm only including the first part because it's quite long. if you need to see something specific, please let me know and I will add to this post)
<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE scenario SYSTEM "sipp.dtd">
<scenario name="UAC with Media">
<send retrans="10000">
<![CDATA[
INVITE sip:[service]#[remote_ip]:[remote_port] SIP/2.0
Via: SIP/2.0/[transport] [local_ip]:[local_port];branch=[branch]
From: sipp <sip:sipp#[local_ip]:[local_port]>;tag=[pid]SIPpTag00[call_number]
To: [service] <sip:[service]#[remote_ip]:[remote_port]>
Call-id: [call_id]
CSeq: 1 INVITE
Contact: <sip:sipp#[local_ip]:[local_port]>
Allow: INVITE, ACK, BYE, CANCEL, OPTIONS, INFO, MESSAGE, SUBSCRIBE, NOTIFY, PRACK, UPDATE, REFER
User-Agent: PolycomVVX-VVX_300-UA/5.5.2.8571
Accept-Language: en
Supported: replaces,100rel
Allow-Events: conference,talk,hold
Max-Forwards: 70
Content-Type: application/sdp
Content-Length: [len]
I'm sure it's something simple I've overlooked. Any pointers would be appreciated.
EDIT:
I added debug level logging and reran the python script. In the logs I can now see what pysipp is actually attempting:
2018-01-31 14:40:32,715 MainThread [DEBUG] pysipp launch.py:63 : launching cmd:
"'/usr/bin/sipp' 'mysipdomain.net':'5060' -s '12345' -sn 'uac' -sf 'numeric.xml' -recv_timeout '5000' -r '1' -l '1' -m '1' -log_file '/tmp/uac_log_file' -screen_file '/tmp/uac_screen_file' -trace_logs -trace_screen "
So comparing that with the original command line I use to run sipp, I see the extra "-sn 'uac'".
Going to see about either getting my SIPP script to work with that tag or ... google to see if I can find other similar posts.
In the meantime, if you see my mistake, i'm all ears.

The problem here (as you noticed) is likely that pysipp.client() sets the -sn uac flag and sipp fails having both -sn and -sf.
To see the actual error you can enable logging before running the client:
import pysipp
pysipp.utils.log_to_stderr("DEBUG")
uac = pysipp.client(destaddr=('mysipdomain.net', 5060))
uac.uri_username = '12345'
uac.auth_password = ''
uac.scen_file = './numeric.xml'
uac()
The hack is to simply do uac.scen_name = None but the proper way to do this is to either use pysipp.scenario() (docs here) and rename your numeric.xml to have uac in the file name (i.e. uac_numeric.xml) or use instead pysipp.ua(scen_file=<path/to/numeric.xml>).
To understand the problem the client is currently applying a default scenario name argument when really the user should be able to override that (though in that case there'll be no guarantee that the user actually is sending client traffic which renders the name client kind of pointless).

Python fabric, send notification on failure

I've been trying to figure out what the best way is to do something when my fabric script fails (for example send a slack notification message via python module slackbot).
I've made an example where I try to do the above here:
fab_failtest.py
my_slackclient.py
You can run above example by downloading both files to a directory, pip install fabric and slackbot, then run:
fab --fabfile=fab_failtest.py fail_test1
or
fab --fabfile=fab_failtest.py fail_test2
(you also have to have a machine you can ssh to, in this example I have mrbluesky#elo with open ssh port on 22)
fail_test1 uses try-except so I can get exception error info and so forth
fail_test2 uses try-finally plus a simple boolean variable so no exception info is available
At first I thought I had it with the fail_test1 example but I've seen it fail several times to send the slack message on failure, I'm wondering if there might be a race condition or something involved? I could start using fail_test2 instead but I really like to have access to the stack-trace like in fail_test1.
Is there a better way to do this, like, something provided in python fabric that does excatly what I'm trying to accomplish in above example?

I disagree with both your approaches. Im a strong believer that less code is better. What do i mean by that? A function should do what its name says, no more no less, if you have to add in a global handler like that i would add it in as a wrapper, fabric functions are difficult enough to read, no need to add error handling to the mix. With that said:
import sys
import traceback
from fabric.api import task, settings, local, abort
from fabric.decorators import _wrap_as_new
from functools import wraps
HOST = 'elo'
PORT = 22
def alert_on_fail(func):
#wraps(func)
def decorated(*args, **kwargs):
try:
return func(*args, **kwargs)
except:
# TODO: add more code here
exception_type, value, tb_msg = sys.exc_info()
traceback_msg = traceback.format_exc()
notify('something went wrong: ' + traceback_msg)
abort('exiting error!!')
return _wrap_as_new(func, decorated)
#task
#alert_on_fail
def fail_test(host=HOST, port=PORT):
notify('fail test', msg_type='info')
local('''python -c "raise Exception('foobar')"''')
notify('script ran successfully', msg_type='success') # this will never run because the function above crashed
#task
#alert_on_fail
def pass_test(host=HOST, port=PORT):
notify('pass test', msg_type='info')
local('whoami')
notify('script ran successfully', msg_type='success')
def notify(msg, **kwargs):
# DISREGARD THIS
print 'sent to slack:', msg
Output:
$ fab fail_test
sent to slack: fail test
[localhost] local: python -c "raise Exception('foobar')"
Traceback (most recent call last):
File "<string>", line 1, in <module>
Exception: foobar
Fatal error: local() encountered an error (return code 1) while executing 'python -c "raise Exception('foobar')"'
Aborting.
sent to slack: something went wrong: Traceback (most recent call last):
File "/private/tmp/fabfile.py", line 21, in decorated
return func(*args, **kwargs)
File "/private/tmp/fabfile.py", line 34, in fail_test
local('''python -c "raise Exception('foobar')"''')
File "/usr/local/lib/python2.7/site-packages/fabric/operations.py", line 1198, in local
error(message=msg, stdout=out, stderr=err)
File "/usr/local/lib/python2.7/site-packages/fabric/utils.py", line 347, in error
return func(message)
File "/usr/local/lib/python2.7/site-packages/fabric/utils.py", line 53, in abort
sys.exit(msg)
SystemExit: local() encountered an error (return code 1) while executing 'python -c "raise Exception('foobar')"'
Fatal error: exiting error!!
Aborting.
exiting error!!
and:
$ fab pass_test
sent to slack: pass test
[localhost] local: whoami
buzzi
sent to slack: script ran successfully
Done.
You'll notice that the functions are now "easy" to read, they are "simple", all the error handling code has been moved over somewhere else.

Can't connect to MongoDB with cgi script

This works fine from the command line, but not through the web:
#!/usr/bin/python
# -*- coding: utf-8 -*-
import cgitb
cgitb.enable()
from pymongo import Connection
print "Content-Type: text/html" # HTML is following
print # blank line, end of headers
print "<TITLE>CGI script output</TITLE>"
print "<HTML>Here</HTML>"
connection = Connection()
The file is in my cgi-bin directory. When I try to run it through the web, I get:
A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred.
/var/www/cgi-bin/test2.py
9 print # blank line, end of headers
10 print "<TITLE>CGI script output</TITLE>"
11 print "<HTML>Here</HTML>"
12
13 connection = Connection()
connection undefined, Connection = <class 'pymongo.connection.Connection'>
/usr/lib64/python2.4/site-packages/pymongo/connection.py in __init__(self=Connection(None, None), host=['localhost'], port=27017, pool_size=None, auto_start_request=None, timeout=None, slave_okay=False, network_timeout=None, document_class=<type 'dict'>, tz_aware=False, _connect=True)
303
304 if _connect:
305 self.__find_master()
306
307 if username:
self = Connection(None, None), self.__find_master = Database(Connection(None, None), u'__find_master')
/usr/lib64/python2.4/site-packages/pymongo/connection.py in __find_master(self=Connection(None, None))
507 return node
508
509 raise AutoReconnect("could not find master/primary")
510
511 def __connect(self):
global AutoReconnect = <class pymongo.errors.AutoReconnect>
AutoReconnect: could not find master/primary
args = ('could not find master/primary',)
MongoDB is running on localhost.

I have resolved this myself. My script's access to the DB was being blocked by SELinux.
Try (as root):
setsebool httpd_can_network_connect
setsebool httpd_can_network_connect_db 1

web.py + lighttpd + matplotlib not working

I'm trying to deploy my web.py app with lighttpd. It doesn't work if import matplotlib.
This works...
hello.py:
#!/usr/bin/python
import web
# Say hello.
class Index:
def GET(self): return 'hello web.py'
if __name__ == "__main__":
app = web.application(('/*', 'Index'), globals())
app.run()
/etc/init.d/lighttpd restart
I go to my site and see "hello web.py".
But if I add import matplotlib to hello.py and restart the server, then when I go to the site I get a 500 - Internal Server Error.
Here's /var/log/lighttpd/error.log:
2010-12-24 00:17:31: (log.c.166) server started
2010-12-24 00:17:42: (mod_fastcgi.c.1734) connect failed: Connection refused on
unix:/tmp/fastcgi.socket-0
2010-12-24 00:17:42: (mod_fastcgi.c.3037) backend died; we'll disable it for 1 s
econds and send the request to another backend instead: reconnects: 0 load: 1
2010-12-24 00:17:43: (mod_fastcgi.c.2582) unexpected end-of-file (perhaps the fa
stcgi process died): pid: 4074 socket: unix:/tmp/fastcgi.socket-0
2010-12-24 00:17:43: (mod_fastcgi.c.3320) child exited, pid: 4074 status: 1
2010-12-24 00:17:43: (mod_fastcgi.c.3367) response not received, request sent: 9
53 on socket: unix:/tmp/fastcgi.socket-0 for /hello.py?, closing connection
2010-12-24 00:20:30: (server.c.1503) server stopped by UID = 0 PID = 4095
2010-12-24 00:20:30: (log.c.166) server started
-- Edit --
Here is my lighttpd.conf: http://pastebin.com/n6sG5z9K
Pretty sure it's just the default (except I set server.document-root = "/var/www/hello/")
Here is my fastcgi.conf:
server.modules += ( "mod_fastcgi" )
server.modules += ( "mod_rewrite" )
fastcgi.server = ( "/hello.py" =>
(( "socket" => "/tmp/fastcgi.socket",
"bin-path" => "/usr/bin/python /var/www/hello/hello.py",
"max-procs" => 1,
"bin-environment" => (
"REAL_SCRIPT_NAME" => ""
),
"check-local" => "disable"
))
)
url.rewrite-once = (
"^/favicon.ico$" => "/static/favicon.ico",
"^/static/(.*)$" => "/static/$1",
"^/(.*)$" => "/hello.py/$1",
)
Any suggestions?

Stumbled into this today (with Apache, but it's likely to be exactly the same issue). I redirected stdout and stderr from the script to see what was happening, and the issue is that matplotlib is trying to create a file:
Traceback (most recent call last):
File "/home/ec2-user/dlea/src/dla.py", line 24, in <module>
import dbm
File "/home/ec2-user/dlea/src/dbm.py", line 7, in <module>
import matplotlib
File "/usr/lib64/python2.6/site-packages/matplotlib/__init__.py", line 709, in <module>
rcParams = rc_params()
File "/usr/lib64/python2.6/site-packages/matplotlib/__init__.py", line 627, in rc_params
fname = matplotlib_fname()
File "/usr/lib64/python2.6/site-packages/matplotlib/__init__.py", line 565, in matplotlib_fname
fname = os.path.join(get_configdir(), 'matplotlibrc')
File "/usr/lib64/python2.6/site-packages/matplotlib/__init__.py", line 240, in wrapper
ret = func(*args, **kwargs)
File "/usr/lib64/python2.6/site-packages/matplotlib/__init__.py", line 439, in _get_configdir
raise RuntimeError("Failed to create %s/.matplotlib; consider setting MPLCONFIGDIR to a writable directory for matplotlib configuration data"%h)
RuntimeError: Failed to create /var/www/.matplotlib; consider setting MPLCONFIGDIR to a writable directory for matplotlib configuration data
Since it's being run as user httpd (Apache), it tries to create the file in /var/www/, which is root-owned, and not writeable by the Apache user.
One valid solution is as simple as setting the MPLCONFIGDIR to a temporary directory before importing matplotlib:
import os
import tempfile
os.environ['MPLCONFIGDIR'] = tempfile.mkdtemp()
import matplotlib
To track the issue, this is how I redirected stdout and stderr to some log file to see what was happening:
sys.stdout = open("/var/log/dla_stdout.txt", 'a')
sys.stderr = open("/var/log/dla_stderr.txt", 'a')
I actually got the solution from this other StackOverflow question: Setting Matplotlib MPLCONFIGDIR: consider setting MPLCONFIGDIR to a writable directory for matplotlib configuration data

I was following this recipe: http://webpy.org/cookbook/fastcgi-lighttpd
I overlooked a link at the top to this thread: http://www.mail-archive.com/webpy#googlegroups.com/msg02800.html
That thread had the solution. I run the python process like so:
/var/www/hello.py fastcgi 9080
and then set my fastcgi.conf like so:
fastcgi.server = ( "/hello.py" =>
((
"host" => "127.0.0.1",
"port" => 9080,
"check-local" => "disable"
))
)
Then it works. (Still not sure I've got everything configured properly, but things seem to be working.)

I fix the issue by:
pip install flup
don't need
/var/www/hello.py fastcgi 9080
my system is: amazon ec2, ubuntu 10.04
lighttpd: 1.4.26

My first guess is that you're getting an ImportError because matplotlib wasn't installed properly or isn't on the PYTHONPATH or some other crazy thing. The only way to know for sure is to look at the traceback. It shows you're running fastcgi, which means that the python code is being executed in another process. Therefore, you can't find the traceback in the lighttpd logs.
How are you running the fastcgi process? The traceback would have been written to its stderr. You might also consider using supervisord. It has support for redirecting stderr to a log file and various other things that make creating daemon processes easier.